Apparatus and method for encoding a spatial audio representation and apparatus and method for decoding an encoded audio signal using transmission metadata

Abstract

A device for encoding a spatial audio representation representing an audio scene to obtain an encoded audio signal, comprising: a transmission representation generator (600) for generating a transmission representation (611) based on the spatial audio representation, and for generating transmission metadata (610) related to the generation of the transmission representation (611) or indicating one or more directional properties of the transmission representation (611); and an output interface (640) for generating an encoded audio signal, the encoded audio signal including information related to the transmission representation (611) and information related to the transmission metadata (610).

Description

Technical Field

[0001] Embodiments of the invention relate to transmit channel or downmix signaling for directional audio coding. Background Art

[0002] The Directional Audio Coding (DirAC) technique [Pulkki07] is an efficient method for analyzing and reproducing spatial sound. DirAC uses a perceptually motivated representation of the sound field based on spatial parameters, namely the direction of arrival (DOA) and diffuseness measured for each frequency band. It is based on the assumption that at a certain moment and a critical frequency band, the spatial resolution of the auditory system is limited to decoding one cue for direction and another cue for inter-auditory coherence. Spatial sound is then represented in the frequency domain by crossfading two streams: a non-directional diffuse stream and a directional non-diffuse stream.

[0003] DirAC was originally designed for recording B-format sound, but can also be extended to match microphone signals to a specific speaker setup, such as 5.1[2] or any configuration of a microphone array[5]. In recent cases, greater flexibility is achieved by recording signals not targeted to a specific speaker setup, but rather to an intermediate format.

[0004] This intermediate format, which is well established in practice, is denoted by (Higher Order) Ambisonics [3]. From an Ambisonics signal, a signal can be generated for each desired loudspeaker setup, including a binaural signal for headphone reproduction. This requires a specific renderer applied to the Ambisonics signal, using either a linear Ambisonics renderer [3] or a parametric renderer such as Directional Audio Coding (DirAC).

[0005] An Ambisonics signal can be represented as a multi-channel signal, where each channel (called an Ambisonics component) is equivalent to the coefficient of a so-called spatial basis function. Using the weighted sum of these spatial basis functions (where the weights correspond to the coefficients), the original sound field at the recording location can be reconstructed [3]. Therefore, the spatial basis function coefficients (i.e., Ambisonics components) represent a compact description of the sound field at the recording location. There are different types of spatial basis functions, such as spherical harmonics (SH) [3] or cylindrical harmonics (CH) [3]. CH can be used when describing the sound field in 2D space (e.g. for 2D sound reproduction), while SH can be used to describe the sound field in both 2D and 3D space (e.g. for 2D and 3D sound reproduction).

[0006] For example, from a certain direction The incoming audio signal f(t) generates a spatial audio signal This spatial audio signal can be represented in Ambisonics format by expanding the spherical harmonics to a truncated order H:

[0007]

[0008] in are the spherical harmonics of order l and mode m, and φ lm (t) is the expansion coefficient. As the truncation order H increases, the expansion leads to a more accurate spatial representation. Figure 1a Spherical harmonics up to H=4th order with Ambisonics Channel Coding (ACN) index are shown in FIG. 1 for nth order and mth mode.

[0009] DirAC has been extended to provide higher-order Ambisonics signals from a first-order Ambisonics signal (FOA, also known as B-format) or from different microphone arrays [5]. This paper focuses on a more efficient method for synthesizing higher-order Ambisonics signals based on DirAC parameters and a reference signal. In this paper, the reference signal (also known as the downmix signal) is considered to be a subset of the higher-order Ambisonics signal or a linear combination of a subset of Ambisonics components.

[0010] In DirAC analysis, the spatial parameters of the DirAC are estimated from the audio input signal. Originally, DirAC was developed for a first-order Ambisonics (FOA) input, which can be obtained, for example, from a B-format microphone, however other input signals are also possible. In DirAC synthesis, output signals for spatial reproduction, such as loudspeaker signals, are calculated from the DirAC parameters and the associated audio signal. Solutions that use only the omnidirectional audio signal for the synthesis or that use the entire FOA signal have been described [Pulkki07]. Alternatively, only a subset of the four FOA signal components can be used for the synthesis.

[0011] Due to its effective representation of spatial sound, DirAC is also very suitable as the basis of a spatial audio coding system. The goal of such a system is to be able to encode a spatial audio scene at a low bit rate and reproduce the original audio scene as faithfully as possible after transmission. In this case, the DirAC analysis is followed by a spatial metadata encoder, which quantizes and encodes the DirAC parameters to obtain a low bit rate parameter representation. Together with the metadata, the downmix signal derived from the original audio input signal is encoded for transmission by a conventional audio core encoder. For example, an EVS-based audio encoder can be used to encode the downmix signal. The downmix signal consists of different channels, referred to as transmission channels. The downmix signal can be, for example, four coefficient signals, stereo pairs, or mono downmixes depending on the target bit rate that make up a B-format signal (i.e., FOA). The encoded spatial parameters and the encoded audio bitstream are multiplexed before transmission.

[0012] Context: System overview of DirAC-based spatial audio codec

[0013] In the following, an overview of a prior art spatial audio coding system based on DirAC, which is designed for immersive voice and audio services (IVAS), is presented. The goal of such a system is to be able to handle different spatial audio formats representing audio scenes, encode them at low bit rates, and reproduce the original audio scene as faithfully as possible after transmission.

[0014] The system can accept different representations of the audio scene as input. The input audio scene can be represented by a multichannel signal intended to be reproduced at different speaker positions, auditory objects and metadata describing the position of the objects over time, or a first-order or higher-order Ambisonics format representing the sound field at a listener or reference position.

[0015] Preferably, the system is based on 3GPP Enhanced Voice Services (EVS), as it is desirable for the solution to operate with low latency to enable conversational services over mobile networks.

[0016] Figure 1b, the encoder side of DirAC-based spatial audio coding supporting different audio formats is shown in . The acoustic / electrical input 1000 is input to the encoder interface 1010, wherein the encoder interface has specific functions for first-order Ambisonics (FOA) or high-order Ambisonics (HOA), as shown in 1013. In addition, the encoder interface has functions for multi-channel (MC) data (such as stereo data, 5.1 data, or data with two or more channels). In addition, the encoder interface 1010 has functions for object encoding, such as the audio object shown at 1011. The IVAS encoder includes a DirAC stage 1020 having a DirAC analysis block 1021 and a downmix (DMX) block 1022. The signal output by block 1022 is encoded by an IVAS core encoder 1040 such as an AAC or EVS encoder, and the metadata generated by block 1021 is encoded using a DirAC metadata encoder 1030.

[0017] Figure 1b The encoder side of DirAC-based spatial audio coding supporting different audio formats is shown. Figure 1b As shown in , the encoder (IVAS encoder) is able to support different audio formats presented to the system individually or simultaneously. The audio signal can essentially be an acoustic signal picked up by a microphone, or it can essentially be an electrical signal assumed to be transmitted to a speaker. The supported audio formats can be multi-channel signals (MC), first-order and higher-order Ambisonics (FOA / HOA) components, and audio objects. Complex audio scenes can also be described by combining different input formats. All audio formats are then transmitted to the DirAC analysis, which extracts a parameter representation of the complete audio scene. The direction of arrival (DOA) and diffusion measured per time-frequency unit form spatial parameters or are part of a larger set of parameters. The DirAC analysis is followed by a spatial metadata encoder, which quantizes and encodes the DirAC parameters to obtain a low-bitrate parameter representation.

[0018] In addition to the described channel-based, HOA-based, and object-based input formats, the IVAS encoder can receive a parameter representation of a spatial sound consisting of spatial and / or directional metadata and one or more associated audio input signals. The metadata may, for example, correspond to DirAC metadata, i.e., the DOA and diffuseness of the sound. The metadata may also include additional spatial parameters, such as multiple DOAs with associated energy measurements, distance or position values, or measurements related to the coherence of the sound field. The associated audio input signal may be composed of a mono signal, a first-order or higher-order Ambisonics signal, an X / Y stereo signal, an A / B stereo signal, or any other combination of signals produced by recording with microphones having various directional patterns and / or spacing from each other.

[0019] For parametric spatial audio input, the IVAS encoder determines the DirAC parameters for transmission based on the input spatial metadata.

[0020] Together with the parameters, a downmix (DMX) signal originating from different sources or audio input signals is encoded for transmission by a conventional audio core encoder. In this case, an EVS-based audio encoder is used to encode the downmix signal. The downmix signal consists of different channels, called transmission channels: the signal can be, for example, four coefficient signals constituting a B-format or First Order Ambisonics (FOA) signal, a stereo pair, or a mono downmix depending on the target bitrate. The encoded spatial parameters and the encoded audio bitstream are multiplexed before transmission over the communication channels.

[0021] Figure 2a The decoder side of DirAC-based spatial audio coding providing different audio formats is shown. In the decoder, Figure 2a As shown in , the transport channels are decoded by the core decoder, while the DirAC metadata is first decoded before being passed to the DirAC synthesis together with the decoded transport channels. At this stage, different options can be considered. It is possible to request that the audio scene be played directly on any speaker or headphone configuration, which is not possible in a conventional DirAC system ( Figure 2a The decoder can also deliver individual objects as they were presented on the encoder side ( Figure 2a Alternatively, you can request that the scene be rendered in Ambisonics format ( Figure 2a The FOA / HOA in the original system can be used for further operations, such as rotation, mirroring or translation of the scene, or the use of external renderers not defined in the original system.

[0022] In the decoder, Figure 2aAs shown in , the transport channels are decoded by the core decoder, while the DirAC metadata is first decoded before being passed to the DirAC synthesis together with the decoded transport channels. At this stage, different options can be considered. It is possible to request that the audio scene be played directly on any speaker or headphone configuration, which is not possible in a conventional DirAC system ( Figure 2a The decoder can also provide individual objects as they are presented on the encoder side ( Figure 2a Alternatively, you can request that the scene be rendered in Ambisonics format for further manipulation, such as rotation, reflection, or translation of the scene ( Figure 2a ), or use external renderers that are not defined in the original system.

[0023] Figure 2a 1045 and a decoder interface 1046 connected thereto. The IVAS decoder 1045 includes an IVAS core decoder 1060, which is configured to perform the decoding of the audio signal encoded by the IVAS decoder 1045. Figure 1b The IVAS core encoder 1040 of FIG. 1040 is used to decode the content encoded by the DirAC metadata encoder 1030. In addition, a DirAC metadata decoder 1050 is provided, which provides decoding functions for decoding the content encoded by the DirAC metadata encoder 1030. The DirAC synthesizer 1070 receives data from blocks 1050 and 1060 and with or without some user interaction, the output is input into a decoder interface 1046, which generates FOA / HOA data shown at 1083, multi-channel data (MC data) shown in block 1082, or object data shown in block 1080.

[0024] Figure 2b Conventional HOA synthesis using the DirAC paradigm is depicted in FIG. The input signal, referred to as the downmix signal, is a time-frequency signal analyzed by a frequency filter bank. The frequency filter bank 2000 may be a complex-valued filter bank such as a complex-valued QMF or a block transform such as an STFT. The HOA synthesis generates at the output a signal containing (H+1) 2 The output is an Ambisonics signal of order H with 1 components. Optionally, it can also output an Ambisonics signal rendered on a specific speaker layout. In the following, we will explain in detail how to obtain the (H+1) from the downmix signal with the input spatial parameters in some cases. 2 A quantity.

[0025] The downmix signal can be a mixture of the original microphone signals or the original signals depicting the original audio scene. For example, if the audio scene is captured by a sound field microphone, the downmix signal can be the omnidirectional component (W) of the scene, a stereo downmix (L / R), or a first-order Ambisonics signal (FOA).

[0026] For each time-frequency tile, the sound direction (also called direction of arrival (DOA)) and the diffuseness factor are estimated by the direction estimator 2020 and the diffuseness estimator 2010, respectively, if the downmix signal contains sufficient information to determine such DirAC parameters. For example, this is the case if the downmix signal is a first-order Ambisonics signal (FOA). Alternatively or if the downmix signal is not sufficient to determine such parameters, the parameters can be transmitted directly to the DirAC synthesis via an input bitstream containing spatial parameters. In the case of audio transmission applications, the bitstream may include quantized and encoded parameters received as auxiliary information, for example. In this case, the parameters are derived from the original microphone signal outside the DirAC synthesis module, or from the input audio format provided to the DirAC analysis module on the encoder side shown in switch 2030 or 2040.

[0027] The directional gain estimator 2050 uses the sound direction to estimate (H+1) for each of the plurality of time-frequency tiles. 2 Directional gain One or more groups of , where H is the order of the synthesized Ambisonics signal.

[0028] The directional gain can be obtained by evaluating the spatial basis functions for each estimated sound direction at the desired order (level) l and mode m of the Ambisonics signal to be synthesized. The sound direction can be, for example, in terms of a unit norm vector n(k,n) or in terms of an azimuth angle and / or elevation angle θ(k,n), which are related, for example, as follows:

[0029]

[0030] After estimating or obtaining the sound direction, the response of the spatial basis functions of the required order (level) l and mode m can be determined, for example, by considering the real-valued spherical harmonics normalized to the spatial basis functions in SN3D:

[0031]

[0032] The range is 0≤l≤H, and -l≤m≤l. l |m| is the Legendre function, and N l |m|is the normalization term of the Legendre function and trigonometric functions, which takes the following form for SN3D:

[0033]

[0034] For m = 0, Kronecker-deltaδ m is 1, otherwise 0. The directional gain is then derived directly for each time-frequency tile with index (k,n):

[0035]

[0036] By deriving the reference signal P from the downmix signal ref And multiply it by the factor function Ψ(k,n) of directional gain and diffusion to calculate the direct sound Ambisonics component

[0037]

[0038] For example, the reference signal P ref It can be the omnidirectional component of the downmix signal or a linear combination of the K channels of the downmix signal.

[0039] The diffuse sound Ambisonics component can be modeled by using the response of the spatial basis functions to the sound coming from all possible directions. An example is by considering the spatial basis function At all possible angles The average response is defined by the integral of the squared magnitude over θ

[0040]

[0041] According to the signal P diff The diffuse sound Ambisonics component is calculated by multiplying the average response and the diffusion coefficient by a factor function Ψ(k,n).

[0042]

[0043] The signal This can be obtained by using a different decorrelator applied to the reference signal.

[0044] Finally, the direct sound Ambisonics component and the diffuse sound Ambisonics component are combined 2060, for example, via a summation operation, to obtain a final Ambisonics component of the desired order (level) l and mode m of the time-frequency tile (k, n) Right now,

[0045]

[0046] The obtained Ambisonics components can be transformed back to the time domain using an inverse filter bank 2080 or an inverse STFT, stored, transmitted, or used for, e.g., spatial sound reproduction applications. Alternatively, a linear Ambisonics renderer 2070 can be applied to each frequency band in order to obtain a signal to be played on a specific loudspeaker layout or via headphones before converting the loudspeaker signal or the binaural signal to the time domain.

[0047] It should be noted that [Thiergart17] also teaches the possibility of synthesizing diffuse sound components only up to order L, where L < H. This reduces the computational complexity while avoiding synthesis artifacts due to the extensive use of decorrelators.

[0048] An object of the present invention is to provide an improved concept for generating a sound field description based on an input signal.

[0049] Prior art: DirAC synthesis for mono and FOA downmix signals

[0050] Conventional DirAC synthesis based on a received DirAC-based spatial audio coding stream is described below. The rendering performed by DirAC synthesis is based on the decoded downmixed audio signal and the decoded spatial metadata.

[0051] The downmixed signal is the input signal for DirAC synthesis. This signal is transformed to the time-frequency domain by a filter bank. The filter bank can be a complex-valued filter bank (such as a complex-valued QMF) or a block transform (such as an STFT).

[0052] DirAC parameters can be directly transmitted to DirAC synthesis via an input bitstream containing spatial parameters. In the case of an audio transmission application, the bitstream can include, e.g., quantized and encoded parameters received as side information.

[0053] To determine the channel signals for loudspeaker-based sound reproduction, each loudspeaker signal is determined based on the downmixed signal and the DirAC parameters. The signal P j (k,n) for the j-th loudspeaker is obtained as a combination of the direct sound component and the diffuse sound component, i.e.,

[0054] P j (k,n) = P dir,j (k,n) + P diff，j (k,n)

[0055] The so-called reference signal P j can be scaled by a factor depending on the diffuseness parameter Ψ(k,n) and the directional gain factor G ref,j(k,n) to obtain the direct sound component P of the j-th speaker channel dir,j (k,n), where the gain factor depends on the direction of arrival (DOA) of the sound and possibly also on the position of the j-th loudspeaker channel. The DOA of the sound can be expressed, for example, as a unit norm vector v(k,n) or as an azimuth angle and / or elevation angle θ(k,n), which are related, for example, as follows:

[0056]

[0057] The directional gain factor G can be calculated using well-known methods j (v(k,n)), such as vector-based amplitude translation (VBAP) [Pulkki97].

[0058] In summary, the direct sound component can be expressed as

[0059]

[0060] The spatial parameters describing the DOA and diffuseness of the sound are estimated at the decoder from the transmission channels or obtained from parametric metadata included in the bitstream.

[0061] The diffuse sound component P can be determined based on the reference signal and the diffuseness parameter diff,j (k,n):

[0062]

[0063] Normalization factor G norm Depends on the playback speaker configuration. Typically, the diffuse sound components P associated with different speaker channels diff,j (k,n) are further processed, i.e., they are decorrelated with each other. This can also be achieved by decorrelating the reference signal for each output channel, i.e.

[0064]

[0065] in Indicates P ref,j The decorrelated version of (k,n).

[0066] The reference signal for the jth output channel is obtained based on the transmitted downmix signal. In the simplest case, the downmix signal consists of a mono omnidirectional signal (e.g., the omnidirectional component W(k,n) of the FOA signal), and the reference signal is the same for all output channels:

[0067] P ref，j (k,n)=W(k,n)

[0068] If the transmission channels correspond to the four components of the FOA signal, the reference signal can be obtained by a linear combination of the FOA components. Typically, the FOA signals are combined such that the reference signal of the jth channel corresponds to a virtual cardioid microphone signal pointing in the direction of the jth loudspeaker [Pulkki07].

[0069] DirAC synthesis generally provides improved sound reproduction quality for an increased number of downmix channels, since the required amount of synthesis decorrelation, the degree of non-linear processing of directional gain factors or crosstalk between different loudspeaker channels may be reduced, and associated artifacts may be avoided or mitigated.

[0070] In general, a direct approach of introducing many different transmission signals into the encoded audio scene is inflexible on the one hand and bitrate consuming on the other hand. Typically, it may not be necessary in all cases to introduce all four component signals of, for example, a first-order Ambisonics signal into the encoded audio signal, because one or more components do not have a significant energy contribution. On the other hand, the bitrate requirements may be strict, which prohibits the introduction of more than two transmission channels into the encoded audio signal representing the spatial audio representation. In the case of such strict bitrate requirements, it is necessary for the encoder and the decoder to pre-negotiate a certain representation, and based on this pre-negotiation, a certain number of transmission signals are generated in a pre-negotiated manner, and then the audio decoder can synthesize the audio scene from the encoded audio signal based on the pre-negotiated knowledge. However, although this is useful for bitrate requirements, it is inflexible and may additionally significantly reduce the audio quality, because the pre-negotiation process may not be the best choice for a certain audio segment, or may not be the best choice for all frequency bands or all time frames of the audio segment.

[0071] Therefore, the prior art process of representing an audio scene is not optimal, is not flexible with respect to bitrate requirements and, moreover, is likely to result in significantly degraded audio quality. Summary of the invention

[0072] It is an object of the present invention to provide an improved concept for encoding a spatial audio representation or decoding an encoded audio signal.

[0073] This object is achieved by the device for encoding a spatial audio representation of claim 1, the device for decoding an encoded audio signal of claim 21, the method for encoding a spatial audio representation of claim 39, the method for decoding an encoded audio signal of claim 41, the computer program of claim 43 or the encoded audio signal of claim 44.

[0074] The invention is based on the finding that significant improvements in bitrate, flexibility and audio quality can be obtained by using, in addition to a transport representation derived from a spatial audio representation, transport metadata related to the generation of the transport representation or indicating one or more directional properties of the transport representation. Thus, a device for encoding a spatial audio representation representing an audio scene generates a transport representation from the audio scene, and additionally the transport metadata is related to the generation of the transport representation or indicating one or more directional properties of the transport representation, or is related to the generation of the transport representation and indicating one or more directional properties of the transport representation. Furthermore, an output interface generates an encoded audio signal comprising information related to the transport representation and information related to the transport metadata.

[0075] On the decoder side, the device for decoding an encoded audio signal comprises an interface for receiving an encoded audio signal, the encoded audio signal comprising information related to a transmission representation and information related to transmission metadata, and then a spatial audio synthesizer synthesizes a spatial audio representation using both the information related to the transmission representation and the information related to the transmission metadata.

[0076] An explicit indication of how a transport representation (such as a downmix signal) has been generated and / or an explicit indication of one or more directional properties of the transport representation, by means of additional transport metadata, allows the encoder to generate the encoded audio scene in a highly flexible manner, which provides good audio quality on the one hand and meets small bitrate requirements on the other hand. In addition, by means of the transport metadata, the encoder can even find the desired optimal balance between the bitrate requirements on the one hand and the audio quality represented by the encoded audio signal on the other hand. Thus, the use of explicit transport metadata allows the encoder to apply different ways of generating transport representations and additionally adapt the transport representation generation not only from audio segment to audio segment, but even from one audio frame to the next, or from one frequency band to another within the same audio frame. Of course, flexibility is obtained by generating the transport representation independently for each time / frequency tile, so that, for example, the same transport representation can be generated for all frequency bins within a time frame, or alternatively, the same transport representation can be generated for the same frequency band over multiple audio time frames, or an independent transport representation can be generated for each frequency bin of each time frame. All this information, i.e. the way the transport representation is generated and whether it relates to a complete frame, only to a time / frequency bin or to a certain frequency band over multiple time frames, is also included in the transport metadata so that the spatial audio synthesizer knows what has been done on the encoder side and can then apply the best process on the decoder side.

[0077] Preferably, some of the transmission metadata alternatives are selection information indicating which components of a particular group of components representing the audio scene have been selected. Another transmission metadata alternative relates to combination information, i.e. whether and / or how certain component signals of the spatial audio representation are combined to generate the transmission representation. Further information used as transmission metadata relates to sector / hemisphere information indicating which sector or hemisphere a certain transmission signal or transmission channel is associated with. Furthermore, metadata useful in the context of the present invention relates to viewing direction information indicating the viewing direction of an audio signal, which is preferably included as a transmission signal of a plurality of different transmission signals in the transmission representation. Other viewing direction information relates to microphone viewing directions, which, when the transmission representation consists of one or more microphone signals, can be recorded, for example, by physical microphones in a (spatially extended) microphone array or by coincident microphones, or alternatively, can be generated synthetically. Other transmission metadata relates to shape parameter data indicating whether the microphone signal is an omnidirectional signal or has a different shape, such as a cardioid shape or a dipole shape. In case there is more than one microphone signal within the transmission representation, additional transmission metadata relates to the position of the microphones. Other useful transmission metadata relates to orientation data of one or more microphones, distance data indicating the distance between two microphones, or a directional pattern of microphones. Furthermore, additional transmission metadata may relate to a description or identification of a microphone array (such as a circular microphone array), or which microphone signals from such a circular microphone array have been selected for the transmission representation.

[0078] Further transmission metadata may relate to information about beamforming, corresponding beamforming weights or corresponding beam directions, and in this case the transmission representation typically preferably consists of a synthetically created signal with a specific beam direction. Further transmission metadata alternatives may relate to pure information whether the included transmission signal is an omnidirectional microphone signal or a non-omnidirectional microphone signal, such as a cardioid signal or a dipole signal.

[0079] It is therefore clear that the different transport metadata alternatives are highly flexible and can be represented in a highly compact manner, so that the additional transport metadata does not typically result in a significant amount of additional bitrate. Alternatively, the bitrate requirement for the additional transport metadata can typically be as small as less than 1% of the amount of the transport representation or even less than 1 / 1000 or even less. However, on the other hand, this very small amount of additional metadata results in a higher flexibility, while at the same time the audio quality is significantly improved due to the additional flexibility and the possibility to have a changed transport representation on different audio segments or even in the same audio segment at different time frames and / or frequency bins.

[0080] Preferably, the encoder additionally comprises a parameter processor for generating spatial parameters from the spatial audio representation, so that in addition to the transmission representation and the transmission metadata, the spatial parameters are also included in the encoded audio signal to enhance the audio quality relative to the quality that can only be obtained by the transmission representation and the transmission metadata. These spatial parameters are preferably time- and / or frequency-dependent direction of arrival (DoA) data and / or frequency- and / or time-dependent diffuseness data, such as known from DirAC encoding.

[0081] On the audio decoder side, an input interface receives an encoded audio signal comprising information relating to a transmission representation and information relating to transmission metadata. Furthermore, a spatial audio synthesizer provided in the device for decoding the encoded audio signal uses both the information relating to the transmission representation and the information relating to the transmission metadata to synthesize the spatial audio representation. In a preferred embodiment, the decoder additionally uses the optionally transmitted spatial parameters to synthesize the spatial audio representation, which uses not only the information relating to the transmission metadata and the information relating to the transmission representation, but also the spatial parameters.

[0082] The device for decoding the encoded audio signal receives transmission metadata, interprets or parses the received transmission metadata, and then controls a combiner to combine transmission representation signals or controls a combiner to select or generate one or more reference signals from the transmission representation signals. The combiner / selector / reference signal generator then forwards the reference signal to a component signal calculator, which calculates the required output components based on a specific selected or generated reference signal. In a preferred embodiment, not only the combiner / selector / reference signal generator as in a spatial audio synthesizer is controlled by transmission metadata, but also the component signal calculator is controlled by it, so that based on the received transmission data, not only the reference signal generation / selection is controlled, but also the actual component calculation is controlled. However, embodiments in which only the component signal calculation is controlled by transmission metadata or only the reference signal generation or selection is controlled only by transmission metadata are also useful and provide improved flexibility over existing solutions.

[0083] A preferred process for different signal selection alternatives is to select one of multiple signals in the transmission representation as a reference signal for a first subset of component signals, and to select another transmission signal in the transmission representation for another orthogonal subset of component signals for multi-channel output, first-order or higher-order Ambisonics output, audio object output, or binaural output. Other processes rely on computing reference signals based on linear combinations of the individual signals included in the transmission representation. Depending on the specific transmission representation implementation, transmission metadata is used to determine reference signals for (virtual) channels from the actual transmitted transmission signals, and to determine missing components, such as transmitted or generated omnidirectional signal components, based on fallback. These processes rely on using spatial basis function responses associated with a specific mode and order of a first-order or higher-order Ambisonics spatial audio representation to preferably compute missing FOA or HOA components.

[0084] Other embodiments relate to transmission metadata describing microphone signals included in the transmission representation, and based on the transmitted shape parameters and / or viewing direction, the reference signal determines the transmission metadata applicable to the received transmission metadata. Furthermore, the calculation of the omnidirectional signal or the dipole signal and the additional synthesis of the residual components are also performed based on the following transmission metadata: indicating that, for example, the first transmission channel is a left cardioid signal or a front cardioid signal, and the second transmission signal is a right cardioid signal or a rear cardioid signal.

[0085] Another process involves determining a reference signal based on a minimum distance from a certain loudspeaker to a certain microphone position, or selecting as a reference signal the microphone signal included in the transmission representation with the closest viewing direction or the closest beamformer or a certain closest array position. Another process is to select an arbitrary transmission signal as a reference signal for all direct sound components, and use all available transmission signals (such as omnidirectional signals sent from spaced microphones) to generate a diffuse sound reference signal, and then generate the corresponding component by adding the direct and diffuse components to obtain the final channel or Ambisonics component or object signal or binaural channel signal. Another process specifically implemented when calculating the actual component signal based on a specific reference signal involves setting (preferably limiting) the relevant quantity based on a specific microphone distance. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] The preferred embodiments of the present invention are disclosed below with reference to the accompanying drawings, in which:

[0087] Figure 1a Spherical harmonics with Ambisonics channel / component numbers are shown;

[0088] Figure 1b shows the encoder side of a DirAC based spatial audio coding processor;

[0089] Figure 2a A decoder for a DirAC-based spatial audio coding processor is shown;

[0090] Figure 2b A high-level Ambisonics synthesis processor known in the art is shown;

[0091] Figure 3 The encoder side of Dirac based spatial audio coding supporting different audio formats is shown.

[0092] Figure 4 shows the decoder side of Dirac based spatial audio coding providing different audio formats;

[0093] Figure 5 Another embodiment of an apparatus for encoding a spatial audio representation is shown;

[0094] Figure 6 Another embodiment of an apparatus for encoding a spatial audio representation is shown;

[0095] Figure 7 Another embodiment of an apparatus for decoding an encoded audio signal is shown;

[0096] Figure 8a A set of implementations of a transport representation generator that can be used alone or together with each other are shown;

[0097] Figure 8b illustrates a table showing different transport metadata alternatives that can be used alone or together with each other;

[0098] Figure 8c Another implementation of a metadata encoder for transmitting metadata or, if appropriate, spatial parameters is shown;

[0099] Figure 9a Shows Figure 7 A preferred implementation of a spatial audio synthesizer;

[0100] Figure 9b shows an encoded audio signal having a transmission representation with n transmission signals, transmission metadata and optional spatial parameters;

[0101] Fig.9c illustrates a table showing the functionality of a reference signal selector / generator depending on speaker identification and transmission metadata;

[0102] Figure 9d Another embodiment of a spatial audio synthesizer is shown;

[0103] Fig.9eAnother table showing different transmission metadata is illustrated;

[0104] Figure 9f Another implementation of a spatial audio synthesizer is shown;

[0105] Figure 9g Another embodiment of a spatial audio synthesizer is shown;

[0106] Figure 9h Another set of implementation alternatives for a spatial audio synthesizer that can be used alone or together with each other is shown;

[0107] Fig.10 An exemplary preferred implementation for calculating low-order or mid-order sound field components using direct and diffuse signals is shown;

[0108] Fig.11 Another implementation method of calculating higher-order sound field components using only direct components without using diffuse components is shown; and

[0109] Fig.12 Another implementation using the direct part combined with the diffuse part to compute a (virtual) loudspeaker signal component or object is shown. DETAILED DESCRIPTION

[0110] Figure 6 An apparatus for encoding a spatial audio representation representing an audio scene is shown. The apparatus comprises a transmission representation generator 600 for generating a transmission representation from the spatial audio representation. In addition, the transmission representation generator 600 generates transmission metadata related to the generation of the transmission representation or indicating one or more directional properties of the transmission representation. The apparatus additionally comprises an output interface 640 for generating an encoded audio signal, wherein the encoded audio signal comprises information related to the transmission representation and information related to the transmission metadata. In addition to the transmission representation generator 600 and the output interface 640, the apparatus preferably comprises a user interface 650 and a parameter processor 620. The parameter processor 620 is configured to derive spatial parameters from the spatial audio representation and preferably provide (encoded) spatial parameters 612. In addition, in addition to the (encoded) spatial parameters 612, the (encoded) transmission metadata 610 and the (encoded) transmission representation 611 are forwarded to the output interface 640 to preferably multiplex the three encoded items into the encoded audio signal.

[0111] Figure 7A preferred implementation of an apparatus for decoding an encoded audio signal is shown. The encoded audio signal is input to an input interface 700, and the input interface receives information about a transport representation and information about transport metadata within the encoded audio signal. The transport representation 711 is forwarded from the input interface 700 to a spatial audio synthesizer 750. Furthermore, the spatial audio synthesizer 750 receives the transport metadata 710 from the input interface and preferably additionally receives spatial parameters 712 if included in the encoded audio signal. The spatial audio synthesizer 750 uses the items 710, 711 and preferably additionally the item 712 in order to synthesize the spatial audio representation.

[0112] Figure 3 Shown is the Figure 3 The instructions in the figure are preferred implementations of an apparatus for encoding a spatial audio representation of a spatial audio signal. Specifically, the spatial audio signal is input into a downmix generation block 610 and into a spatial audio analysis block 621. The spatial parameters 615 derived from the spatial audio analysis block 621 according to the spatial audio signal are input into a metadata encoder 622. In addition, the downmix parameters 630 generated by the downmix generation block 601 are also input into a metadata encoder 603. The metadata encoder 621 and the metadata encoder 603 are connected in a Figure 3 603, and the core-encoded representation 611 is input to a bitstream generator 641, which additionally receives the encoded downmix parameters 610 and the encoded spatial parameters 612. Thus, Figure 6 The transmission representation generator 600 shown in FIG. 6 includes Figure 3 The downmix generation block 601 and the core encoder block 603 in the embodiment of FIG. Figure 6 The parameter processor 620 shown in FIG. 6 comprises a spatial audio analyzer block 621 and a metadata encoder block 622 for the spatial parameters 615. In addition, Figure 6 The transmission representation generator 60 of additionally comprises a metadata encoder block 603 for transmission metadata 630, which is output by the metadata encoder 603 as encoded transmission metadata 610. Figure 3 In the embodiment of FIG. 6 , the output interface 640 is implemented as a bit stream generator 641 .

[0113] Figure 4A preferred implementation of an apparatus for decoding an encoded audio signal is shown. Specifically, the apparatus comprises a metadata decoder 752 and a core decoder 751. The metadata decoder 752 receives as input the encoded transport metadata 710, and the core decoder 751 receives the encoded transport representation 711. Furthermore, the metadata decoder 752 preferably receives the encoded spatial parameters 712 when available. The metadata decoder decodes the transport metadata 710 to obtain the downmix parameters 720, and the metadata decoder 752 preferably decodes the encoded spatial parameters 712 to obtain the decoded spatial parameters 722. The decoded transport representation or downmix audio representation 721 is input into a spatial audio synthesis block 753 together with the transport metadata 720, and in addition, the spatial audio synthesis block 753 may receive spatial parameters 722 in order to generate a spatial audio representation including a first-order or higher-order (FOA / HOA) representation 754 or including a multi-channel (MC) representation 755 or including an object representation (object) 756 using two components 721 and 720 or all three components 721, 720 and 722, as shown in FIG. Figure 4 Therefore, Figure 7 The apparatus for decoding the encoded audio signal shown in FIG. 7 includes in the spatial audio synthesizer 750 Figure 4 Blocks 752, 751 and 753, and the spatial audio representation may include Figure 4 One of the alternatives shown at 754, 755 and 756.

[0114] Figure 5 Another implementation of an apparatus for encoding a spatial audio representation representing an audio scene is shown. Here, the spatial audio representation representing the audio scene is provided as a microphone signal and preferably as additional spatial parameters associated with the microphone signal. Figure 5 In the embodiment of Figure 6 The transport representation 600 in question comprises a downmix generation block 601, a metadata encoder 603 for downmix parameters 613 and a core encoder 602 for the downmix audio representation. Figure 3 Compared to the embodiment of , the spatial audio analyser block 621 is not included in the device for encoding, since the microphone input already has (preferably in separated form) the microphone signal on the one hand and the spatial parameters on the other hand.

[0115] In About Figures 3 to 5 In the discussed embodiment, the downmix audio 614 represents a transport representation and the downmix parameters 613 represent an alternative to transport metadata related to the generation of the transport representation or, as will be outlined later, indicating one or more directional properties of the transport representation.

[0116] Preferred embodiments of the present invention: Downmix signaling for flexible transmission channel configuration

[0117] In some applications, due to bit rate limitations, it is not possible to send all four components of the FOA signal as transmission channels, but only a downmix signal with a reduced number of signal components or channels can be sent. In order to achieve improved reproduction quality at the decoder, the generation of the sent downmix signal can be done in a time-varying manner and can be adapted to the spatial audio input signal. If the spatial audio coding system allows the inclusion of flexible downmix signals, it is important to send not only these transmission channels but also metadata specifying the important spatial characteristics of the downmix signal. Then, the DirAC synthesis at the decoder of the spatial audio coding system can adapt the rendering process in the best way taking into account the spatial characteristics of the downmix signal. Therefore, the present invention proposes to include metadata related to the downmix in the parameter spatial audio coding stream, which metadata is used to specify or describe the important spatial characteristics of the downmix transmission channels so as to improve the rendering quality at the spatial audio decoder.

[0118] In the following, illustrative examples of actual downmix signal configurations are described.

[0119] If the input spatial audio signal mainly includes sound energy in the horizontal plane, only the first three signal components of the FOA signal corresponding to the omnidirectional signal, the dipole signal aligned with the x-axis of the Cartesian coordinate system, and the dipole signal aligned with the y-axis are included in the downmix signal, while the dipole signal aligned with the z-axis is excluded.

[0120] In another example, only two downmix signals may be sent to further reduce the bit rate required to transmit the channels. For example, if there is primary sound energy originating from the left hemisphere, it is advantageous to generate a downmix channel comprising sound energy primarily from the left direction and an additional downmix channel comprising sound primarily originating from the opposite direction (i.e., the right hemisphere in this example). This can be achieved by a linear combination of the FOA signal components so that the resulting signal corresponds to a directional microphone signal having a cardioid directivity pattern pointing to the left and right, respectively. Similarly, downmix signals corresponding to first-order directivity patterns pointing forward and backward, respectively, or any other desired directivity pattern, can be generated by appropriately combining the FOA input signals.

[0121] In the DirAC synthesis stage, the speaker output channels must be adapted to the downmix configuration actually used based on the sent spatial metadata (e.g., DOA and diffuseness of the sound) and the audio transmission channels. More specifically, the reference signal P for the jth speaker is ref,j The most appropriate choice of (k,n) depends on the directional characteristics of the downmix signal and the position of the jth loudspeaker.

[0122] For example, if the downmix signal corresponds to two cardioid microphone signals pointing to the left and right, respectively, the reference signal of the loudspeaker located in the left hemisphere should use the cardioid signal pointing to the left as the reference signal P ref,j (k,n). The centrally located loudspeaker may instead use a linear combination of the two downmix signals.

[0123] On the other hand, if the downmixed signal corresponds to two cardioid microphone signals pointing forward and backward, respectively, the reference signal of the loudspeaker located in the front hemisphere should use the cardioid signal pointing forward alone as the reference signal P ref,j (k,n).

[0124] It is important to note that if the DirAC synthesis uses the wrong downmix signal as reference signal for rendering, a significant degradation in spatial audio quality must be expected. For example, if a downmix signal corresponding to a cardioid microphone pointing to the left is used to generate output channel signals for a loudspeaker located in the right hemisphere, signal components originating from the left hemisphere of the input sound field will be directed mainly to the right hemisphere of the reproduction system, resulting in an incorrect spatial image at the output.

[0125] Therefore, preferably parameter information is included in the spatial audio coding stream, which parameter information specifies the spatial characteristics of the downmix signal, such as the directionality pattern of the corresponding directional microphone signal. The DirAC synthesis at the decoder of the spatial audio coding system can then adapt the rendering process in an optimal way taking into account the spatial characteristics of the downmix signal described in the downmix related metadata.

[0126] Flexible downmixing of FOA and HOA audio inputs using Ambisonics component selection

[0127] In this embodiment, the spatial audio signal (ie, the audio input signal of the encoder) corresponds to a FOA (First Order Ambisonics) or HOA (Higher Order Ambisonics) audio signal. Figure 3 The corresponding block scheme of the encoder is depicted in . The input of the encoder is a spatial audio signal, such as a FOA or HOA signal. In the "Spatial Audio Analysis" block, the DirAC parameters, i.e. the spatial parameters (e.g. DOA and diffuseness) are estimated as described before. The downmix signal of the proposed flexible downmix is ​​generated in the "Downmix Generation" block, which will be explained in more detail below. The generated downmix signal is called D m(k,n), where m is the index of the downmix channel. The generated downmix signal is then encoded in the "core encoder" block, for example using an EVS-based audio encoder as explained before. The downmix parameters, i.e. parameters describing relevant information about how to create the downmix or other directional properties of the downmix signal, are encoded in the metadata encoder together with the spatial parameters. Finally, the encoded metadata and the encoded downmix signal are converted into a bitstream, which can be sent to the decoder.

[0128] In the following, the "downmix generation" block and the downmix parameters are explained in more detail. For example, if the input spatial audio signal mainly comprises sound energy in the horizontal plane, only the three signal components of the FOA / HOA signal with the omnidirectional signal W(k,n), the dipole signal X(k,n) aligned with the x-axis of the Cartesian coordinate system and the dipole signal Y(k,n) aligned with the y-axis are included in the downmix signal, while the dipole signal Z(k,n) aligned with the z-axis (and all other higher-order components, if present) are excluded. This means that the downmix signal is given by:

[0129] D1(k,n)=W(k,n), D2(k,n)=X(k,n), D3(k,n)=Y(k,n)

[0130] Alternatively, if, for example, the input spatial audio signal comprises mainly acoustic energy in the xz plane, the downmix signal comprises the dipole signal Z(k,n) but not the dipole signal Y(k,n).

[0131] In this embodiment, Figure 3 The downmix parameters depicted in contain information about which FOA / HOA components have been included in the downmix signal. For example, the information may be a set of integers corresponding to the indices of the selected FOA components, e.g., {1, 2, 4} if W(k, n), X(k, n) and Z(k, n) components are included.

[0132] Note that the FOA / HOA components may be selected for the downmix signal, for example, based on manual user input or automatically. For example, when recording a spatial audio input signal on an airport runway, it may be assumed that most of the sound energy is contained in a particular vertical Cartesian plane. In this case, for example, the W(k,n), X(k,n) and Z(k,n) components are selected. In contrast, if the recording is made at a street intersection, it may be assumed that most of the sound energy is contained in a horizontal Cartesian plane. In this case, for example, the W(k,n), X(k,n) and Y(k,n) components are selected. Alternatively, if, for example, a camera is used with the audio recording, a facial recognition algorithm may be used to detect in which Cartesian plane the speaker is located, and therefore, the FOA components corresponding to that plane may be selected for downmixing. Alternatively, the plane of the Cartesian coordinate system with the highest energy may be determined by using a prior art sound source localization algorithm.

[0133] Note also that the FOA / HOA component selection and the corresponding downmix metadata may be time and frequency dependent, e.g., a different set of components and indices may be automatically selected for each frequency band and time instance, respectively (e.g., by automatically determining the Cartesian plane with the highest energy at each time-frequency point). For example, direct sound energy may be localized by exploiting information contained in time-frequency dependent spatial parameters [Thiergart09].

[0134] Figure 4 The decoder block scheme corresponding to this embodiment is depicted in . The input of the decoder is a bitstream containing encoded metadata and an encoded downmix audio signal. The downmix audio signal is decoded in the "core decoder" and the metadata is decoded in the "metadata decoder". The decoded metadata consists of spatial parameters (e.g., DOA and diffuseness) and downmix parameters. The decoded downmix audio signal and spatial parameters are used in the "spatial audio synthesis" block to create the required spatial audio output signal, which can be, for example, a FOA / HOA signal, a multi-channel (MC) signal (e.g., a speaker signal), an audio object, or a binaural stereo output for headphone playback. Spatial audio synthesis is additionally controlled by the downmix parameters, as described below.

[0135] The spatial audio synthesis (DirAC synthesis) described above requires a suitable reference signal P for each output channel j ref,j In the present invention, it is proposed to use additional downmix metadata to determine the downmix signal D m (k,n) calculate P ref,j In this embodiment, the downmix signal D m (k,n) consists of specifically selected components of the FOA or HOA signal, and the downmix metadata describes which FOA / HOA components have been sent to the decoder.

[0136] When rendering to speakers (i.e., the MC output of the decoder), a high quality output can be achieved when a so-called virtual microphone signal is calculated for each speaker channel, which is directed to the corresponding speaker, as explained in [Pulkki07]. Typically, calculating the virtual microphone signal requires that all FOA / HOA components are available in the DirAC synthesis. However, in this embodiment, only a subset of the original FOA / HOA components are available at the decoder. In this case, the virtual microphone signal can only be calculated for the Cartesian plane for which the FOA / HOA components are available (as indicated by the downmix metadata). For example, if the downmix metadata indicates that the W(k,n), X(k,n), and Y(k,n) components have been sent, we can calculate the virtual microphone signals for all speakers in the xy plane (horizontal plane), where the calculation can be performed as described in [Pulkki07]. For elevated speakers outside the horizontal plane, we can calculate the reference signal P ref,j (k,n) Using a fallback solution, for example, we can use the omnidirectional component W(k,n).

[0137] Note that a similar concept can be used when rendering to binaural stereo output (e.g., for headphone playback). In this case, the two virtual microphones for the two output channels are pointed at virtual stereo speakers, where the position of the speakers depends on the listener's head orientation. If the virtual speakers are located in the Cartesian plane for which the FOA / HOA components have been sent as indicated by the following mixing data, then we can compute the corresponding virtual microphone signals. Otherwise, the reference signal P ref,j (k,n), for example the omnidirectional component W(k,n) uses a fallback solution.

[0138] When rendering to FOA / HOA( Figure 4 The downmix metadata is used as follows when the FOA / HOA output of the decoder is sent: The downmix metadata indicates which FOA / HOA components have been sent. These components do not need to be calculated in the spatial audio synthesis, because the sent components can be used directly in the decoder output. In the spatial sound synthesis, for example, by using the omnidirectional component W(k,n) as the reference signal P ref,j (k,n) to compute all remaining FOA / HOA components. For example, the use of spatial metadata to synthesize FOA / HOA components from the omnidirectional component W(k,n) is described in [Thiergart17].

[0139] Flexible downmixing of FOA and HOA audio inputs using combined Ambisonics components

[0140] In this embodiment, the spatial audio signal (ie, the audio input signal of the encoder) corresponds to a FOA (First Order Ambisonics) or HOA (Higher Order Ambisonics) audio signal. Figure 3 and Figure 4 The corresponding block scheme of the encoder is depicted in . In this embodiment, only two downmix signals can be sent from the encoder to the decoder to further reduce the bit rate required for the transmission channels. For example, if there is a main sound energy originating from the left hemisphere, it is advantageous to generate a downmix channel including sound energy mainly from the left hemisphere and an additional downmix channel including sound mainly from the opposite direction (i.e., the right hemisphere in this example). This can be achieved by a linear combination of the FOA or HOA audio input signal components so that the resulting signal corresponds to a directional microphone signal having, for example, a cardioid directivity pattern pointing to the left and right hemispheres, respectively. Similarly, downmix signals corresponding to first-order (or higher-order) directivity patterns or any other desired directivity patterns pointing to the front and rear directions, respectively, can be generated by appropriately combining the FOA or HOA audio input signals.

[0141] exist Figure 3 The downmix signal is generated in the encoder in the "Downmix Generation" block in . The downmix signal is obtained from a linear combination of the FOA or HOA signal components. For example, in the case of a FOA audio input signal, the four FOA signal components correspond to an omnidirectional signal W(k,n) and three dipole signals X(k,n), Y(k,n) and Z(k,n), where the directivity pattern is aligned with the x-axis, y-axis and z-axis of a Cartesian coordinate system. These four signals are generally referred to as B-format signals. The resulting directivity pattern that can be obtained by the linear combination of the four B-format components is generally referred to as a first-order directivity pattern. The first-order directivity pattern or the corresponding signal can be expressed in different ways. For example, the m-th downmix signal D m (k,n) can be represented by a linear combination of B-format signals with associated weights, i.e.,

[0142] D m (k,n)=a m,W W(k,n)+a m,X X(k,n)+a m,Y Y(k,n)+a m,Z Z(k,n).

[0143] Note that in the case of an HOA audio input signal, a linear combination can be similarly performed using the available HOA coefficients. In this example, the weights of the linear combination (i.e., weights a m,W 、a m,X 、a m,Y and a m,Z ) determines the resulting directional microphone signal (ie, the mth downmix signal Dm (k,n)). In the case of a FOA audio input signal, the required weights for the linear combination can be calculated as

[0144] a m,W =c m

[0145] [a m,X a m,Y a m,Z ] T =(1-c m ) m

[0146] in

[0147]

[0148] Here, c m is the so-called first-order parameter or shape parameter, and Φ m and θ m are the desired azimuth and elevation of the viewing direction of the generated m-th directional microphone signal. m = 0.5, realizing a directional microphone with cardioid directivity, c m = 1 corresponds to omnidirectional characteristics, c m = 0 corresponds to a dipole characteristic. In other words, the parameter c m Describes the general shape of the first-order directional mode.

[0149] The weight of the linear combination (e.g. a m,W 、a m,X 、a m,Y and a m,Z Or the corresponding parameter c m , Φ m and θ m ) describes the directional pattern of the corresponding directional microphone signal. This information is provided by Figure 3 The downmix parameters are represented in the encoder and sent to the decoder as part of the metadata.

[0150] Different coding strategies can be used to efficiently represent the downmix parameters in the bitstream, including quantization of the directional information or referencing a table entry by an index, where the table includes all relevant parameters.

[0151] In some embodiments, for the viewing direction Φ m and θ m And the shape parameter c m It is sufficient or more efficient to use only a limited number of presets. This obviously corresponds to also applying the weights a m,W 、a m,X 、a m,Yand a m,Z Use a limited number of presets. For example, the shape parameters can be limited to represent only three different directivity patterns: omnidirectional, cardioid, and dipole characteristics. Possible viewing directions Φ m and θ m The number of can be limited so that they only represent the left, right, front, back, top and bottom situations.

[0152] In another simpler embodiment, the shape parameters remain fixed and always correspond to a cardioid pattern, or no shape parameters are defined at all. The downmix parameters associated with the viewing direction are used to signal whether a pair of downmix channels corresponds to a left / right or front / rear channel pair configuration, so that a rendering process at a decoder can use the best downmix channels as reference signals for rendering a certain speaker channel located in the left, right or front hemisphere.

[0153] In practical applications, the parameter c can be manually defined, for example m (usually c m = 0.5). You can automatically set the viewing direction Φ m and θ m (eg, by localizing an active sound source using a prior art sound source localization method, and directing the first downmix signal towards the localized source and the second downmix signal in the opposite direction).

[0154] Note that, similar to the previous embodiments, the downmix parameters may be time-frequency dependent, i.e., a different downmix configuration may be used for each time and frequency (e.g., when the downmix signal is directed according to the direction of active sources located independently in each frequency band). For example, localization may be performed by exploiting the information contained in the time-frequency dependent spatial parameters [Thiergart09].

[0155] exist Figure 4 In the "spatial audio synthesis" stage in the decoder, the transmitted spatial parameters (such as DOA and diffuseness of the sound) and the downmixed audio channels D as described above are used. m The computation of the decoder output signal (FOA / HOA output, MC output or object output) for (k,n) has to be adapted to the actually used downmix configuration, which is specified by the downmix metadata.

[0156] For example, when generating a speaker output channel (MC output), the reference signal P ref,j The calculation of (k,n) must be adapted to the actual downmix configuration used. More specifically, the reference signal P for the jth loudspeaker ref,jThe most appropriate choice of (k,n) depends on the directional properties of the downmix signal (e.g., its viewing direction) and the location of the jth loudspeaker. For example, if the downmix metadata indicates that the downmix signal corresponds to two cardioid microphone signals pointing to the left and right, respectively, the reference signal for the loudspeaker located in the left hemisphere should mainly or exclusively use the cardioid downmix signal pointing to the left as the reference signal P ref,j (k, n). The centrally located loudspeaker may instead use a linear combination of the two downmix signals (e.g., the sum of the two downmix signals). On the other hand, if the downmix signals correspond to two cardioid microphone signals pointing forward and backward, respectively, the reference signal for the loudspeaker located in the front hemisphere should mainly or solely use the cardioid signal pointing forward as the reference signal P ref,j (k,n).

[0157] When Figure 4 When generating FOA or HOA output in the decoder, the reference signal P ref,j The calculation of (k,n) must also be adapted to the downmix configuration actually used, which is described by the downmix metadata. For example, if the downmix metadata indicates that the downmix signal corresponds to two cardioid microphone signals pointing to the left and right, respectively, then the reference signal P for synthesizing the first FOA component (omnidirectional component) is ref,1 (k,n) can be calculated as the sum of two cardioid downmix signals, that is,

[0158] P ref,1 (k,n)=D1(k,n)+D2(k,n).

[0159] In fact, it is well known that the sum of cardioid signals with opposite viewing directions will produce an omnidirectional signal. In this case, P ref,1 (k,n) directly produces the first component of the desired FOA or HOA output signal, i.e. this component does not require further spatial sound synthesis. Similarly, the third FOA component (the dipole component in the y direction) can be calculated as the difference of the two cardioid downmix signals, i.e.,

[0160] P ref,3 (k,n)=D1(k,n)-D2(k,n).

[0161] In fact, it is well known that the difference of a cardioid signal with opposite viewing directions results in a dipole signal. In this case, P ref,3(k,n) directly produces the third component of the desired FOA or HOA output signal, i.e. this component does not require further spatial sound synthesis. All remaining FOA or HOA components can be synthesized based on the omnidirectional reference signal containing audio information from all directions. This means that, in this example, the sum of the two downmix signals is used to synthesize the remaining FOA or HOA components. If the downmix metadata indicates different directivities of the two audio downmix signals, the reference signal P can be adjusted accordingly ref,j (k,n) is calculated. For example, if the two cardioid audio downmix signals are directed to the front and rear (instead of the left and right), the difference of the two downmix signals can be used to generate the second FOA component (the dipole component in the x direction) instead of the third FOA component. In general, as shown in the example above, the optimal reference signal P can be found by a linear combination of the received downmix audio signals. ref,j (k,n), that is,

[0162] P ref,j (k,n)=A 1,j D1(k,n)+A 2,j D2(k,n)

[0163] The weight of the linear combination is A 1,j and A 2,j depends on the downmix metadata, ie on the transmission channel configuration and on the considered j-th reference signal (eg when rendering to the j-th loudspeaker).

[0164] Note that the use of spatial metadata to synthesize FOA or HOA components from omnidirectional components is described, for example, in [Thiergart17].

[0165] In general, it is important to note that if the spatial audio synthesis uses the wrong downmix signal as reference signal for rendering, a significant degradation in spatial audio quality must be expected. For example, if a downmix signal corresponding to a cardioid microphone pointing to the left is used to generate output channel signals for a loudspeaker located in the right hemisphere, signal components originating from the left hemisphere of the input sound field will be directed mainly to the right hemisphere of the reproduction system, resulting in an incorrect spatial image at the output.

[0166] Flexible downmixing of parametric spatial audio input

[0167] In the present embodiment, the input of the encoder corresponds to a so-called parametric spatial audio input signal comprising an audio signal of two or more microphones with an arbitrary array configuration together with the spatial parameters of the spatial sound (e.g. DOA and diffuseness).

[0168] exist Figure 5The encoder of the present embodiment is depicted in . The microphone array signal is used to generate one or more audio downmix signals in the "Downmix Generation" block. The downmix parameters describing the transmission channel configuration (e.g. how the downmix signals are calculated or some of their properties) together with the spatial parameters represent encoder metadata, which is encoded in the "Metadata Encoder" block. Note that (in contrast to the previous embodiments) the parameter spatial audio input generally does not require a spatial audio analysis step, since the spatial parameters are already provided as input to the encoder. However, note that the spatial parameters of the parameter spatial audio input signal and the spatial parameters included in the bitstream for transmission generated by the spatial audio encoder are not necessarily the same. In this case, transcoding or mapping of the input spatial parameters and the parameters for transmission must be performed at the encoder. The downmix audio signal is encoded in the "Core Encoder" block, for example using an EVS-based audio codec. The encoded audio downmix signal and the encoded metadata form a bitstream that is sent to the decoder. For the decoder, Figure 4 The same block scheme in applies to the previous embodiments.

[0169] In the following, it is described how an audio downmix signal and corresponding downmix metadata may be generated.

[0170] In a first example, an audio downmix signal is generated by selecting a subset of available input microphone signals. The selection can be done manually (e.g., based on a preset) or automatically. For example, if a microphone signal of a uniform circular array of M spaced omnidirectional microphones is used as an input to a spatial audio encoder and is transmitted using two audio downmix transmission channels, the manual selection may include, for example, selecting a pair of signals corresponding to the microphones at the front and rear of the array, or a pair of signals corresponding to the microphones at the left and rear of the array. When synthesizing spatial sound at the decoder, selecting the front and rear microphones as downmix signals can well distinguish the sound in front from the sound from the back. Similarly, when rendering spatial sound at the decoder end, selecting the left and right microphones can well distinguish the spatial sound along the y-axis. For example, if the recorded sound source is located on the left side of the microphone array, there is a difference in the arrival time of the source signal at the left and right microphones, respectively. In other words, the signal first arrives at the left microphone and then arrives at the right microphone. Therefore, during the rendering process at the decoder, it is also important to render the loudspeakers located in the left hemisphere using the downmix signal associated with the left microphone signal, and similarly to render the loudspeakers located in the right hemisphere using the downmix signal associated with the right microphone signal. Otherwise, the time differences respectively included in the left downmix signal and the right downmix signal will be directed to the loudspeakers in an incorrect manner, and the resulting perceptual cues caused by the loudspeaker signals will be incorrect, i.e. the spatial audio image perceived by the listener will also be incorrect. Similarly, it is important to be able to distinguish at the decoder the downmix channels corresponding to front and back or top and bottom, in order to achieve an optimal rendering quality.

[0171] The appropriate microphone signal can be selected by considering the Cartesian plane that contains most of the sound energy or is expected to contain the most relevant sound energy. In order to perform an automatic selection, for example, prior art sound source localization can be performed and then two microphones closest to the axis corresponding to the direction of the sound source can be selected. A similar concept can be applied, for example, if the microphone array consists of M coincident directional microphones (such as cardioid directional microphones) instead of spaced omnidirectional microphones. In this case, two directional microphones can be selected in the direction of the Cartesian axis that contains (or is expected to contain) most of the sound energy and in the opposite direction.

[0172] In this first example, the downmix metadata contains relevant information about the selected microphone. This information may contain, for example, microphone positions (e.g., according to absolute or relative coordinates in a Cartesian coordinate system) and / or inter-microphone distances and / or orientations (e.g., according to coordinates in a polar coordinate system, i.e., according to an azimuth angle Φ) of the selected microphone. m and elevation angle and Θ m ). In addition, the downmix metadata may include information about the directivity pattern of the selected microphone, for example, by using the first-order parameters c described previouslym .

[0173] On the decoder side ( Figure 4 ), the downmix metadata is used in the "Spatial Audio Synthesis" block to obtain the best rendering quality. For example, for the loudspeaker output (MC output), when the downmix metadata indicates that two omnidirectional microphones at two specific positions are sent as downmix signals, the reference signal P from which the loudspeaker signal is generated as described above can be selected. ref,j (k, n) corresponds to the downmix signal with the smallest distance to the jth speaker position. Similarly, if the downmix metadata indicates that a signal with a viewing direction {Φ m ,Θ m}, you can select P ref,j (k,n) to correspond to the downmix signal with the closest viewing direction towards the loudspeaker position. Alternatively, as explained in the second embodiment, a linear combination of the transmitted coincident directional downmix signals may be performed.

[0174] When generating the FOA / HOA output at the decoder, if the downmix metadata indicates that spaced omnidirectional microphones have been sent, a single downmix signal can be (arbitrarily) selected to generate the direct sound for all FOA / HOA components. In fact, due to the omnidirectional nature, each omnidirectional microphone contains the same information about the direct sound to be reproduced. However, in order to generate the diffuse sound reference signal All transmitted omni-downmix signals may be considered. In fact, if the sound field is diffuse, the spaced-apart omni-downmix signals will be partially decorrelated, so that mutually uncorrelated reference signals are generated. Less decorrelation is required. Uncorrelated reference signals can be generated from the transmitted downmix audio signal by using, for example, the covariance-based rendering method proposed in [Vilkamo13].

[0175] It is well known that the correlation between the signals of two microphones in a diffuse sound field depends strongly on the distance between the microphones: the greater the distance between the microphones, the less correlated the signals recorded in the diffuse sound field [Laitinen11]. Information related to the microphone distances included in the downmix parameters can be used at the decoder to determine the extent to which the downmix channels must be synthetically decorrelated in order to be suitable for rendering the diffuse sound component. In case the downmix signal is already sufficiently decorrelated due to a sufficiently large microphone spacing, even the artificial decorrelation can be discarded and any decorrelation-related artifacts can be avoided.

[0176] When the downmix metadata indicates, for example, if coincident directional microphone signals have been sent as downmix signals, a reference signal P for FOA / HOA output may be generated as explained in the second embodiment. ref,j(k,n).

[0177] Note that instead of selecting a subset of microphones as the downmix audio signal in the encoder, all available microphone input signals (e.g., two or more) may be selected as the downmix audio signal. In this case, the downmix metadata describes the entire microphone array configuration, e.g., in terms of Cartesian microphone positions, microphone viewing directions Φ in a polar coordinate system m and θ m , or according to the first-order parameter c m The directivity of the microphone.

[0178] In a second example, a downmix audio signal is generated in the encoder in a "downmix generation" block using a linear combination of the input microphone signals (eg using spatial filtering (beamforming)). In this case, the downmix signal D m (k,n) can be calculated as

[0179]

[0180] Here, x(k,n) is a vector containing all the input microphone signals, and are the weights for the linear combination of the mth audio downmix signal, i.e., the weights of the spatial filter or beamformer. There are various methods to compute the spatial filter or beamformer in an optimal way [Veen88]. In many cases, the viewing direction {Φ m ,Θ m}, the beamformer points in that direction. The beamformer weights can then be calculated, for example, as a delay-sum beamformer or MVDR beamformer [Veen88]. In this embodiment, a beamformer viewing direction {Φ m ,Θ m This can be done manually (e.g. based on a preset) or automatically in the same way as described in the second embodiment. The viewing directions {Φ m ,Θ m} can then be expressed as being transferred to Figure 4 Downmix metadata for the decoder in .

[0181] Another example is particularly suitable for using the loudspeaker output at the decoder (MC output). In this case, the downmix signal D m (k,n) is used as the beamformer's viewing direction closest to the loudspeaker's direction P ref,j (k,n). The desired beamformer look direction is described by the downmix metadata.

[0182] Please note that in all examples, the transmission channel configuration (ie, downmix parameters) may be adjusted in terms of time and frequency, for example, based on spatial parameters, similar to the previous embodiments.

[0183] Subsequently, further embodiments of the invention or previously described embodiments are discussed with respect to the same or additional or further aspects.

[0184] Preferably, Figure 6 The transmission representation generator 600 includes Figure 8a One or several of the features shown in . In particular, an energy position determiner 606 for controlling block 602 is provided. Block 602 may include a selector for selecting from the Ambisonics coefficient signals when the input is a FOA or HOA signal. Alternatively or additionally, the energy position determiner 606 controls a combiner for combining the Ambisonics coefficient signals. Additionally or alternatively, the selection from a multi-channel representation or from a microphone signal is done. In this case, the input has a microphone signal or a multi-channel representation instead of FOA or HOA data. Additionally or alternatively, as Figure 8a As shown in 602 in , channel combining or combining of microphone signals is performed. For the following two alternatives, a multi-channel representation or microphone signal is input.

[0185] The transmission data generated by one or more blocks 602 are input to Figure 6 The transmission representation generator 600 is included in the transmission metadata generator 605 to generate the (encoded) transmission metadata 610.

[0186] Any of blocks 602 generates a preferably uncoded transmission representation 614 which is then used by a processor such as Figure 3 or Figure 5 The core encoder 603 is shown to further encode.

[0187] It is outlined that a practical implementation of the transmission representation generator 600 may only include Figure 8a One of the blocks 602 in or Figure 8a In the latter case, the transport metadata generator 605 is configured to additionally include in the transport metadata 610 a further transport metadata item indicating for which (time and / or frequency) portion of the spatial audio representation any of the alternatives indicated at item 602 has been employed. Thus, Figure 8a Situations are shown in which only one alternative 602 is active or in which two or more are active and a signal-dependent switching can be performed between different alternatives for transmission representation generation or downmixing and corresponding transmission metadata.

[0188] Figure 8b It shows that it can be Figure 6 The transmission representation generator 600 generates and can be used by Figure 7 The transmission metadata alternatives include selection information for indicating which subset of a set of audio input data components has been selected as metadata for the transmission representation. For example, examples are that only two or three components out of, for example, four FOA components have been selected. Alternatively, the selection information may indicate which microphone signals of a microphone signal array have been selected. Figure 8b Another alternative is combination information indicating how a certain audio representation input component or signal is combined. Certain combination information may refer to weights for linear combination, or which channels have been combined, for example in the case of equal or predefined weights. Further information refers to sector or hemispheric information associated with a certain transmission signal. The sector of the hemispheric information may refer to a left sector or a right sector or a front sector or a rear sector for the listening position, or alternatively, to a sector smaller than a 180° sector.

[0189] Further embodiments relate to transmission metadata indicating shape parameters relating to, for example, the shape of a particular physical or virtual microphone directivity that generates a corresponding transmission representation signal. The shape parameter may indicate an omnidirectional microphone signal shape or a cardioid microphone signal shape or a dipole microphone signal shape or any other relevant shape. Further transmission metadata alternatives relate to microphone positions, microphone orientations, distances between microphones or directivity patterns of microphones that have generated or recorded the transmission representation signal included in the (encoded) transmission representation 614. Further embodiments relate to a viewing direction or multiple viewing directions of a signal included in the transmission representation, or information about beamforming weights or beamformer directions, or alternatively or additionally, relating to whether the included microphone signal is an omnidirectional microphone signal, a cardioid microphone signal or other signal. Very little transmission metadata side information (relative to bit rate) may be generated by simply including a single flag indicating whether the transmission signal is from an omnidirectional microphone or from a microphone signal of any other microphone than an omnidirectional microphone.

[0190] Figure 8c It shows a preferred implementation of the transmission metadata generator 605. In particular, for digital transmission metadata, the transmission metadata generator comprises a transmission metadata quantizer 605a or 622 and a subsequently connected transmission metadata entropy encoder 605b. Figure 8c The process shown in can also be applied to parameter metadata, in particular also to spatial parameters.

[0191] Figure 9a Shows Figure 7The preferred implementation of the spatial audio synthesizer 750 in FIG. The spatial audio synthesizer 750 includes a transport metadata parser for interpreting the (decoded) transport metadata 710. The output data from block 752 is introduced into a combiner / selector / reference signal generator 760, which additionally receives the Figure 7 The transmit signal 711 included in the transmit representation obtained by the input interface 700 of the synthesizer. Based on the transmit metadata, the combiner / selector / reference signal generator generates one or more reference signals and forwards these reference signals to the component signal calculator 770, which calculates the components of the synthesized spatial audio representation, such as the common components of the multi-channel output, the Ambisonics components of the FOA or HOA output, the left and right channels of the binaural representation or the audio object components, where the audio object components are mono or stereo object signals.

[0192] Figure 9b The n transmission signals T1, T2, T n The encoded audio signal is composed and additionally consists of transport metadata 610 and optional spatial parameters 612 . Figure 9b The order of different data blocks and the size of a data block relative to another data block are only schematically shown.

[0193] Fig.9c An overview table of the process of the combiner / selector / reference signal generator 760 for specific transport metadata, specific transport representation and specific speaker setup is shown. In particular, Fig.9c In an embodiment of the invention, the transmission representation for example comprises a left transmission signal (or a front transmission signal or an omnidirectional or cardioid signal) and the transmission representation additionally comprises a second transmission signal T2 as a right transmission signal (or a rear transmission signal, an omnidirectional transmission signal or a cardioid transmission signal). In the left / right case, the reference signal of the left loudspeaker A is selected as the first transmission signal T1, while the reference signal of the right loudspeaker is selected as the transmission signal T2. For left surround and right surround, the left signal and the right signal are selected for the respective channels as outlined in Table 771. For the center channel, the sum of the left transmission signal T1 and the right transmission signal T2 is selected as the reference signal for the center channel component of the synthesized spatial audio representation.

[0194] exist Fig.9c , another selection is shown when the first transmission signal T1 is the front transmission signal and the second transmission signal T2 is the right transmission signal. Then, the first transmission signal T1 is selected for left, right, center, and the second transmission signal T2 is selected for left surround and right surround.

[0195] Figure 9d Shows Figure 7Another preferred implementation of a spatial audio synthesizer. In block 910, transmit or downmix data is calculated for a first-order Ambisonics or higher-order Ambisonics selection. For example, Figure 9d Four different selection alternatives are shown in , where in the fourth alternative only the two transmission signals T1 , T2 are selected instead of the third component, which in the other alternatives is the omnidirectional component.

[0196] Reference signals for the (virtual) channels are determined based on the transmitted downmix data and a fallback process is used for the missing components, i.e. for the Figure 9d Then, at block 912, the channel signals are generated using the directional parameters received or derived from the transmission data. Thus, directional or spatial parameters may be additionally received, such as Figure 7 As shown in 712 in, or it can be derived from the transmission representation by signal analysis of the transmission representation signal.

[0197] In an alternative implementation, selecting the component as a FOA component is performed as indicated in block 913, and using Figure 9d The spatial basis function response shown in item 914 in is used to perform the calculation of the missing components. Fig.10 A specific process using spatial basis function responses is shown at block 410 in FIG. Fig.10 , block 826 provides the average response of the diffuse portion, while Fig.10 Block 410 in provides a specific response for each mode m and order l of the direct signal portion.

[0198] Fig.9e Another table indicating specific transmission metadata is shown, which specifically includes shape parameters, or includes viewing direction in addition to or as an alternative to shape parameters. The shape parameters may include a shape factor c of 1, 0.5 or 0. m Factor c M =1 indicates an omnidirectional shape of the microphone recording characteristic, while a factor of 0.5 indicates a cardioid shape and a value of 0 indicates a dipole shape.

[0199] In addition, different viewing directions may include left, right, front, back, up, down, by azimuth and an elevation angle θ, or, alternatively, comprises short metadata consisting of an indication that the signal pairs in the transmission representation comprise left / right pairs or front / rear pairs.

[0200] exist Figure 9f , another implementation of a spatial audio synthesizer is shown, wherein in block 910, the transmission metadata is read, for example, by Figure 7The input interface 700 or the input port of the spatial audio synthesizer 750 is completed. In block 950, the reference signal determines the transmission metadata suitable for reading, for example, performed by block 760. Then, in block 916, the reference signal obtained via block 915 and the optionally transmitted parameter data 712 (if available) are used to calculate the multi-channel, FOA / HOA, object or binaural output, in particular specific components of these types of data output.

[0201] Figure 9g Another implementation of the combiner / selector / reference signal generator 760 is shown. For example, when the transmission metadata shows that the first transmission signal T1 is a left cardioid signal and the second transmission signal T2 is a right cardioid signal, in block 920, the omnidirectional signal is calculated by adding T1 and T2. As outlined in block 921, the dipole signal Y is calculated by obtaining the difference between T1 and T2 or the difference between T2 and T1. Then, in block 922, the remaining components are synthesized using the omnidirectional signal as a reference. The omnidirectional signal used as a reference in block 922 is preferably the output of block 920. Additionally, as outlined in item 712, optional spatial parameters may also be used to synthesize the remaining components, such as FOA or HOA components.

[0202] Figure 9h Another implementation of different alternatives of the process that can be performed by the spatial audio synthesizer or combiner / selector / reference signal generator 760 when two or more microphone signals are received as a transmission representation and associated transmission metadata as outlined in block 930 is shown. As outlined in block 931, the following operations can be performed: the transmission signal with the smallest distance from, for example, a certain speaker position is selected as the reference signal for a certain signal component. Another alternative shown in block 932 includes selecting the following microphone signal as the reference signal for a specific speaker: having the closest viewing direction, or, for example, having the closest beamformer or error position relative to a certain speaker or virtual sound source (such as left / right in a binaural representation). Another alternative shown in block 933 is to select an arbitrary transmission signal as the reference signal for all direct sound components, such as for the calculation of FOA or HOA components or the calculation of speaker signals. Another alternative shown at 934 involves using all available transmission signals (such as omnidirectional signals) to calculate a diffuse sound reference signal. A further alternative involves setting or limiting the correlation quantities used to calculate the component signals based on microphone distances included in the transmission metadata.

[0203] For the purpose of performing one or several of the alternatives 931 to 935, several associated transmission metadata are useful, which are Figure 9h The right side of is indicated as including the following items: microphone position of the selective microphone, distance between microphones, such as cM The microphone orientation or directivity pattern, array description, beamforming factor w m Or for example, for each transmitted channel, the actual direction of arrival or with azimuth and the elevation angle θ of the sound direction.

[0204] Fig.10 A preferred implementation of a low-order or mid-order component generator for a direct / diffuse process is shown. In particular, the low-order or mid-order component generator comprises a reference signal generator 821, which is preferably controlled by the transmission metadata and receives an input signal and generates a reference signal by copying or adopting it as is when the input signal is a mono signal, or by deriving the reference signal from the input signal using calculations as previously discussed or as shown in WO 2017 / 157803 A1, the entire teaching of which is incorporated herein by reference in its entirety.

[0205] also, Fig.10 The directional gain calculator 410 is shown, which is configured to calculate the directional gain G according to the specific DOA information (Φ, θ) and according to the specific mode number m and the specific order l. l m In a preferred embodiment, processing is performed in the time / frequency domain for each individual tile referenced by k,n, and a directional gain is calculated for each such time / frequency tile. A weighter 820 receives the reference signal and the diffusion data for a particular time / frequency tile, and the result of the weighter 820 is the direct portion. The diffuse portion is generated by processing performed by a decorrelation filter 823 and a subsequent weighter 824, which receives the diffuse value Ψ for a particular time frame and frequency bin, and in particular receives a response to D generated by an average response provider 826. l The average response provider indicates the average response for a particular mode m and order l, the average response provider receiving as input the desired mode m and the desired order l.

[0206] The result of the weighting device 824 is the diffuse part, and the diffuse part is added to the direct part by the adder 825 to obtain a specific mid-order sound field component of a specific m-mode and a specific l-order. Figure 6 The diffusion compensation gain discussed is applied only to the diffuse part generated by block 823. This can advantageously be done within a process performed by a (diffusion) weighter. Thus, only the diffuse part of the signal is enhanced in order to compensate for the diffuse energy loss caused by not receiving the fully synthesized higher components, such as Fig.10 as shown in .

[0207] For high-order component generators, Fig.11Only the direct part is generated. Basically, with respect to the direct branch, the high-order component generator is implemented in the same way as the low-order or mid-order component generator, but does not include blocks 823, 824, 825 and 826. Therefore, the high-order component generator only includes a (direct) weighter 822, which receives input data from the directional gain calculator 410 and receives a reference signal from the reference signal generator 821. Preferably, only a single reference signal is generated for the high-order component generator and the low-order or mid-order component generator. However, depending on the circumstances, the two blocks may also have separate reference signal generators. However, it is preferred to have only a single reference signal generator. Therefore, the processing performed by the high-order component generator is very efficient because only the processing with a specific directional gain G is to be performed. l m As a result, high-order sound field components can be generated very efficiently and quickly, and any errors caused by the fact that diffuse components are not generated or not used in the output signal can be easily compensated by enhancing only the diffuse parts of low-order sound field components or preferably mid-order sound field components. Fig.11 The process shown in can also be used to generate low-order or mid-order components.

[0208] therefore, Fig.10 shows the generation of low-order or mid-order sound field components with a diffuse part, while Fig.11 A process for calculating higher order sound field components, or components that generally do not require or receive any diffuse portion, is shown.

[0209] However, when generating sound field components, especially for FOA or HOA representations, it is possible to apply Fig.10 process or without diffusion part Fig.11 The reference signal generators 821 and 760 are Fig.10 and Fig.11 In both processes, the transmission metadata is controlled. In addition, the weighter 822 is not only composed of the spatial basis function response G l n , but preferably also by spatial parameters such as diffuseness parameters 712, 722. Furthermore, in a preferred embodiment, the weighter 824 of the diffuse part is also controlled by the transmission metadata, in particular by the microphone distance. Fig.10The specific relationship between the microphone distance D and the weighting factor W is shown in the schematic diagram in . A large distance D results in a small weighting factor, while a small distance results in a large weighting factor. Therefore, when the transmission signal representation includes two microphone signals that are far apart from each other, it can be assumed that the two microphone signals are already very de-correlated, so that the output of the decorrelation filter can be weighted with a weighting factor close to zero, so that the signal finally input to the adder 825 is very small compared to the signal input to the adder from the direct weighter 822. In extreme cases, the correlation branch can even be turned off, which can be achieved, for example, by setting the weight W=0. Of course, there are other methods to turn off the diffusion branch by using switches calculated using threshold operations, etc.

[0210] certainly, Fig.10 The component generation shown may be performed by controlling the reference signal generator 821 , 760 only by transmission metadata without control of the weighter 804 , or alternatively, by controlling only the weighter 804 without any reference signal generation control block 821 , 760 .

[0211] Fig.11 shows that the propagation branch is missing and therefore no Fig.10 Any control situation of the diffusion weighter 824.

[0212] Fig.10 and Fig.12 A specific diffuse signal generator 830 is shown including a decorrelation filter 823 and a weighter 824. Of course, the order of signal processing between the weighter 824 and the decorrelation filter 823 can be exchanged so that the weighting of the generated reference signal or the output of the reference signal generator 821, 760 is performed before the signal is input to the decorrelation filter 823.

[0213] Although Fig.10 The generation of low-order or mid-order sound field components of a sound field component representation such as FOA or HOA, i.e. a representation with spherical or cylindrical component signals, but Fig.12 An alternative or general implementation for the calculation of loudspeaker component signals or objects is shown. In particular, for the generation and calculation of loudspeaker signals / objects, a corresponding Figure 9a The reference signal generator 821, 760 of block 760. In addition, for the direct branch, Figure 9a The component signal calculator 770 shown in FIG. 8 includes a weighter 822, and for the diffusion branch, the diffusion signal generator 830 includes a decorrelation filter 823 and a weighter 824. In addition, Figure 9a The component signal calculator 770 further includes executing the direct signal P dir and the diffusion signal P diffThe output of the adder is a (virtual) loudspeaker signal or an object signal or a binaural signal, as shown by example reference numerals 755, 756. In particular, the reference signal calculators 821, 760 are controlled by the transmission metadata 710, and the diffusion weighter 824 can also be controlled by the transmission metadata 710. Typically, the component signal calculator calculates the direct part, for example using a translation gain such as a VBAP (virtual base amplitude translation) gain. The gain is derived from the arrival direction information given by the elevation angle θ. This results in the direct portion P dir .

[0214] In addition, the reference signal calculator P ref The generated reference signal is input to a decorrelation filter 823 to obtain a decorrelated reference signal, which is then weighted, preferably using a diffuseness parameter and preferably also using the microphone distance obtained from the transmission metadata 710. The output of the weighter 824 is the diffuseness component P diff , and the adder 825 adds the direct component and the diffuse component to obtain a certain loudspeaker signal or object signal or binaural channel for the corresponding representation. Specifically, when calculating the virtual loudspeaker signal, it can be as follows Fig.9c The process performed by the reference signal calculator 821, 760 in response to the transmission metadata is shown in . Alternatively, the reference signal may be generated as a channel directed to a specific speaker from a defined listening position, and such calculation of the reference signal may be performed using a linear combination of the signals included in the transmission representation.

[0215] Preferred embodiments of the present invention as a list

[0216] Based on FOA input

[0217] Spatial audio scene encoder

[0218] o receiving a spatial audio input signal (e.g., a FOA component) representing a spatial audio scene

[0219] o Generate or receive spatial audio parameters including at least one directional parameter

[0220] o Generate a downmix audio signal based on the received audio input signal (option: also use spatial audio parameters for adaptive downmix generation)

[0221] o generating downmix parameters (eg downmix coefficients or directionality patterns) describing directional properties of the downmix signal

[0222] o Encoding the downmix signal, the spatial audio parameters and the downmix parameters.

[0223] Spatial audio scene decoder

[0224] o receiving an encoded spatial audio scene comprising a downmix audio signal, spatial audio parameters and downmix parameters

[0225] o Decoding of downmix audio signals, spatial audio parameters and downmix / transmit channel parameters

[0226] o a spatial audio renderer for spatially rendering the decoded representation based on the downmix audio signal, the spatial audio parameters and the downmix (position) parameters.

[0227] Input based on spaced microphone recordings and associated spatial metadata (parametric spatial audio input):

[0228] Spatial audio scene encoder

[0229] o generating or receiving at least two spatial audio input signals generated from recorded microphone signals

[0230] o Generate or receive spatial audio parameters including at least one directional parameter

[0231] o generating or receiving position parameters describing geometric or positional characteristics of a spatial audio input signal generated from recorded microphone signals (e.g., relative or absolute positions of microphones or microphone spacing)

[0232] o Encoding a spatial audio input signal or a downmix signal derived from the spatial audio input signal, the spatial audio parameters and the position parameters.

[0233] Spatial audio scene decoder

[0234] o Receiving an encoded spatial audio scene comprising at least two audio signals, spatial audio parameters and position parameters (related to positional properties of the audio signals).

[0235] οDecoding of audio signals, spatial audio parameters and position parameters

[0236] o A spatial audio renderer for spatially rendering the decoded representation based on the audio signal, the spatial audio parameters and the position parameters.

[0237] Although some aspects have been described in the context of an apparatus, it is clear that these aspects also represent a description of a corresponding method, where a block or device corresponds to a method step or a feature of a method step. Similarly, aspects described in the context of a method step also represent a description of a corresponding block or item or feature of a corresponding apparatus.

[0238] Depending on certain implementation requirements, embodiments of the present invention may be implemented in hardware or software. The implementation may be performed using a digital storage medium, such as a floppy disk, DVD, CD, ROM, PROM, EPROM, EEPROM or FLASH memory, which has electronically readable control signals stored thereon, which cooperate with a programmable computer system (or can cooperate with a programmable computer system) to perform the corresponding method.

[0239] Some embodiments according to the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.

[0240] Generally, embodiments of the present invention can be implemented as a computer program product with a program code, which is operable to perform one of the methods when the computer program product runs on a computer.The program code may, for example, be stored on a machine-readable carrier.

[0241] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier or a non-transitory storage medium.

[0242] In other words, an embodiment of the inventive method is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0243] A further embodiment of the inventive methods is, therefore, a data carrier (or a digital storage medium, or a computer-readable medium) on which is recorded the computer program for performing one of the methods described herein.

[0244] Therefore, another embodiment of the inventive method is a data stream or a sequence of signals representing the computer program for performing one of the methods described herein.The data stream or the sequence of signals may, for example, be configured to be transmitted via a data communication connection, for example via the Internet.

[0245] A further embodiment comprises a processing means, for example a computer or a programmable logic device, configured to or adapted to perform one of the methods described herein.

[0246] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0247] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, a field programmable gate array may collaborate with a microprocessor to perform one of the methods described herein. Typically, these methods are preferably performed by any hardware device.

[0248] The above embodiments are only used to illustrate the principles of the present invention. It should be understood that modifications and variations of the arrangements and details described herein will be apparent to those skilled in the art. Therefore, it is intended that the invention be limited only to the scope of the upcoming patent claims, rather than to the specific details presented by the description and explanation of the embodiments herein.

[0249] References

[0250] [Pulkki07] V. Pulkki, “Spatial Sound Reproduction with Directional Audio Coding”, J. Audio Eng. Soc., Vol. 55, No. 6, pp. 503-516; June 2007.

[0251] [Pulkki97] V. Pulkki, "Virtual Sound Source Positioning Using VectorBase Amplitude Panning," J. Audio Eng. Soc., vol. 45, no. 6, pp. 456-466; June 1997.

[0252] [Thiergart09] O. Thiergart, R. Schultz-Amling, G. Del Galdo, D. Mahne, F. Kuech, “Localization of Sound Sources in Reverberant Environments Based on Directional Audio Coding Parameters,” AES Convention 127, Paper No. 7853, October 2009.

[0253] [Thiergart17]O.Thiergart et al. WO2017157803 A1, “APPARATUS, METHOD OR COMPUTER PROGRAM FOR GENERATING A SOUND FIELD DESCRIPTION”.

[0254] [Laitinen11] M. Laitinen, F. Kuech, and V. Pulkki, “Using Spaced Microphones with Directional Audio Coding,” AES Convention 130, Paper No. 8433, May 2011.

[0255] [Vilkamo13] J. Vilkamo and V. Pulkki, “Minimization of Decorrelator Artifacts in Directional Audio Coding by Covariance Domain Rendering,” J. Audio Eng. Soc, vol. 61, no. 9, September 2013.

[0256] [Veen88] BD Van Veen and KM Buckley, “Beamforming: a versatile approach to spatial filtering,” IEEE ASSP Mag., vol. 5, no. 2, pp. 4–24, 1998.

[0257] [1] V. Pulkki, MV Laitinen, J Vilkamo, J Ahonen, T Lokki and T “Directional audio coding-perception-based reproduction of spatial sound,” International Symposium on Principles and Applications of Spatial Hearing, November 2009, Zao; Miyagi, Japan.

[0258] [2] M.V. Laitinen and V. Pulkki, “Converting 5.1 audio recordings to B-format for directional audio coding reproduction,” in 2011 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Prague, 2011, pp. 61–64.

[0259] [3] RK Furness, “Ambisonics—An overview,” AES 8th International Conference, April 1990, pp. 181–189.

[0260] [4] C. Nachbar, F. Zotter, E. Deleflie, and A. Sontacchi, “AMBIX–A Suggested Ambisonics Format,” in Proceedings of the 2011 Ambisonics Symposium.

Claims

1. An apparatus for encoding a spatial audio representation representing an audio scene to obtain an encoded audio signal, the apparatus comprising: a transport representation generator (600) for generating a transport representation (611) based on the spatial audio representation and for generating transport metadata (610) related to the generation of the transport representation (611); a parameter processor (620) for deriving spatial parameters (612) from the spatial audio representation; and an output interface (640) for multiplexing information related to the transmission representation (611), information related to the transmission metadata (610) and information related to the spatial parameters (612) to obtain the encoded audio signal, wherein the spatial audio representation is a first-order Ambisonics representation or a higher-order Ambisonics representation comprising a plurality of coefficient signals, or a multi-channel representation comprising a plurality of audio channels, wherein the transmitted representation generator (600) is configured to linearly combine two or more coefficient signals from the higher order Ambisonics representation or the first order Ambisonics representation, or to linearly combine two or more audio channels from the multi-channel representation, and wherein the transmission representation generator (600) is configured to generate the following as the transmission metadata (610): information about how the two or more coefficient signals from the higher-order Ambisonics representation or the first-order Ambisonics representation or the two or more audio channels from the multi-channel representation are linearly combined, or which coefficient signals from the two or more coefficient signals from the first-order Ambisonics representation or from the higher-order Ambisonics representation or which audio channels from the two or more audio channels from the multi-channel representation are linearly combined.

2. The apparatus according to claim 1, wherein the parameter processor (620) is configured to derive at least one of time- or frequency-dependent direction of arrival DoA data and frequency- or time-dependent diffuseness data as the spatial parameter (612) from the spatial audio representation.

3. The device according to claim 1, wherein the transmission representation generator (600) is configured to determine whether the primary acoustic energy originates from a particular sector or hemisphere, wherein the transmission representation generator (600) is configured to generate a first transmission signal based on the particular sector or hemisphere from which the primary acoustic energy originates, and to generate a second transmission signal based on a different sector or hemisphere, the different sector or hemisphere being a sector or hemisphere having an opposite orientation relative to a reference position and relative to the particular sector or hemisphere, and The transmission representation generator (600) is configured to determine the transmission metadata (610) such that the transmission metadata (610) includes information identifying the particular sector or hemisphere, or information identifying the different sectors or hemispheres. The device according to claim 3 , wherein the hemisphere is a left hemisphere or a right hemisphere or a front hemisphere or a back hemisphere.

5. The device according to claim 1, wherein the transmission representation generator (600) is configured to linearly combine the coefficient signals of the spatial audio representation such that a first resulting signal as a first transmission signal corresponds to a directional microphone signal directed toward a particular sector or hemisphere, and a second resulting signal as a second transmission signal corresponds to a directional microphone signal directed toward a different sector or hemisphere.

6. The device according to claim 1, The transmission representation generator (600) is configured to generate the transmission representation (611) and the transmission metadata (610) in a time-varying or frequency-dependent manner so that the transmission representation (611) and the transmission metadata (610) of a first frame are different from the transmission representation (611) and the transmission metadata (610) of a second frame, or so that the transmission representation (611) and the transmission metadata (610) of a first frequency band are different from the transmission representation (611) and the transmission metadata (610) of a different second frequency band.

7. The device according to claim 1, wherein the transmission representation generator (600) is configured to generate one or two transmission signals by weighted linear combinations (602) of two or more coefficient signals of the spatial audio representation, and The transmission representation generator (600) is configured to calculate the transmission metadata (610) so that the transmission metadata (610) includes information related to weights used in the weighted linear combination, or information related to azimuth and / or elevation angles as viewing directions of the generated directional microphone signal, or information related to shape parameters indicating directional characteristics of the directional microphone signal.

8. The device according to claim 1, wherein the transport representation generator (600) is configured to generate quantitative transport metadata (610), to quantize (605a) the quantitative transport metadata to obtain quantized transport metadata, and to entropy encode (605b) the quantized transport metadata, and wherein the output interface (640) is configured to include the encoded transport metadata into the encoded audio signal.

9. The device according to claim 1, wherein the transmission representation generator (600) is configured to convert the transmission metadata (610) into a table index or preset parameters, and The output interface (640) is configured to include the table index or the preset parameter in the encoded audio signal.

10. The device according to claim 1, wherein the spatial audio representation comprises at least two audio signals and the spatial parameters (612), wherein the parameter processor (620) is configured to derive the spatial parameters (612) from the spatial audio representation by extracting the spatial parameters (612) from the spatial audio representation, wherein the output interface (640) is configured to include the spatial parameters (612) or information related to processed spatial parameters derived from the spatial parameters (612) as information related to the spatial parameters in the encoded audio signal, or wherein the transmission representation generator (600) is configured to linearly combine the at least two audio signals or a subset of the at least two audio signals and to calculate the transmission metadata (610) such that the transmission metadata (610) comprises information about the linear combination of the audio signals performed to calculate the transmission representation (611) of the spatial audio representation.

11. The device according to claim 1, wherein the transmission representation generator (600) is configured to linearly combine (602) the audio signals included in the spatial audio representation using spatial filtering or beamforming, and The transmission representation generator (600) is configured to include information about a viewing direction of the transmission representation (611) or information about beamforming weights used when calculating the transmission representation (611) in the transmission metadata (610).

12. The device according to claim 1, wherein the spatial audio representation is a description of the sound field relative to a reference position, and wherein the spatial parameter (612) defines a time-varying parameter or a frequency-dependent parameter related to the direction of arrival of the sound at the reference position, or a time-varying parameter or a frequency-dependent parameter related to the diffusion of the sound field at the reference position, or The transmission representation generator (600) comprises a downmixer (601) for generating a downmix representation (614) as the transmission representation (611), the downmix representation having a second number of independent signals less than a first number of independent signals included in the spatial audio representation, wherein the downmixer (601) is configured to linearly combine the independent signals included in the spatial audio representation so as to reduce the first number of signals to the second number of signals.

13. The device according to claim 1, wherein the parameter processor (620) comprises a spatial audio analyzer (621) for deriving the spatial parameters (612) from the spatial audio representation by performing an audio signal analysis, and wherein the transport representation generator (600) is configured to generate the transport representation (611) based on a result of the spatial audio analyzer (621), or wherein the transmission representation generator (600) comprises a core encoder (603) for performing core encoding on one or more audio signals of the transmission signal of the transmission representation (611); or wherein the parameter processor (620) is configured to quantize and entropy encode (622) the spatial parameters (612), and The output interface (640) is configured to include the core-encoded transmission representation (611) in the encoded audio signal as information related to the transmission representation, or to include the entropy-encoded spatial parameters (612) in the encoded audio signal as information related to the spatial parameters (612).

14. An apparatus for decoding an encoded audio signal, comprising: An input interface (700) for receiving an encoded audio signal, the encoded audio signal comprising in multiplexed form information about a transmission representation (711), information about spatial parameters (712) and information about transmission metadata (710), the transmission representation (711) comprising a plurality of transmission signals; and a spatial audio synthesizer (750) for synthesizing a spatial audio representation using information about the transmission representation (711) comprising the plurality of transmission signals, the information about spatial parameters (712) and the information about transmission metadata (710), wherein the transmission metadata (710) indicates a first transmission signal of the plurality of transmission signals as relating to a first sector or hemisphere relative to a reference position of the spatial audio representation, and indicates a second transmission signal of the plurality of transmission signals as relating to a second, different sector or hemisphere relative to the reference position of the spatial audio representation, Wherein the spatial audio synthesizer (750) is configured to: generating (915) a component signal of the spatial audio representation associated with the first sector or hemisphere using the first transmission signal and not using the second transmission signal, or generating (915) another component signal of the spatial audio representation associated with the second sector or hemisphere using the second transmission signal instead of the first transmission signal, or generating (915) another component signal of the spatial audio representation associated with the first sector or hemisphere using a first linear combination of the first transmitted signal and the second transmitted signal, or generating (915) another component signal of the spatial audio representation associated with the second sector or hemisphere using a second linear combination of the first transmitted signal and the second transmitted signal, wherein the first linear combination is more strongly influenced by the first transmitted signal than the second linear combination, or wherein the second linear combination is more strongly influenced by the second transmitted signal than the first linear combination, or wherein the transmission metadata (710) indicates information related to weighting coefficients or viewing directions of a plurality of transmission signals of the transmission representation (711), and wherein the spatial audio synthesizer (750) is configured to use the information related to the viewing directions or the weighting coefficients and to calculate (932) different first-order Ambisonics components of the spatial audio representation using the transmission signals.

15. The apparatus according to claim 14, wherein the input interface (700) is configured to receive at least one of time- or frequency-dependent direction of arrival DoA data and frequency- or time-dependent dispersivity data as the information related to the spatial parameters (712), and wherein the spatial audio synthesizer (750) is configured to synthesize the spatial audio representation using additionally the at least one of the time or frequency dependent direction of arrival DoA data and the frequency or time dependent diffuseness data.

16. The apparatus of claim 14, wherein the spatial audio synthesizer (750) comprises: a core decoder (751) for core decoding two or more encoded transmission signals representing the information related to the transmission representation (711) to obtain two or more decoded transmission signals, or wherein the spatial audio synthesizer (750) is configured to compute a first order Ambisonics or higher order Ambisonics representation (754) or a multi-channel signal (755) or an object representation (756) or a binaural representation of the spatial audio representation, or The spatial audio synthesizer (750) includes a metadata decoder (752) for decoding the information related to the transmission metadata (710) to derive decoded transmission metadata (720), and for decoding the information related to the spatial parameters (712) to obtain decoded spatial parameters (722).

17. The device according to claim 14, wherein the spatial audio representation comprises a plurality of component signals, wherein the spatial audio synthesizer (750) is configured to determine (760) a reference signal for a component signal of the spatial audio representation using the information related to the transmission representation (711) and the information related to the transmission metadata (710), and Component signals of the spatial audio representation are calculated (770) using the reference signal and the information about the spatial parameters (712), or component signals of the spatial audio representation are calculated (770) using the reference signal.

18. The device according to claim 14, wherein the transmission metadata (710) comprises information relating to: a directional characteristic associated with the transmission signal of the transmission representation (711), wherein the spatial audio synthesizer (750) is configured to calculate (911) a virtual microphone signal using a first order Ambisonics signal or a higher order Ambisonics signal, a speaker position and the transmission metadata (710), or wherein the spatial audio synthesizer (750) is configured to determine (911) a directional characteristic of the transmission signal using the transmission metadata (710), and to determine the first-order Ambisonics component or the higher-order Ambisonics component (754) from the transmission signal consistent with the determined directional characteristic of the transmission signal, and Another first-order Ambisonics component or another higher-order Ambisonics component (754) that is independent of the directional characteristic of the transmission signal is determined (911) according to a back-off process.

19. The device according to claim 14, wherein the transmission metadata (710) includes information related to: a first viewing direction associated with the first transmission signal, and a second viewing direction associated with the second transmission signal, The spatial audio synthesizer (750) is configured to select (771) a reference signal for calculating the component signal or the further component signal of the spatial audio representation based on the transmission metadata (710) and the position of the loudspeaker associated with the component signal or the further component signal of the spatial audio representation.

20. The device according to claim 19, wherein the first viewing direction indicates a left hemisphere or a front hemisphere, wherein the second viewing direction indicates a right hemisphere or a rear hemisphere, in, For calculation of the component signal or the further component signal for a loudspeaker in the left hemisphere, the first transmission signal is used (771) instead of the second transmission signal, or wherein for calculation of the component signal or the further component signal for a loudspeaker signal in the right hemisphere, the second transmission signal is used (771) instead of the first transmission signal, or wherein for the calculation of the component signal or the further component signal for the loudspeaker in the front hemisphere, the first transmission signal is used (771) instead of the second transmission signal, or wherein for the calculation of the loudspeaker in the rear hemisphere, the second transmission signal is used (771) instead of the first transmission signal, or wherein for the calculation of the component signal or the further component signal for a speaker in a central area, a linear combination of (771) the first transmission signal and the second transmission signal is used, or wherein for the calculation of the component signal or the further component signal for a speaker signal associated with a speaker in an area between a front hemisphere and a rear hemisphere, a linear combination of (771) the first transmission signal and the second transmission signal is used.

21. The device according to claim 14, wherein the information related to the transmission metadata (710) indicates a left direction of the first transmission signal as a first viewing direction, and indicates a right viewing direction of the second transmission signal as a second viewing direction, wherein the spatial audio synthesizer (750) is configured to calculate a first Ambisonics component by adding (920) the first transmission signal and the second transmission signal, or to calculate a second Ambisonics component by subtracting (921) the first transmission signal and the second transmission signal, or wherein another Ambisonics component is calculated (922) using the sum of the first transmission signal and the second transmission signal.

22. The device according to claim 14, wherein the transmission metadata (710) indicates a forward viewing direction for the first transmission signal and a rearward viewing direction for the second transmission signal, wherein the spatial audio synthesizer (750) is configured to calculate a first-order Ambisonics component in the x-direction by performing a calculation of a difference (921) between the first transmission signal and the second transmission signal, and to calculate an omnidirectional first-order Ambisonics component using an addition (920) of the first transmission signal and the second transmission signal, and Another first order Ambisonics component is calculated (922) using the sum of the first transmission signal and the second transmission signal.

23. The device according to claim 14, The spatial audio synthesizer (750) is configured to additionally use the spatial parameters (712) to calculate (932) the different first-order Ambisonics components of the spatial audio representation.

24. The device according to claim 14, wherein the transmission metadata (710) comprises information about a transmission signal derived from a microphone signal at two different locations or under different viewing directions, The spatial audio synthesizer (750) is configured to select (931) a reference signal positioned closest to a speaker position, or to select (932) a reference signal having a closest viewing direction relative to a direction from a reference position of the spatial audio representation to the speaker position, or to perform a linear combination (771) of the transmission signals to determine a reference signal for a speaker placed between two viewing directions indicated by the transmission metadata (710).

25. The device according to claim 14, wherein the transmission metadata (710) includes information related to: distances between microphone locations associated with the transmission signal, wherein the spatial audio synthesizer (750) comprises a diffuse signal generator (830, 823, 824), and wherein the diffuse signal generator (830, 823, 824) is configured to use information related to the distance to control an amount of a decorrelated signal in a diffuse signal generated by the diffuse signal generator so that a greater amount of the decorrelated signal is included in the diffuse signal for a first distance than an amount of the decorrelated signal for a second distance, wherein the first distance is less than the second distance, or The spatial audio synthesizer (750) is configured to: for a first distance between the microphone positions, calculate the component signals of the spatial audio representation using the output signal of the decorrelation filter (823) and the reference signal weighted (822) with a gain derived from the sound arrival direction information, the decorrelation filter being configured to decorrelate the reference signal or the scaled reference signal; and for a second distance between the microphone positions, calculate the component signals of the spatial audio representation using the reference signal weighted (822) with a gain derived from the sound arrival direction information without performing any decorrelation processing, the second distance being greater than the first distance or greater than a distance threshold.

26. The device according to claim 14, wherein the transmission metadata (710) comprises information relating to beamforming or spatial filtering associated with the transmission signal of the transmission representation (711), and The spatial audio synthesizer (750) is configured to generate (932) a speaker signal for a speaker using a transmission signal having a viewing direction closest to a reference position of the spatial audio representation to a viewing direction of the speaker.

27. The device according to claim 14, wherein the spatial audio synthesizer (750) is configured to determine a component signal of the spatial audio representation as a combination (825) of a direct sound component and a diffuse sound component, wherein the direct sound component is obtained by scaling (822) a reference signal using a factor depending on a diffuseness parameter or a directional parameter, wherein the directional parameter depends on the direction of arrival of the sound, wherein the determination of the reference signal is performed (821, 760) based on the information related to the transmission metadata (710), and wherein the diffuse sound component is determined (823, 824) using the same reference signal and the diffuseness parameter.

28. The device according to claim 14, wherein the spatial audio synthesizer (750) is configured to determine a component signal of the spatial audio representation as a combination (825) of a direct sound component and a diffuse sound component, wherein the direct sound component is obtained by scaling (822) a reference signal using a factor depending on a diffuseness parameter or a directional parameter, wherein the directional parameter depends on the direction of arrival of the sound, wherein the determination of the reference signal is performed (821, 760) based on the information related to the transmission metadata (710), and wherein the diffuse sound component is determined (823, 824) using a decorrelation filter (823), the same reference signal and the diffuseness parameter.

29. The apparatus according to claim 14, wherein the transmission representation (711) comprises at least two different microphone signals, wherein the transmission metadata (710) includes information indicating whether the at least two different microphone signals are at least one of omnidirectional signals, dipole signals, or cardioid signals, and The spatial audio synthesizer is configured to: apply (915) a reference signal determination to the transmission metadata (710) to determine independent reference signals for components of the spatial audio representation; and calculate (916) the respective components using the independent reference signals determined for the respective components.

30. A method for encoding a spatial audio representation representing an audio scene to obtain an encoded audio signal, the method comprising: generating a transport representation based on the spatial audio representation (611); generating transport metadata associated with generation of the transport representation (610); deriving spatial parameters from the spatial audio representation (612); as well as multiplexing the information related to the transmission representation (611), the information related to the spatial parameters (612) and the information related to the transmission metadata (610) to obtain the encoded audio signal, wherein the spatial audio representation is a first-order Ambisonics representation or a higher-order Ambisonics representation comprising a plurality of coefficient signals, or a multi-channel representation comprising a plurality of audio channels, wherein generating the transmission representation (611) comprises linearly combining two or more coefficient signals from the higher order Ambisonics representation or the first order Ambisonics representation, or linearly combining two or more audio channels from the multi-channel representation, and Wherein generating the transmission metadata (610) comprises generating the following as the transmission metadata (610): information about how the two or more coefficient signals from the higher order Ambisonics representation or the first order Ambisonics representation or the two or more audio channels from the multi-channel representation are linearly combined, or which coefficient signals from the two or more coefficient signals from the first order Ambisonics representation or from the higher order Ambisonics representation or which audio channels from the two or more audio channels from the multi-channel representation are linearly combined.

31. A method for decoding an encoded audio signal, the method comprising: receiving an encoded audio signal comprising, in multiplexed form, information relating to a transmission representation (711), information relating to spatial parameters (712) and information relating to transmission metadata (710), the transmission representation (711) comprising a plurality of transmission signals; and synthesizing a spatial audio representation using the information about the transmission representation (711), the information about the spatial parameters (712) and the information about the transmission metadata (710), wherein the transmission metadata (710) indicates a first transmission signal of the plurality of transmission signals as relating to a first sector or hemisphere relative to a reference position of the spatial audio representation, and indicates a second transmission signal of the plurality of transmission signals as relating to a second, different sector or hemisphere relative to the reference position of the spatial audio representation, Wherein the synthesis comprises: generating (915) a component signal of the spatial audio representation associated with the first sector or hemisphere using the first transmission signal and not using the second transmission signal, or generating (915) another component signal of the spatial audio representation associated with the second sector or hemisphere using the second transmission signal instead of the first transmission signal, or generating (915) another component signal of the spatial audio representation associated with the first sector or hemisphere using a first linear combination of the first transmitted signal and the second transmitted signal, or generating (915) another component signal of the spatial audio representation associated with the second sector or hemisphere using a second linear combination of the first transmitted signal and the second transmitted signal, wherein the first linear combination is more strongly influenced by the first transmitted signal than the second linear combination, or wherein the second linear combination is more strongly influenced by the second transmitted signal than the first linear combination, or wherein the transmission metadata (710) indicates information related to weighting coefficients or viewing directions of a plurality of transmission signals of the transmission representation (711), and wherein the synthesis comprises using the information related to the viewing directions or the weighting coefficients and using the transmission signals to calculate (932) different first-order Ambisonics components of the spatial audio representation.

32. A computer readable storage medium storing a computer program for executing the method according to claim 30 or 31 when the computer program is run on a computer or a processor.

33. An encoded audio signal comprising, in multiplexed form: information relating to a transport representation (611) of a spatial audio representation, the transport representation (611) comprising a plurality of transport signals; information relating to spatial parameters (612) derived from the spatial audio representation; as well as Information relating to transmission metadata (610), wherein the transmission metadata (610) indicates a first transmission signal of the plurality of transmission signals as relating to a first sector or hemisphere relative to a reference position of the spatial audio representation and a second transmission signal of the plurality of transmission signals as relating to a second, different sector or hemisphere relative to the reference position of the spatial audio representation, or wherein the transmission metadata (610) indicates information relating to weighting coefficients or viewing directions of a plurality of transmission signals of the transmission representation (611).

Images