Representation of spatial audio by means of audio signal and associated metadata

Abstract

To solve the problem of prior art.SOLUTION: A method for encoding and decoding to represent spatial audio which is combination of directional sound and diffuse sound is provided. An exemplary encoding method includes the steps of: inter alia, creating a single- or multi-channel downmix audio signal by down-mixing input audio signals from a plurality of microphones in an audio capture unit capturing spatial audio; determining a first metadata parameter associated with the downmix audio signal, in which the first metadata parameters are indicative of one or more of a relative time delay value, a gain value, and a phase value, which are associated with each input audio signal; and combining the created downmix audio signal and the first metadata parameter into representation of spatial audio.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] (Reference to Related Application) This application claims the benefit of priority to U.S. Provisional Patent Application No. 62 / 760,262, filed November 13, 2018, U.S. Provisional Patent Application No. 62 / 795,248, filed January 22, 2019, U.S. Provisional Patent Application No. 62 / 828,038, filed April 2, 2019, and U.S. Provisional Patent Application No. 62 / 926,719, filed October 28, 2019, the contents of which are incorporated herein by reference.

[0002] The disclosure herein relates generally to coding of audio scenes containing audio objects. In particular, the present invention relates to methods, systems, computer program products and data formats for representing spatial audio, and associated encoders, decoders and renderers for encoding, decoding and rendering spatial audio. [Background technology]

[0003] The introduction of 4G / 5G high-speed wireless access into communications networks, combined with the availability of increasingly powerful hardware platforms, is providing the foundation for advanced communications and multimedia services to be developed faster and easier than ever before.

[0004] The Third Generation Partnership Project (3GPP) Enhanced Voice Service (EVS) codecs have brought very significant improvements in user experience with the introduction of super-wideband (SWB) and full-band (FB) speech and audio coding, along with improvements in packet loss resiliency. However, expanded audio bandwidth is only one dimension required for a truly immersive experience. Support beyond the mono and multi-mono currently offered by EVS is ideally needed to immerse users in compelling virtual worlds in a resource-efficient manner.

[0005] In addition, audio codecs currently specified in 3GPP provide suitable quality and compression for stereo content, but lack the conversational features (e.g., sufficiently low latency) required for conversational voice and videoconferencing. These coders lack the multi-channel functionality required for immersive services, such as live streaming, virtual reality (VR), and immersive videoconferencing.

[0006] To fill this technology gap and address the growing demand for rich multimedia services, an extension to the EVS codec has been proposed for Immersive Voice and Audio Services (IVAS). In addition, 4G / 5G and beyond videoconferencing applications will benefit from the IVAS codec being used as an improved speech coder supporting multi-stream coding (e.g., channel, object, and scene-based audio). Use cases for this next-generation codec include, but are not limited to, speech voice, multi-stream videoconferencing, VR conversation, and user-generated live and non-live content streaming.

[0007] Although the goal is to develop a single codec with attractive configuration and performance (e.g., good audio quality, low latency, spatial audio coding support, adequate bitrate range, high quality error resilience, practical implementation complexity), there is currently no final agreement on the audio input format of the IVAS codec. Metadata Assisted Spatial Audio Format (MASA) has been proposed as one possible audio input format. However, traditional MASA parameters make certain ideal assumptions, such as that audio capture is performed at a single point. However, in real-world scenarios where a mobile phone or tablet is used as an audio capture device, such an assumption of sound capture at a single point may not hold. Rather, depending on the form factor of a particular device, the various microphones of the device may be positioned at a certain distance apart, and the differently captured microphone signals may not be perfectly time-aligned. This is especially true when how the audio source moves spatially is also considered.

[0008] Another underlying assumption of the MASA format is that all microphone channels are presented at equal levels and that there are no differences in frequency and phase response between them. Again, in real-world scenarios, microphone channels may have different direction-dependent frequency and phase characteristics, which may also be time-variant. For example, it can be assumed that an audio capture device is temporarily held such that one of the microphones is occluded or there is some object in the vicinity of the phone that causes reflection or diffraction of the arriving sound waves. Thus, there are many additional factors to consider when determining which audio format is appropriate with a codec such as the IVAS codec. Summary of the Invention

[0009] Exemplary embodiments will now be described with reference to the accompanying drawings. [Brief description of the drawings]

[0010] [Figure 1] 1 is a flowchart of a method for representing spatial audio according to an example embodiment.

[0011] [Diagram 2] 1 is a schematic diagram of an audio capture device and a directional diffusion sound source in accordance with an exemplary embodiment;

[0012] [Figure 3A] 1 shows a table (Table 1A) of how the channel bit value parameter indicates the number of channels used for the MASA format, according to an exemplary embodiment.

[0013] [Figure 3B] 1B shows a table of metadata structures that can be used to represent planar FOA and FOA capture with downmix in two MASA channels according to an exemplary embodiment.

[0014] [Figure 4] 1 shows a table (Table 2) of delay compensation values ​​for each microphone and per TF tile according to an exemplary embodiment.

[0015] [Diagram 5] 13 shows a table (Table 3) of a metadata structure that can be used to indicate which set of compensation values ​​applies to which TF tiles, according to an example embodiment.

[0016] [Figure 6] 4 shows a table of metadata structures (Table 4) that can be used to represent gain adjustments for each microphone, according to an exemplary embodiment.

[0017] [Figure 7] 1 illustrates a system including an audio capture device, an encoder, a decoder, and a renderer according to an exemplary embodiment.

[0018] [Figure 8] 1 illustrates an audio capture device in accordance with an exemplary embodiment.

[0019] [Figure 9] 3 illustrates a decoder and renderer according to an exemplary embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] All figures are schematic and generally show only parts necessary to elucidate the present disclosure, while other parts may be omitted or may merely be suggested. Unless otherwise stated, like reference numbers refer to like parts in the different figures.

[0021] In view of the above, it is therefore an object to provide a method, system, computer program product and data format for an improved representation of spatial audio. Encoders, decoders and renderers for spatial audio are also provided.

[0022] (I. Overview - Spatial Audio Expression) According to a first aspect, there is provided a method, system, computer program product and data format for representing spatial audio.

[0023] According to an exemplary embodiment, there is provided a method for representing spatial audio, which is a combination of directional and diffuse sound, the method comprising: ● Producing a single channel or multi-channel downmix audio signal by downmixing input audio signals from multiple microphones in an audio capture unit that captures spatial audio; determining first metadata parameters associated with the downmix audio signal, the first metadata parameters indicating one or more of a relative time delay value, a gain value, and a phase value associated with each input audio signal; and - combining the resulting downmix audio signal with the first metadata parameters into a representation of spatial audio; Includes.

[0024] In the above-mentioned arrangements, improved representation of spatial audio may be achieved by taking into account the different characteristics and / or spatial positions of multiple microphones. Moreover, the use of metadata in subsequent processing stages of encoding, decoding or rendering may contribute to faithfully representing and reconstructing the captured audio while representing the audio in a bitrate-efficient coded format.

[0025] According to an exemplary embodiment, combining the created downmix audio signal and the first metadata parameters into a representation of spatial audio may further include including second metadata parameters within the representation of spatial audio, the second metadata parameters indicating a downmix configuration for the input audio signal.

[0026] This is advantageous in that it allows to reconstruct the input audio signal at the decoder. Moreover, by providing a second metadata, a further downmix may be performed by a separate unit before encoding the representation of spatial audio into the bitstream.

[0027] According to an example embodiment, a first metadata parameter may be determined for one or more frequency bands of the microphone input audio signal.

[0028] This is advantageous in that it allows for individually adapted delay, gain and / or phase adjustment parameters, for example taking into account different frequency responses for different frequency bands of the microphone signal.

[0029] According to an exemplary embodiment, the downmix producing the single-channel or multi-channel downmix audio signal x is

number

[0030] According to an exemplary embodiment, the downmix coefficients may be selected to select the input audio signal of the microphone that currently has the best signal-to-noise ratio for the directional sound, and discard the input audio signals from any other microphones.

[0031] This is advantageous in that it allows to achieve a good quality representation of spatial audio with reduced computational complexity in the audio capture unit. In this embodiment, only one input audio signal is selected to represent the spatial audio in a particular audio frame and / or time-frequency tile. As a result, the computational complexity of the downmixing operation is reduced.

[0032] According to an example embodiment, the selection may be determined on a time-frequency (TF) tile basis.

[0033] This is advantageous in that it allows for an improved downmixing operation, for example taking into account different frequency responses for different frequency bands of the microphone signals.

[0034] According to an exemplary embodiment, the selection may be made for a particular audio frame.

[0035] Advantageously, this allows for adaptation to time-varying microphone-captured signals and thus improved audio quality.

[0036] According to an example embodiment, the downmix coefficients may be selected to maximize the signal-to-noise ratio for directional sound when combining input audio signals from different microphones.

[0037] This is advantageous in that it allows for an improved quality of the downmix due to the attenuation of undesired signal components that do not originate from directional sources.

[0038] According to an example embodiment, the maximization may be performed for a particular frequency band.

[0039] According to an example embodiment, the maximization may be performed for a particular audio frame.

[0040] According to an example embodiment, determining the first metadata parameter may include analyzing one or more of delay, gain and phase characteristics of input audio signals from multiple microphones.

[0041] According to an example embodiment, the first metadata parameter may be determined on a time-frequency (TF) tile basis.

[0042] According to an example embodiment, at least a portion of the downmixing may occur within the audio capture unit.

[0043] According to an example embodiment, at least a part of the downmix may occur within the encoder.

[0044] According to an example embodiment, when more than one directional sound source is detected, the first metadata may be determined for each sound source.

[0045] According to an example embodiment, the representation of the spatial audio may include at least one of the following parameters: direction index, direct-to-total energy ratio, spread coherence, arrival time, gain and phase for each microphone, diffuse-to-total energy ratio, surround coherence, remainder-to-total energy ratio, and distance.

[0046] According to an example embodiment, a metadata parameter of the second or first metadata parameters may indicate whether the created downmix audio signal is generated from left and right stereo signals, a planar First Order Ambisonics (FOA) signal, or an FOA component signal.

[0047] According to an example embodiment, the representation of spatial audio may include metadata parameters organized into a definition field and a selector field, where the definition field specifies at least one delay compensation parameter set associated with a plurality of microphones, and the selector field specifies a selection of the delay compensation parameter set.

[0048] According to an example embodiment, a selector field may specify which set of delay compensation parameters applies to any given time-frequency tile.

[0049] According to an exemplary embodiment, the relative time delay values ​​may be in the interval approximately [-2.0 ms, 2.0 ms].

[0050] According to an exemplary embodiment, the metadata parameters in the representation of the spatial audio may further include a field specifying the gain adjustment to be applied and a field specifying the phase adjustment.

[0051] According to an exemplary embodiment, the gain adjustments may be in the interval of approximately [+10 dB, -30 dB].

[0052] According to an exemplary embodiment, at least a portion of the first and second metadata elements are determined at the audio capture device using a stored look-up table.

[0053] According to an exemplary embodiment, at least a portion of the first and second metadata elements are determined at a remote device connected to the audio capture device.

[0054] (II. Overview - System) According to a second aspect, a system for representing spatial audio is provided.

[0055] According to an exemplary embodiment, a receiving component configured to receive input audio signals from a plurality of microphones in an audio capture unit that captures spatial audio; a downmixing component configured to downmix a received audio signal to produce a single-channel or multi-channel downmix audio signal; a metadata determination component configured to determine first metadata parameters associated with the downmix audio signal, the first metadata parameters representing one or more of a relative time delay value, a gain value, and a phase value associated with each input audio signal; a combining component configured to combine the produced downmix audio signal and the first metadata parameters into a representation of spatial audio. A system for rendering spatial audio is provided.

[0056] (III. Overview - Data Format) According to a third aspect, a data format for representing spatial audio is provided. The data format may be advantageously used with spatial audio related physical components such as audio capture devices, encoders, decoders, renderers, etc., various types of computer program products, and other equipment used to transmit spatial audio between devices and / or locations.

[0057] According to an exemplary embodiment, the data format is: a downmix audio signal resulting from downmixing input audio signals from multiple microphones in an audio capture unit capturing spatial audio; and first metadata parameters indicating a downmix configuration for the input audio signals, and one or more of a relative time delay value, a gain value, and a phase value associated with each input audio signal.

[0058] According to one example, the data format is stored in a non-transitory memory.

[0059] (IV. Overview - Encoders) According to a fourth aspect, there is provided an encoder for encoding a representation of spatial audio.

[0060] According to an exemplary embodiment, A representation of spatial audio, The expression is, A single or multi-channel downmix audio signal produced by downmixing input audio signals from multiple microphones in an audio capture unit that captures spatial audio; and a first metadata parameter associated with the downmix audio signal, the first metadata parameter indicating one or more of a relative time delay value, a gain value, and a phase value associated with each input audio signal; Receive a representation of spatial audio, encoding a single-channel or multi-channel downmix audio signal into a bitstream using the first metadata, or encoding the single channel or multi-channel downmix audio signal and the first metadata into a bitstream; It is configured as follows: Encoder provided

[0061] (V. Overview - Decoder) According to a fifth aspect, there is provided a decoder for decoding a representation of spatial audio.

[0062] According to an exemplary embodiment, 1. An encoded representation of spatial audio, comprising: The expression is, A single-channel or multi-channel downmix audio signal produced by downmixing input audio signals from multiple microphones in an audio capture unit that captures spatial audio; and a first metadata parameter associated with the downmix audio signal, the first metadata parameter indicating one or more of a relative time delay value, a gain value, and a phase value associated with each input audio signal; receiving a bitstream representing an encoded representation of spatial audio; Decoding the bitstream into an approximation of the spatial audio by using the first metadata parameters; It is configured as follows: A decoder is provided.

[0063] (VI. Overview - Renderer) According to a sixth aspect, there is provided a renderer for rendering a representation of spatial audio.

[0064] According to an exemplary embodiment, A representation of spatial audio, The expression is, A single-channel or multi-channel downmix audio signal produced by downmixing input audio signals from multiple microphones in an audio capture unit that captures spatial audio; and a first metadata parameter associated with the downmix audio signal, the first metadata parameter indicating one or more of a relative time delay value, a gain value, and a phase value associated with each input audio signal; Receive a representation of spatial audio, Rendering spatial audio using the first metadata; It is configured as follows: A renderer is provided.

[0065] (VII. Overview - General) The second to sixth aspects may generally have the same configurations and advantages as the first aspect.

[0066] Other objects, features and advantages of the present invention will become apparent from the following detailed description, from the attached dependent claims, as well as from the drawings.

[0067] The steps of any method disclosed herein do not have to be performed in the exact order disclosed, unless explicitly stated.

[0068] VIII. EXEMPLARY EMBODIMENTS As mentioned above, capturing and rendering spatial audio presents a particular set of challenges so that the captured audio can be faithfully reproduced at the receiving end. Various embodiments of the invention described herein address different aspects of these problems by including various metadata parameters along with the downmix audio signal when transmitting the downmix audio signal.

[0069] The invention will be described with reference to the MASA audio format, as an example, however, it is important to recognize that the general principles of the invention are applicable to a wide range of formats that may be used to represent audio, and that the description herein is not limited to MASA.

[0070] Furthermore, it should be recognized that the metadata parameters described below are not a complete list of metadata parameters, but that there may be additional metadata parameters (or a smaller subset of metadata parameters) that can be used to convey data about the downmix audio signal to various devices used when encoding, decoding, and rendering the audio.

[0071] Also, while the examples herein are described in the context of an IVAS encoder, it should be noted that this is but one type of encoder to which the general principles of the invention can be applied, and that there may be many other types of encoders, decoders, and renderers that may be used in conjunction with the various embodiments described herein.

[0072] Finally, although the terms "upmixing" and "downmixing" are used throughout this document, they may not necessarily mean an increase and a decrease in the number of channels, respectively. It should be recognized that either term can mean either a decrease or an increase in the number of channels, although this often occurs. Thus, both terms fall under the more general concept of "mixing." Similarly, although the term "downmix audio signal" is used throughout this document, it should be recognized that sometimes other terms may be used, such as "MASA channel," "transport channel," or "downmix channel," all of which have essentially the same meaning as "downmix audio signal."

[0073] Now, referring to Fig. 1, a method 100 for representing spatial audio is described according to one embodiment. As can be seen in Fig. 1, the method begins by capturing spatial audio using an audio capturing device (step 102). Fig. 2 shows a schematic diagram of a sound environment 200 in which an audio capturing device 202, such as a mobile phone or tablet computer, captures audio from a diffuse ambient source 204 and a directional source 206, such as a talker. In the illustrated embodiment, the audio capturing device 202 has three microphones m1, m2, and m3, respectively.

[0074] Directional sound is incident from a direction of arrival (DOA) represented by azimuth and elevation angles. Ambient sound is presumed to be omnidirectional, i.e., spatially invariant or spatially uniform. The following discussion also takes into account the potential occurrence of a second directional sound source, not shown in FIG. 2.

[0075] Then, the signals from the microphones are downmixed to produce a single-channel or multi-channel downmix audio signal (step 104). There are many reasons to propagate only a mono downmix audio signal. For example, there may be an intention or a bitrate limitation to make a high-quality mono downmix audio signal available after certain proprietary enhancements such as beamforming and equalization or noise suppression are performed. In other embodiments, the downmix results in a multi-channel downmix audio signal. Generally, the number of channels in the downmix audio signal is less than the number of input audio signals, but in some cases the number of channels in the downmix audio signal may be equal to the number of input audio signals, and the downmix rather achieves an increased SNR or reduces the amount of data in the resulting downmix audio signal compared to the input audio signals. This will be explained in more detail below.

[0076] Propagating the relevant parameters used during the downmix as part of the MASA metadata to the IVAS codec may provide the possibility to restore the stereo signal and / or the spatial downmix audio signal with the best possible fidelity.

[0077] In this scenario, a single MASA channel is obtained by the following downmix operation:

number

[0078] The signals m and x may not necessarily be represented as full-band time signals during the various processing stages, but may also possibly be represented as component signals of various sub-bands in the time or frequency domain (TF tiles), in which case they are eventually recombined and potentially transformed back to the time domain before being propagated to the IVAS codec.

[0079] An audio encoding / decoding system typically divides a time-frequency space into time / frequency tiles, for example by applying appropriate filter banks to the input audio signal. A time / frequency tile generally means a part of a time-frequency space corresponding to a time interval and a frequency band. The time interval may typically correspond to the duration of a time frame used in the audio encoding / decoding system. A frequency band is a part of the total frequency range of an audio signal / object to be encoded or decoded. A frequency band may typically correspond to one or several adjacent frequency bands defined by a filter bank used in the encoding / decoding system. If a frequency band corresponds to several adjacent frequency bands defined by a filter bank, this allows to have a non-uniform frequency band in the decoding process of the downmix audio signal, for example a wider frequency band for higher frequencies of the downmix audio signal.

[0080] In an implementation using a single MASA channel, there are at least two options for how the downmix matrix D may be defined. One option is to select the microphone signal with the best signal-to-noise ratio (SNR) for directional sound. In the configuration shown in FIG. 2, microphone m1 is likely to capture the best signal because it is oriented towards the directional sound source. Signals from the other microphones can then be discarded. The downmix matrix may then be:

number

[0081] While the sound source is moving relative to the audio capturing device, another, more suitable microphone can be selected so that either signal m2 or m3 is used as the resulting MASA channel.

[0082] When switching microphone signals, it is important that the MASA channel signals do not suffer from any potential discontinuities, which may occur due to different arrival times of directional sound sources at different microphones, or due to different gain or phase characteristics of the acoustic paths from the sound sources to the microphones. As a result, the individual delay, gain and phase characteristics of the different microphone inputs must be analyzed and compensated for. Therefore, the actual microphone signals may undergo some specific delay adjustment and filtering operations before the MASA downmix.

[0083] In another embodiment, the coefficients of the downmix matrix are set such that the SNR of the MASA channel for directional sources is maximized. This can be achieved, for example, by using appropriately adjusted weights k 1,1 , k 1,2 , k 1,3 This can be achieved by adding different microphone signals with the same delay, gain and phase characteristics of the different microphone inputs again to perform this task in an effective manner, which can also be understood as acoustic beamforming towards a directional sound source.

[0084] The gain / phase adjustments have to be understood as frequency selective filtering operations, so that the corresponding adjustments may be optimized to achieve acoustic noise reduction or directional sound signal enhancement, for example according to the Wiener approach.

[0085] As a further variant, there may be an example with three MASA channels, in which case the downmix matrix D can be defined by the following 3×3 matrix:

number

[0086] As a result, there are now three signals x1, x2, x3 that can be encoded with the IVAS codec (instead of one signal as in the first example).

[0087] The first MASA channel may be generated as described in the first example. If there is a second directional sound, the second MASA channel may be used to convey the second directional sound. However, in that case, the downmix matrix coefficients may be selected according to similar principles as for the first MASA channel, such that the SNR of the second directional sound is maximized. The downmix matrix coefficients k for the third MASA channel may be selected according to similar principles as for the first MASA channel, such that the SNR of the second directional sound is maximized. 3,1 , k 3,2 , k 3,3 may be configured to extract the diffuse sound components while minimizing directional sound.

[0088] Typically, a stereo capture of a dominant directional sound source in the presence of some ambient sounds may be performed as shown in Fig. 2 and as described above. This may occur frequently in certain use cases, e.g., telephony. According to various embodiments described herein, metadata parameters are also determined in conjunction with the downmixing step 104, which are subsequently added to and propagated along with the single mono downmix audio signal.

[0089] In one embodiment, three main metadata parameters are associated with each captured audio signal: a relative time delay value, a gain value, and a phase value. According to a general approach, the MASA channel is obtained according to the following operations: ● Quantity τ i = Δτ i +τ ref Each microphone signal m i ( i=1,2) delay adjustment. Gain and phase adjustment of each time-frequency (TF) component / tile of each delay-adjusted microphone signal by only the gain and phase adjustment parameters α and φ, respectively.

[0090] The delay adjustment term τ in the above formula i can be interpreted as the arrival time of a plane sound wave from the direction of a directional sound source, and thus it can be expressed as the arrival time of a reference point τ ref Although it can be conveniently expressed as the arrival time relative to the arrival time of the sound wave at , any reference point can be used. For example, when two microphones are used, the delay adjustment can be formulated as the difference between τ1 and τ2, which is equivalent to moving the reference point to the position of the second microphone. In one embodiment, the arrival time parameters allow modeling the relative arrival times in the interval [-2.0ms, 2.0ms], which corresponds to a maximum displacement of the microphones relative to the origin of about 68cm.

[0091] Regarding gain and phase adjustments, in one embodiment, they are parameterized for each TF tile such that gain changes can be modeled in the range of [+10 dB, -30 dB], while phase changes can be expressed in the range of [-Pi, +Pi].

[0092] In the basic case with only a single dominant directional source, such as source 206 shown in Figure 2, the delay adjustment is typically constant across the entire frequency spectrum. Since the position of the directional source 206 may change, the two delay adjustment parameters (one for each microphone) change over time. Thus, the delay adjustment parameters are signal dependent.

[0093] In more complex cases where there are multiple directional sound sources 206, one source from a first direction may dominate in a particular frequency band, while a different source from another direction may dominate in another frequency band. In such scenarios, delay adjustments are advantageously performed for each frequency band instead.

[0094] In one embodiment, this can be done by delay-compensating the microphone signals within a given time-frequency (TF) tile with respect to the sound direction that is deemed to be dominant. If no dominant sound direction is detected in the TF tile, no delay compensation is performed.

[0095] In different embodiments, microphone signals within a given TF tile can be delay compensated with the goal of maximizing the signal-to-noise ratio (SNR) for directional sound as captured by all microphones.

[0096] In one embodiment, a suitable limit of different sound sources for which delay compensation can be performed is three. This gives the possibility of either performing delay compensation in a TF tile for one of the three main sound sources or none at all. Thus, only two bits per TF tile can signal a corresponding set of delay compensation values ​​(set applied to all microphone signals). This has the advantage that it covers most practically relevant capture scenarios and the amount of metadata or their bitrate remains low.

[0097] Another possible scenario is when a First Order Ambisonics (FOA) signal is taken rather than a stereo signal and downmixed, for example, to a single MASA channel. The concept of FOA is well known to those skilled in the art, but can be simply described as a method for recording, mixing, and playing back three-dimensional 360 degree audio. The basic approach of Ambisonics is to treat the audio scene as a full 360 degree sphere of sound coming from different directions around a central point where the microphone is located during recording or where the listener's "sweet spot" is located during playback.

[0098] Planar FOA and FOA capture with downmix to a single MASA channel are relatively simple extensions of the stereo capture case described above. The planar FOA case is characterized by a microphone triple as shown in Figure 2, where capture occurs before downmix. In the latter FOA case, capture is performed with four microphones, whose placement or directional selectivity spans all three spatial dimensions.

[0099] The delay compensation, amplitude and phase adjustment parameters can be used to recover the three or four original captured signals, respectively, allowing a more faithful spatial rendering with MASA metadata than would be possible based on the mono downmix signal alone. Alternatively, the delay compensation, amplitude and phase adjustment parameters can be used to generate a more accurate (planar) FOA representation that more closely resembles that captured with a regular microphone grid.

[0100] In yet another scenario, a planar FOA or FOA may be captured and downmixed to two or more MASA channels. This case is an extension of the previous case with the difference that three or four microphone signals are captured and downmixed to two MASA channels rather than just one. The same principle applies, and the purpose of providing delay compensation, amplitude and phase adjustment parameters is then to enable the best possible reconstruction of the original signal before downmixing.

[0101] As the skilled reader will appreciate, in order to accommodate all these usage scenarios, the spatial audio representation needs to include not only metadata on delay, gain and phase, but also metadata on parameters indicative of the downmix configuration for the downmix audio signal.

[0102] Returning now to Figure 1, the determined metadata parameters are combined with the downmix audio signal into a representation of spatial audio (step 108), which completes process 100. Below is a description of how these metadata parameters may be represented in accordance with one embodiment of the present invention.

[0103] To support the above mentioned use cases of downmixing to single or multiple MASA channels, two metadata elements are used. One metadata element is signal-independent configuration metadata indicating the downmix. This metadata element is described below in relation to Figures 3A-3B. The other metadata element is associated with the downmix. This metadata element is described below in relation to Figures 4-6 and may be determined as described above in relation to Figure 1. This metadata element is needed when a downmix is ​​signaled.

[0104] Table 1A, shown in FIG. 3A, is a metadata structure that can be used to indicate the number of MASA channels, ranging from a single (mono) MASA channel to two (stereo) MASA channels up to a maximum of four MASA channels, represented by channel bit values ​​00, 01, 10, and 11, respectively.

[0105] Table 1B, shown in FIG. 3B, includes the channel bit values ​​from Table 1A (in this particular case, only channel values ​​"00" and "01" are shown for illustrative purposes) and shows how the microphone capture configuration can be represented. For example, for a single (mono) MASA channel, as can be seen in Table 1B, it can be signaled whether the capture configuration is mono, stereo, planar FOA or FOA. As can further be seen in Table 1B, the microphone capture configuration is coded as a 2-bit field (in the column named Bit Value). Table 1B also includes additional descriptions of metadata. A further signal-independent configuration could represent, for example, that the audio originated from the microphone grid of a smartphone or similar device.

[0106] If the downmix metadata is signal dependent, some further details are needed, as described next. For the specific case, as shown in Table 1B, when the transport signal is a mono signal obtained through downmixing of a multi-microphone signal, these details are provided in a signal-dependent metadata field. The information provided in that metadata field describes the delay adjustments applied (for possible acoustic beamforming towards directional sound sources) and filtering of the microphone signals (for possible equalization / noise suppression) before the downmix. This provides additional information that may benefit the encoding, decoding and / or rendering.

[0107] In one embodiment, the downmix metadata includes four fields: a define field and a selector field for signaling the delay compensation to be applied, followed by two fields each for signaling the gain and phase adjustments to be applied, respectively.

[0108] The number n of downmixed microphone signals is signaled by the “Bit Value” field of Table 1B, i.e., n=2 (“Bit Value=01”) for a stereo downmix, n=3 (“Bit Value=10”) for a planar FOA downmix, and n=4 (“Bit Value=11”) for a FOA downmix.

[0109] Up to three different sets of delay compensation values ​​for up to n microphone signals can be defined and signalled per TF tile, one set for each directional source direction. The signalling of which set applies to which TF tile and the definition of the set of delay compensation values ​​are done in two separate (Define and Selector) fields.

[0110] In one embodiment, the definition field is the applied delay compensation Δτ i,j An 8-bit element B that encodes i,j These parameters are the respective sets they belong to, i.e. the respective directions of the directional sound source (j=1...3). The 8-bit elements are furthermore the respective capture microphones (or associated capture signals) (i=1...n, n≦4). This is illustrated diagrammatically in Table 2 shown in FIG. 4.

[0111] Thus, Fig. 4 together with Fig. 3 shows an embodiment in which the representation of spatial audio includes metadata parameters organized into a definition field and a selector field. The definition field specifies at least one delay compensation parameter set associated with multiple microphones, and the selector field specifies a selection of the delay compensation parameter set. Advantageously, the representation of the relative time delay values ​​between the microphones is compact and thus requires less bitrate when transmitted to a subsequent encoder or the like.

[0112] The delay compensation parameter represents the relative arrival time of an estimated plane sound wave from the direction of the sound source compared to the arrival of the wave at the (random) geometric center point of the audio capturing device 202. The coding of that parameter by an 8-bit integer codeword B is done according to the following equation (Equation No. (1)):

number

[0113] This quantizes the relative delay parameter linearly in the interval [-2.0ms, 2.0ms], which corresponds to a maximum displacement of the microphone relative to the origin of approximately 68cm. This is, of course, only one example, and other quantization characteristics and solutions may be considered.

[0114] The signaling of which set of delay compensation values ​​is applied to which TF tile is done using a selector field representing 4x24 TF tiles in a 20 ms frame assuming 24 frequency bands and 4 subframes in a 20 ms frame. Each field element contains a 2-bit entry encoding set 1...3 of delay compensation values ​​with respective codes "01", "10", "11". If no delay compensation is applied to a TF tile then the "00" entry is used. This is illustrated diagrammatically in Table 3 shown in Figure 5.

[0115] Gain adjustment is performed in metadata fields 2-4, one for each microphone. Each field contains an 8-bit gain adjustment code B α The matrix of integer codewords B α The coding of the gain adjustment parameter using is performed according to the following equation (Equation No. (2)).

number

[0116] The metadata fields 2-4 for each microphone are organized as shown in Table 4, shown in FIG.

[0117] The phase adjustments are signaled similarly to the gain adjustments in metadata fields 2-4, one for each microphone. Each field is a matrix of 8-bit phase adjustment codes Bφ, respectively for 4 × 24 TF tiles of the 20 ms frame. The coding of the phase adjustment parameters using integer codewords Bφ is done according to the following equation (Equation No. (3)):

number

[0118] The metadata fields for each microphone 2-4 are organized as shown in Table 4, with the only difference being that the field element is the phase adjustment codeword Bφ.

[0119] This representation of the MASA signal, including associated metadata, can then be used by encoders, decoders, renderers and other types of audio equipment used to transmit, receive and faithfully restore the recorded spatial sound environment. Techniques for doing this are well known by those skilled in the art and can be readily adapted to fit the representation of spatial audio described herein. Further discussion of these particular devices is therefore not deemed necessary in this context.

[0120] As will be appreciated by those skilled in the art, the above mentioned metadata elements may exist or be determined in different ways. For example, the metadata may be determined locally on a device (such as an audio capture device, an encoder device, etc.), may be derived from other data (e.g., from a cloud or other remote service), or may be stored in a table of predefined values. For example, based on delay adjustments between microphones, delay compensation values ​​for the microphones (FIG. 4) may be determined by a look-up table stored at the audio capture device, or may be received from a remote device based on delay adjustment calculations performed at the audio capture device, or may be received from such remote device based on delay adjustment calculations performed at the remote device (i.e., based on the input signal).

[0121] FIG. 7 illustrates a system 700 according to an exemplary embodiment in which the above-mentioned configurations of the present invention can be implemented. The system 700 includes an audio capture device 202, an encoder 704, a decoder 706, and a renderer 708. The different components of the system 700 can communicate with each other through wired or wireless connections, or any combination thereof, and data is typically transmitted between the units in the form of a bitstream. The audio capture device 202 is described above in connection with FIG. 2 and is configured to capture spatial audio, which is a combination of directional and diffuse sound. The audio capture device 202 creates a single-channel or multi-channel downmix audio signal by downmixing input audio signals from multiple microphones in an audio capture unit that captures the spatial audio. The audio capture device 202 then determines a first metadata parameter associated with the downmix audio signal, which is further described below in connection with FIG. 8. The first metadata parameter indicates a relative time delay value, a gain value, and / or a phase value associated with each input audio signal. Finally, the audio capture device 202 combines the downmix audio signal and the first metadata parameters into a representation of spatial audio. In the current embodiment, all audio capture and combining occurs in the audio capture device 202, although there may be alternative embodiments in which certain parts of the producing, determining, and combining operations occur in the encoder 704.

[0122] The encoder 704 receives a representation of spatial audio from the audio capture device 202. That is, the encoder 704 receives a data format including a single-channel or multi-channel downmix audio signal resulting from a downmix of input audio signals from multiple microphones in an audio capture unit capturing the spatial audio, and first metadata parameters indicating a downmix configuration for the input audio signals, relative time delay values, gain values, and / or phase values ​​associated with each input audio signal. It should be noted that the data format may be stored in a non-transient memory before / after being received by the encoder. The encoder 704 then uses the first metadata to encode the single-channel or multi-channel downmix audio signal into a bitstream. In some embodiments, the encoder 704 may be an IVAS encoder, as described above, although as one skilled in the art will recognize, other types of encoders 704 may have similar capabilities or may be available.

[0123] The encoded bitstream representing a coded representation of the spatial audio is then received by a decoder 706. The decoder 706 decodes the bitstream into an approximation of the spatial audio by using the metadata parameters included in the bitstream from the encoder 704. Finally, a renderer 708 receives the decoded representation of the spatial audio and renders the spatial audio using the metadata to produce a faithful reproduction of the spatial audio at the receiving end, for example through one or more speakers.

[0124] 8 illustrates an audio capture device 202 according to some embodiments. The audio capture device 202 may, in some embodiments, include a memory 802 with stored look-up tables for determining the first and / or second metadata. The audio capture device 202 may, in some embodiments, be connected to a remote device 804 (which may be located in the cloud or may be a physical device connected to the audio capture device 202), which may include a memory 806 with stored look-up tables for determining the first and / or second metadata. The audio capture device may, in some embodiments, perform the necessary calculations / processing (e.g., using a processor 803) to determine, for example, relative time delay values, gain values, and phase values ​​associated with each input audio signal, transmit such parameters to the remote device, and receive the first and / or second metadata from the device. In other embodiments, the audio capture device 202 transmits the input signal to the remote device 804, which performs the necessary calculations / processing (e.g., with the processor 805) and determines the first and / or second metadata for transmission back to the audio capture device 202. In yet another embodiment, the remote device 804, which performs the necessary calculations / processing, transmits the parameters back to the audio capture device 202, which determines the first and / or second metadata locally based on the received parameters (e.g., by use of the memory 806 with stored look-up tables).

[0125] 9 shows a decoder 706 and a renderer 708 (each including a processor 910, 912 for performing various processing, e.g., decoding, rendering, etc.), according to an embodiment. The decoder and renderer may be separate devices or may be in the same device. The processor(s) 910, 912 may be shared between the decoder and the renderer or separate processors. As described in connection with FIG. 8, interpretation of the first and / or second metadata may be performed using a look-up table stored in either a memory 902 in the decoder 706, a memory 904 in the renderer 708, or a memory 906 in a remote device 905 (including a processor 908) connected to either the decoder or the renderer.

[0126] (Equivalents, Extensions, Substitutes and Others) Further embodiments of the present disclosure will be apparent to those skilled in the art after studying the above description. The present description and drawings disclose embodiments and examples, but the present disclosure is not limited to these specific examples. Numerous modifications and variations can be made without departing from the scope of the present disclosure, which is defined by the appended claims. Reference signs appearing in the claims should not be understood as limiting their scope.

[0127] In addition, those skilled in the art can understand and implement variations to the disclosed embodiments from a study of the drawings, the disclosure, and the appended claims when practicing the disclosure. In the claims, the word "comprises" does not exclude other elements or steps, and the singular does not exclude a plurality. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

[0128] The systems and methods disclosed above may be implemented as software, firmware, hardware, or a combination thereof. In hardware implementations, the division of tasks between functional units referred to in the above description does not necessarily correspond to a division into physical units. Conversely, one physical component may have multiple functionalities, and one task may be performed by multiple physical components in cooperation. Certain or all components may be implemented as software executed by a digital signal processor or microprocessor, or may be implemented as hardware or as an application specific integrated circuit. Such software may be distributed on computer readable media, which may include computer storage media (or non-transitory media) and communication media (or transitory media). As is well known to those skilled in the art, the term computer storage media includes both volatile and non-volatile, removable and non-removable media, implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be accessed by a computer. Additionally, those skilled in the art will appreciate that communication media typically embodies computer-readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media.

[0129] All figures are schematic and generally show only those parts necessary to elucidate the present disclosure, whereas other parts may be omitted or may merely be suggested. Unless otherwise noted, like reference numbers refer to like parts in the different figures.

Claims

1. 1. A method for generating an encoded bitstream representing a spatial audio signal that is a combination of directional and diffuse sound, comprising: receiving a downmix audio signal comprising two channels, a first channel of the downmix audio signal comprising a first directional sound component corresponding to a first direction, a second channel of the downmix audio signal comprising a second directional sound component corresponding to a second direction, and at least one of the first channel and the second channel further comprising a diffuse sound component; for the first channel of the downmix audio signal a first direction indicator indicative of a direction of arrival (DOA) of the first directional sound component, the direction of arrival corresponding to a first azimuth angle and a first elevation angle; a first direct-to-total energy ratio indicating a ratio of the energy of the first directional sound component to a total energy; Receiving and for the second channel of the downmix audio signal a second direction indicator indicative of a direction of arrival (DOA) of the second directional sound component, the direction of arrival corresponding to a second azimuth angle and a second elevation angle; and a second direct-to-total energy ratio indicating a ratio of the energy of the second directional sound component to the total energy; Receiving and receiving a diffuse-to-total energy ratio indicative of a ratio of diffuse sound energy to the total energy; encoding the downmix audio signal into an encoded downmix audio signal; incorporating the encoded downmix audio signal, the first directional indicator, the first direct-to-total energy ratio, the second directional indicator, the second direct-to-total energy ratio, and the spread-to-total energy ratio into an encoded bitstream. method.

2. 2. The method of claim 1, wherein the first directional indicator, the first direct-to-total energy ratio, the second directional indicator, the second direct-to-total energy ratio, and the spread-to-total energy ratio are received for each of a plurality of frequency bands.

3. The method of claim 1 , further comprising receiving source format parameters and incorporating the source format parameters into the encoded bitstream.

4. The method of claim 3 , wherein the source format parameters indicate that the downmix audio signal is derived from Ambisonics component signals.

5. The method of claim 3 , wherein the source format parameters indicate that the downmix audio signal is derived from left / right stereo component signals.

6. The method of claim 1 , wherein the encoding is performed by an Enhanced Voice Services (EVS) encoder or an Immersive Voice and Audio Services (IVAS) encoder.

7. 13. An encoder comprising one or more processors configured to perform the method of claim 1.

8. 13. A computer program product comprising a non-transitory computer readable medium having instructions for performing the method of claim 1.

9. 1. A method for decoding an encoded bitstream representing a spatial audio signal that is a combination of directional and diffuse sound, comprising: receiving the encoded bitstream; and decompressing an encoded downmix audio signal from the encoded bitstream; decoding the encoded downmix audio signal to provide a decoded downmix audio signal comprising two channels, wherein a first channel of the decoded downmix audio signal comprises a first directional sound component corresponding to a first direction, and a second channel of the decoded downmix audio signal comprises a second directional sound component corresponding to a second direction, and at least one of the first channel of the decoded downmix audio signal and the second channel of the decoded downmix audio signal further comprises a diffuse sound component; for the first channel of the decoded downmix audio signal, from the encoded bitstream: a first direction indicator indicative of a direction of arrival (DOA) of the first directional sound component, the direction of arrival corresponding to a first azimuth angle and a first elevation angle; a first direct-to-total energy ratio indicating a ratio of the energy of the first directional sound component to a total energy; To defrost and for the second channel of the decoded downmix audio signal, from the encoded bitstream: a second direction indicator indicative of a direction of arrival (DOA) of the second directional sound component, the direction of arrival corresponding to a second azimuth angle and a second elevation angle; and a second direct-to-total energy ratio indicating a ratio of the energy of the second directional sound component to the total energy; To defrost and extracting from the encoded bitstream a diffuse-to-total energy ratio indicative of a ratio of diffuse sound energy to the total energy; outputting the decoded downmix audio signal, the first directional indicator, the first direct-to-total energy ratio, the second directional indicator, the second direct-to-total energy ratio, and the diffuse-to-total energy ratio. method.

10. 10. The method of claim 9, wherein the first directional index, the first direct-to-total energy ratio, the second directional index, the second direct-to-total energy ratio, and the spread-to-total energy ratio are decompressed for each of a plurality of frequency bands.

11. The method of claim 9 , further comprising extracting source format parameters from the encoded bitstream.

12. The method of claim 11 , wherein the source format parameters indicate that the decoded downmix audio signal is derived from Ambisonics component signals.

13. The method of claim 11 , wherein the source format parameters indicate that the decoded downmix audio signal is derived from left / right stereo component signals.

14. The method of claim 9 , wherein the decoding is performed by an Enhanced Voice Services (EVS) decoder or an Immersive Voice and Audio Services (IVAS) decoder.

15. 10. The method of claim 9, further comprising: reconstructing the spatial audio signal by rendering the decoded downmix audio signal based on one or more of the first directional indicator, the first direct-to-total energy ratio, the second directional indicator, the second direct-to-total energy ratio, and the diffuse-to-total energy ratio.

16. A decoder comprising one or more processors configured to perform the method of claim 9.

17. 10. A computer program product comprising a non-transitory computer readable medium having instructions for performing the method of claim 9.

Images