Spatial audio parameter encoding and related decoding

Abstract

An apparatus includes components configured to: acquire at least one audio signal; for the at least one audio signal, acquire spatial audio signal parameter values ​​distributed in a time-frequency domain (106); determine a merging metric to control the merging of the spatial audio signal parameter values ​​in the time-frequency domain (201); and merge the spatial audio signal parameter values ​​into a smaller number of spatial audio signal parameter values ​​in the time-frequency domain based on the merging metric (202) in time and / or frequency (203).

Description

Technical Field

[0001] This application relates to apparatus and methods for encoding sound field related parameters, but not exclusively to apparatus and methods for encoding time-frequency domain directional related parameters for audio encoders and decoders. Background Technology

[0002] Parametric spatial audio processing is a field of audio signal processing where the spatial aspect of sound is described using a set of parameters. For example, in parametric spatial audio captured from a microphone array, estimating a set of spatial metadata parameters (such as the direction of sound in a frequency band and the ratio between the directional and non-directional portions of the captured sound in that band) from the microphone array signal is a typical and effective choice. These parameters are well-known for describing the perceived spatial characteristics of sound captured at the location of the microphone array. These parameters can then be used accordingly for the synthesis of spatial sound, in binary form for headphones, for speakers, or for other formats such as Ambisonics.

[0003] Therefore, spatial metadata such as direction in the frequency band and direct-to-total energy ratios are particularly effective parameterizations for spatial audio capture.

[0004] A set of spatial metadata parameters, consisting of one or more directional values ​​for each frequency band and an energy ratio parameter associated with each directional value, can also be used as spatial metadata for an audio codec (which may also include other parameters such as spread coherence, number of directions, distance, etc.). The spatial metadata parameter set may also include other parameters, or may be associated with other parameters considered non-directional (such as surround coherence, diffuse-to-total energy ratio, and remainder-to-total energy ratio). For example, these parameters can be estimated from audio signals captured by a microphone array, and, for example, stereo signals can be generated from microphone array signals to be transmitted along with the spatial metadata.

[0005] Because some codecs are expected to operate at a wide range of bit rates, from very low to relatively high, various strategies are needed to compress spatial metadata to optimize codec performance at each point of operation. The raw bit rate (metadata) of the encoded parameters is relatively high, so especially at lower bit rates, it is expected that only the most critical parts of the metadata can be transmitted from the encoder to the decoder.

[0006] The decoder can decode audio signals into PCM signals and process the sound in the frequency band (using spatial metadata) to obtain spatial output, such as binaural output.

[0007] The above solution is particularly well-suited for encoding spatial sound captured from microphone arrays (e.g., in mobile phones, video cameras, VR cameras, standalone microphone arrays). However, for such encoders, it is desirable to have other input types besides the signals captured by the microphone array, such as speaker signals, audio object signals, or Ambisonics signals. Summary of the Invention

[0008] According to a first aspect, an apparatus is provided, the apparatus comprising components configured to: acquire at least one audio signal; for the at least one audio signal, acquire spatial audio signal parameter values ​​distributed in a time-frequency domain; determine a merging metric to control the merging of the spatial audio signal parameter values ​​in the time-frequency domain; and merge the spatial audio signal parameter values ​​into a smaller number of spatial audio signal parameter values ​​in the time-frequency domain based on the merging metric in time and / or frequency.

[0009] A component configured to determine a merging metric to control the merging of spatial audio signal parameter values ​​in the time-frequency domain can be configured to determine an initial metric for detecting the start of a sound event.

[0010] The component configured to determine the initial metric can be configured to: determine an energy parameter for at least one audio signal within a time period; determine a slow audio signal envelope based on the energy parameter and a slow decay time; determine a fast audio signal envelope based on the energy parameter and a fast decay time; and determine the initial metric based on the slow audio signal envelope and the fast audio signal envelope.

[0011] A component configured to combine spatial audio signal parameter values ​​into a smaller number of spatial audio signal parameter values ​​in the time and / or frequency domain based on a merging metric can be configured to: determine the spatial audio signal parameter value band that best represents the spatial audio signal parameter value band within a time period when the initiation metric indicates the start of a sound event.

[0012] A component configured to merge spatial audio signal parameter values ​​into a smaller number of spatial audio signal parameter values ​​in the time-frequency domain based on a merging metric can be configured to: for a determined frequency band of the spatial audio signal parameter values, determine whether the energy ratio of the frequency band is greater than the weighted average of the energy ratios of the frequency bands within a time period; and when the energy ratio of the determined frequency band of the spatial audio signal parameter values ​​is greater than the weighted average of the energy ratios of the frequency bands within a time period, merge the spatial audio signal parameter values ​​into a smaller number of spatial audio signal parameter values ​​in the frequency domain.

[0013] A component configured to merge spatial audio signal parameter values ​​into a smaller number of spatial audio signal parameter values ​​in the time and / or frequency domain based on a merging metric can be configured to merge spatial audio signal parameter values ​​into a smaller number of spatial audio signal parameter values ​​in time when the energy ratio of the determined frequency band of the spatial audio signal parameter value is less than the weighted average of the energy ratios of the frequency bands within the time period.

[0014] A component configured to merge spatial audio signal parameter values ​​into a smaller number of spatial audio signal parameter values ​​in the time-frequency domain based on a merging metric can be configured to merge spatial audio signal parameter values ​​into a smaller number of spatial audio signal parameter values ​​in time when the start metric indicates the absence of the start of a sound event.

[0015] This component can also be configured to encode the combined spatial audio signal parameter values.

[0016] The component configured to encode the combined spatial audio signal parameter values ​​can be configured to quantize the combined spatial audio signal parameter values.

[0017] The component configured to encode the combined spatial audio signal parameter values ​​can be configured to entropy encode the combined spatial audio signal parameter values.

[0018] According to a second aspect, an apparatus is provided, comprising components configured to: acquire at least one encoded spatial audio signal, the at least one encoded spatial audio signal comprising: at least one encoded audio signal and encoded spatial audio signal parameter values ​​associated with the at least one encoded audio signal; decode the at least one encoded audio signal; decode the encoded spatial audio signal parameter values ​​associated with the at least one encoded audio signal, the encoded spatial audio signal parameter values ​​being distributed in a time-frequency domain, wherein the components configured to decode the encoded spatial audio signal parameter values ​​associated with the at least one encoded audio signal are configured to: separate a larger number of spatial audio signal parameter values ​​from the encoded spatial audio signal parameter values ​​in time and / or frequency in the time-frequency domain.

[0019] A component configured to separate a larger number of spatial audio signal parameter values ​​from encoded spatial audio signal parameter values ​​in the time and / or frequency domain can be configured to: identify previous merging of spatial audio signal parameter values ​​in the time and / or frequency domain, and based on the identification, separate a larger number of spatial audio signal parameter values ​​from encoded spatial audio signal parameter values ​​in the time and / or frequency domain.

[0020] At least one encoded spatial audio signal may further include at least one indicator associated with a previous merge, wherein a component configured to identify a previous merge of spatial audio signal parameter values ​​in time and / or frequency, and based on the identification, to separate a larger number of spatial audio signal parameter values ​​from the encoded spatial audio signal parameter values ​​in time and / or frequency in the time-frequency domain, may be configured to: based on at least one indicator, and based on the identification, separate a larger number of spatial audio signal parameter values ​​from the encoded spatial audio signal parameter values ​​in time and / or frequency in the time-frequency domain.

[0021] According to a third aspect, a method is provided, the method comprising: acquiring at least one audio signal; for the at least one audio signal, acquiring spatial audio signal parameter values ​​distributed in a time-frequency domain; determining a merging metric to control the merging of the spatial audio signal parameter values ​​in the time-frequency domain; and based on the merging metric, merging the spatial audio signal parameter values ​​into a smaller number of spatial audio signal parameter values ​​in the time-frequency domain in terms of time and / or frequency.

[0022] Determining a merging metric to control the merging of spatial audio signal parameter values ​​in the time-frequency domain may include: determining an initial metric for detecting the start of a sound event.

[0023] Determining the starting metric may include: determining an energy parameter for at least one audio signal within a time period; determining a slow audio signal envelope based on the energy parameter and a slow decay time; determining a fast audio signal envelope based on the energy parameter and a fast decay time; and determining the starting metric based on the slow audio signal envelope and the fast audio signal envelope.

[0024] Merging spatial audio signal parameter values ​​into a smaller number of spatial audio signal parameter values ​​in the time and / or frequency domain based on a merging metric may include: determining the spatial audio signal parameter value band that best represents the spatial audio signal parameter value band within a time period when the initiation metric indicates the start of a sound event.

[0025] Merging spatial audio signal parameter values ​​into a smaller number of spatial audio signal parameter values ​​in the time and / or frequency domain based on merging metrics may include: for a determined frequency band of the spatial audio signal parameter value, determining whether the energy ratio of the frequency band is greater than the weighted average of the energy ratios of the frequency bands within a time period; and when the energy ratio of the determined frequency band of the spatial audio signal parameter value is greater than the weighted average of the energy ratios of the frequency bands within a time period, merging the spatial audio signal parameter values ​​into a smaller number of spatial audio signal parameter values ​​in the frequency domain.

[0026] Merging spatial audio signal parameter values ​​into a smaller number of spatial audio signal parameter values ​​in the time and / or frequency domain based on merging metrics may include: when the energy ratio of the determined frequency band of the spatial audio signal parameter value is less than the weighted average of the energy ratios of the frequency bands within the time period, merging the spatial audio signal parameter values ​​into a smaller number of spatial audio signal parameter values ​​in time.

[0027] Merging spatial audio signal parameter values ​​into a smaller number of spatial audio signal parameter values ​​in the time-frequency domain based on merging metrics may include: merging spatial audio signal parameter values ​​into a smaller number of spatial audio signal parameter values ​​in time when the initiation metric indicates the absence of the start of a sound event.

[0028] The method may also include encoding the parameter values ​​of the merged spatial audio signal.

[0029] Encoding the combined spatial audio signal parameter values ​​can include quantizing the combined spatial audio signal parameter values.

[0030] Encoding the parameter values ​​of the merged spatial audio signal can include entropy coding of the merged spatial audio signal parameter values.

[0031] According to a fourth aspect, a method is provided, the method comprising: acquiring at least one encoded spatial audio signal, the at least one encoded spatial audio signal comprising: at least one encoded audio signal and encoded spatial audio signal parameter values ​​associated with the at least one encoded audio signal; decoding the at least one encoded audio signal; decoding the encoded spatial audio signal parameter values ​​associated with the at least one encoded audio signal, the encoded spatial audio signal parameter values ​​being distributed in a time-frequency domain, the decoding of the encoded spatial audio signal parameter values ​​associated with the at least one encoded audio signal comprising: separating a larger number of spatial audio signal parameter values ​​from the encoded spatial audio signal parameter values ​​in time and / or frequency in the time-frequency domain.

[0032] Separating a larger number of spatial audio signal parameter values ​​from encoded spatial audio signal parameter values ​​in the time and / or frequency domain can include: identifying previous merging of spatial audio signal parameter values ​​in the time and / or frequency domain, and based on the identification, separating a larger number of spatial audio signal parameter values ​​from encoded spatial audio signal parameter values ​​in the time and / or frequency domain.

[0033] At least one encoded spatial audio signal may further include at least one indicator associated with a previous merge, wherein the previous merge of spatial audio signal parameter values ​​in time and / or frequency is identified, and based on the identification, separating a larger number of spatial audio signal parameter values ​​from the encoded spatial audio signal parameter values ​​in time and / or frequency within the time-frequency domain may include: based on at least one indicator, while based on the identification, separating a larger number of spatial audio signal parameter values ​​from the encoded spatial audio signal parameter values ​​in time and / or frequency within the time-frequency domain.

[0034] According to a fifth aspect, an apparatus is provided, comprising at least one processor and at least one memory including computer program code, the at least one memory and the computer program code being configured together with the at least one processor such that the apparatus at least: acquires at least one audio signal; for the at least one audio signal, acquires spatial audio signal parameter values ​​distributed in a time-frequency domain; determines a merging metric to control the merging of the spatial audio signal parameter values ​​in the time-frequency domain; and based on the merging metric, merges the spatial audio signal parameter values ​​into a smaller number of spatial audio signal parameter values ​​in the time-frequency domain in terms of time and / or frequency.

[0035] The means by which a merging metric is determined to control the merging of spatial audio signal parameter values ​​in the time-frequency domain can be made to: determine an initial metric for detecting the start of a sound event.

[0036] The means for determining the initial metric can be made to: determine an energy parameter for at least one audio signal within a time period; determine a slow audio signal envelope based on the energy parameter and a slow decay time; determine a fast audio signal envelope based on the energy parameter and a fast decay time; and determine the initial metric based on the slow audio signal envelope and the fast audio signal envelope.

[0037] An apparatus that enables the merging of spatial audio signal parameter values ​​into a smaller number of spatial audio signal parameter values ​​in the time and / or frequency domain based on a merging metric can be made to: determine the spatial audio signal parameter value band that best represents the spatial audio signal parameter value band within a time period when the initiation metric indicates the start of a sound event.

[0038] An apparatus that enables the merging of spatial audio signal parameter values ​​into a smaller number of spatial audio signal parameter values ​​in the time and / or frequency domain based on a merging metric can be configured to: determine, for a determined frequency band of the spatial audio signal parameter values, whether the energy ratio of the frequency band is greater than the weighted average of the energy ratios of the frequency bands within a time period; and when the energy ratio of the determined frequency band of the spatial audio signal parameter values ​​is greater than the weighted average of the energy ratios of the frequency bands within a time period, merge the spatial audio signal parameter values ​​into a smaller number of spatial audio signal parameter values ​​in the frequency domain.

[0039] An apparatus that enables the merging of spatial audio signal parameter values ​​into a smaller number of spatial audio signal parameter values ​​in the time-frequency domain based on a merging metric can be configured to merge spatial audio signal parameter values ​​into a smaller number of spatial audio signal parameter values ​​in time when the energy ratio of the determined frequency band of the spatial audio signal parameter value is less than the weighted average of the energy ratios of the frequency bands in the time period.

[0040] An apparatus that enables the merging of spatial audio signal parameter values ​​into a smaller number of spatial audio signal parameter values ​​in the time and / or frequency domain based on a merging metric can be made to merge spatial audio signal parameter values ​​into a smaller number of spatial audio signal parameter values ​​in time when the initiation metric indicates the absence of the start of a sound event.

[0041] The device can also be used to encode the parameter values ​​of the merged spatial audio signal.

[0042] The apparatus that enables encoding of the combined spatial audio signal parameter values ​​can also enable quantization of the combined spatial audio signal parameter values.

[0043] The apparatus that enables encoding of the combined spatial audio signal parameter values ​​can enable entropy encoding of the combined spatial audio signal parameter values.

[0044] According to a sixth aspect, an apparatus is provided, comprising at least one processor and at least one memory including computer program code, the at least one memory and the computer program code being configured, together with the at least one processor, such that the apparatus at least: acquires at least one encoded spatial audio signal, the at least one encoded spatial audio signal including at least one encoded audio signal and encoded spatial audio signal parameter values ​​associated with the at least one encoded audio signal; decodes the at least one encoded audio signal; decodes the encoded spatial audio signal parameter values ​​associated with the at least one encoded audio signal, the encoded spatial audio signal parameter values ​​being distributed in a time-frequency domain, the apparatus for decoding the encoded spatial audio signal parameter values ​​associated with the at least one encoded audio signal being configured to: separate a larger number of spatial audio signal parameter values ​​from the encoded spatial audio signal parameter values ​​in time and / or frequency in the time-frequency domain.

[0045] An apparatus that enables the separation of a larger number of spatial audio signal parameter values ​​from encoded spatial audio signal parameter values ​​in the time and / or frequency domain can be made to: identify a previous merging of spatial audio signal parameter values ​​in the time and / or frequency domain, and based on the identification, separate a larger number of spatial audio signal parameter values ​​from encoded spatial audio signal parameter values ​​in the time and / or frequency domain.

[0046] At least one encoded spatial audio signal may further include at least one indicator associated with a previous merging, wherein means for identifying the previous merging of spatial audio signal parameter values ​​in time and / or frequency, and for separating a larger number of spatial audio signal parameter values ​​from the encoded spatial audio signal parameter values ​​in the time and / or frequency domain based on the identification, may be made to: separate a larger number of spatial audio signal parameter values ​​from the encoded spatial audio signal parameter values ​​in the time and / or frequency domain based on at least one indicator, and based on the identification.

[0047] According to a seventh aspect, an apparatus is provided, comprising: means for acquiring at least one audio signal; means for acquiring spatial audio signal parameter values ​​for the at least one audio signal, the spatial audio signal parameter values ​​being distributed in a time-frequency domain; means for determining a merging metric to control the merging of the spatial audio signal parameter values ​​in the time-frequency domain; and means for merging the spatial audio signal parameter values ​​into a smaller number of spatial audio signal parameter values ​​in the time-frequency domain based on the merging metric.

[0048] According to an eighth aspect, an apparatus is provided, the apparatus comprising: means for acquiring at least one encoded spatial audio signal, the at least one encoded spatial audio signal comprising: at least one encoded audio signal and encoded spatial audio signal parameter values ​​associated with the at least one encoded audio signal; means for decoding the at least one encoded audio signal; and means for decoding the encoded spatial audio signal parameter values ​​associated with the at least one encoded audio signal, the encoded spatial audio signal parameter values ​​being distributed in a time-frequency domain, the means for decoding the encoded spatial audio signal parameter values ​​associated with the at least one encoded audio signal being configured to: separate a larger number of spatial audio signal parameter values ​​from the encoded spatial audio signal parameter values ​​in time and / or frequency in the time-frequency domain.

[0049] According to a ninth aspect, a computer program [or a computer-readable medium including program instructions] is provided, the instructions being configured to cause a device to perform at least the following operations: acquiring at least one audio signal; for the at least one audio signal, acquiring spatial audio signal parameter values ​​distributed in a time-frequency domain; determining a merging metric to control the merging of the spatial audio signal parameter values ​​in the time-frequency domain; and merging the spatial audio signal parameter values ​​into a smaller number of spatial audio signal parameter values ​​in the time-frequency domain based on the merging metric in time and / or frequency.

[0050] According to a tenth aspect, a computer program [or a computer-readable medium including program instructions] is provided, the instructions being configured to cause an apparatus to perform at least the following operations: acquiring at least one encoded spatial audio signal, the at least one encoded spatial audio signal comprising: at least one encoded audio signal and encoded spatial audio signal parameter values ​​associated with the at least one encoded audio signal; decoding the at least one encoded audio signal; decoding the encoded spatial audio signal parameter values ​​associated with the at least one encoded audio signal, the encoded spatial audio signal parameter values ​​being distributed in a time-frequency domain, the decoding of the encoded spatial audio signal parameter values ​​associated with the at least one encoded audio signal comprising: separating a larger number of spatial audio signal parameter values ​​from the encoded spatial audio signal parameter values ​​in time and / or frequency in the time-frequency domain.

[0051] According to an eleventh aspect, a non-transitory computer-readable medium is provided, comprising program instructions for causing a device to perform at least the following operations: acquiring at least one audio signal; for the at least one audio signal, acquiring spatial audio signal parameter values ​​distributed in a time-frequency domain; determining a merging metric to control the merging of the spatial audio signal parameter values ​​in the time-frequency domain; and merging the spatial audio signal parameter values ​​into a smaller number of spatial audio signal parameter values ​​in the time-frequency domain based on the merging metric in time and / or frequency.

[0052] According to a twelfth aspect, a non-transitory computer-readable medium is provided, the non-transitory computer-readable medium comprising program instructions for causing a device to perform at least the following operations: acquiring at least one encoded spatial audio signal, the at least one encoded spatial audio signal including at least one encoded audio signal and encoded spatial audio signal parameter values ​​associated with the at least one encoded audio signal; decoding the at least one encoded audio signal; decoding the encoded spatial audio signal parameter values ​​associated with the at least one encoded audio signal, the encoded spatial audio signal parameter values ​​being distributed in a time-frequency domain, wherein the means for causing the decoding of the encoded spatial audio signal parameter values ​​associated with the at least one encoded audio signal includes: separating a larger number of spatial audio signal parameter values ​​from the encoded spatial audio signal parameter values ​​in time and / or frequency in the time-frequency domain.

[0053] According to a thirteenth aspect, an apparatus is provided, comprising: an acquisition circuit system configured to acquire at least one audio signal; an acquisition circuit system configured to acquire spatial audio signal parameter values ​​for the at least one audio signal, the spatial audio signal parameter values ​​being distributed in a time-frequency domain; a determination circuit system configured to determine a merging metric to control the merging of the spatial audio signal parameter values ​​in the time-frequency domain; and a merging circuit system configured to merge the spatial audio signal parameter values ​​into a smaller number of spatial audio signal parameter values ​​in the time-frequency domain based on the merging metric.

[0054] According to a fourteenth aspect, an apparatus is provided, comprising: an acquisition circuit system configured to acquire at least one encoded spatial audio signal, the at least one encoded spatial audio signal comprising: at least one encoded audio signal and encoded spatial audio signal parameter values ​​associated with the at least one encoded audio signal; a decoding circuit system configured to decode the at least one encoded audio signal; and a decoding circuit system configured to decode the encoded spatial audio signal parameter values ​​associated with the at least one encoded audio signal, the encoded spatial audio signal parameter values ​​being distributed in a time-frequency domain, wherein the decoding circuit system configured to decode the encoded spatial audio signal parameter values ​​associated with the at least one encoded audio signal includes: a separation circuit system configured to separate a larger number of spatial audio signal parameter values ​​from the encoded spatial audio signal parameter values ​​in time and / or frequency within the time-frequency domain.

[0055] According to a fifteenth aspect, a computer-readable medium is provided, comprising program instructions for causing a device to perform at least the following operations: acquiring at least one audio signal; for the at least one audio signal, acquiring spatial audio signal parameter values ​​distributed in a time-frequency domain; determining a merging metric to control the merging of the spatial audio signal parameter values ​​in the time-frequency domain; and merging the spatial audio signal parameter values ​​into a smaller number of spatial audio signal parameter values ​​in the time-frequency domain based on the merging metric in time and / or frequency.

[0056] According to a sixteenth aspect, a computer-readable medium is provided, comprising program instructions for causing a device to perform at least the following operations: acquiring at least one encoded spatial audio signal, the at least one encoded spatial audio signal comprising: at least one encoded audio signal and encoded spatial audio signal parameter values ​​associated with the at least one encoded audio signal; decoding the at least one encoded audio signal; decoding the encoded spatial audio signal parameter values ​​associated with the at least one encoded audio signal, the encoded spatial audio signal parameter values ​​being distributed in a time-frequency domain, the decoding of the encoded spatial audio signal parameter values ​​associated with the at least one encoded audio signal comprising: separating a larger number of spatial audio signal parameter values ​​from the encoded spatial audio signal parameter values ​​in time and / or frequency in the time-frequency domain.

[0057] An apparatus includes components for performing the actions described above.

[0058] An apparatus is configured to perform the actions described above.

[0059] A computer program includes program instructions for causing a computer to perform the methods described above.

[0060] A computer program product stored on a medium can cause a device to perform the methods described herein.

[0061] An electronic device may include the means as described herein.

[0062] A chipset may include devices as described herein.

[0063] The embodiments of this application are intended to solve problems associated with the prior art. Attached Figure Description

[0064] To better understand this application, reference will now be made to the accompanying drawings by way of example, in which:

[0065] Figure 1 A system suitable for implementing some embodiments is illustrated schematically;

[0066] Figure 2 A metadata encoder according to some embodiments is illustrated schematically;

[0067] Figure 3 Examples of some embodiments are shown. Figure 2 The flowchart shown illustrates the operation of an example metadata encoder.

[0068] Figure 4 The illustration shows, for example, some embodiments. Figure 2 The starting determiner shown;

[0069] Figure 5 Examples of some embodiments are shown. Figure 4 The flowchart of the operation of the starter determinant is shown below;

[0070] Figure 6 The illustration shows, for example, some embodiments. Figure 2 The frequency band selector shown;

[0071] Figure 7 and Figure 8 Examples of some embodiments are shown. Figure 6 The flowchart shown illustrates the operation of the frequency band selector; and

[0072] Figure 9 An example device suitable for implementing the illustrated apparatus is shown schematically. Detailed Implementation

[0073] The following describes in further detail suitable apparatus and possible mechanisms for encoding parameterized spatial audio streams using transmitted audio signals and spatial metadata. In the following discussion, multi-channel systems are discussed in relation to multi-channel microphone implementations. However, as mentioned above, the input format can be any suitable input format, such as multi-channel speakers, Ambisonics (FOA / HOA), etc. It should be understood that in some embodiments, channel positions are based on the microphone's location or a virtual location or orientation.

[0074] Furthermore, in the following examples, the output of the example system is a multi-channel speaker arrangement. In other embodiments, the output may be rendered to the user via components other than speakers. The multi-channel speaker signal may also be summarized as two or more playback audio signals.

[0075] Metadata-Assisted Spatial Audio (MASA) is a parametric spatial audio format and representation. It can be thought of as an audio representation consisting of "N channels + spatial metadata". This is a scene-based audio format, particularly well-suited for spatial audio capture on practical devices such as smartphones. The idea is to describe a sound scene based on the direction of the sound source, which varies in time and frequency. Sound energy not defined (described) by direction is described as diffusion (from all directions).

[0076] As described above, spatial metadata associated with an audio signal can include multiple parameters for each time-frequency tile (such as multiple directions and the direct-to-total ratio, distance, etc. associated with each direction). Spatial metadata can also include other parameters, or can be associated with other parameters considered non-directional (such as surround coherence, diffusion-to-total energy ratio, residual-to-total energy ratio), but when combined with directional parameters, can be used to define the characteristics of the audio scene. For example, a reasonable design choice that produces high-quality output is one where the spatial metadata includes one or more directions (and the direct-to-total ratio, distance values, etc. associated with each direction) for each time-frequency subframe. However, as mentioned above, bandwidth and / or storage limitations may require the codec not to send spatial metadata parameter values ​​for each frequency band and time subframe.

[0077] As mentioned above, parameterized spatial metadata representation can utilize multiple concurrent spatial directions. For MASA, the proposed maximum number of concurrent directions is two. For each concurrent direction, there can be associated parameters such as: direction index; direct to total ratio; propagation coherence; diffusion to total energy ratio; circumferential coherence; residual to total energy ratio; and distance.

[0078] The direction index can be encoded using multiple bits, such as 16 bits, which defines the direction of arrival of sound at time-frequency parameter intervals. In some embodiments, encoding with a 16-bit spherical representation enables direction accuracy of approximately 1 degree, where all directions are covered. The direct-to-total ratio describes how much energy comes from a particular direction and can be calculated as the energy in that direction relative to the total energy. Propagation coherence represents the propagation of energy associated with the direction index of the time-frequency tile (i.e., a measure of the "energy concentration" in the direction of the time-frequency subframe and defines whether that direction is to be reproduced as a point source or coherently reproduced around that direction). The diffusion-to-total energy ratio defines the energy ratio of non-directional sound over surrounding directions and can be calculated as the energy of non-directional sound relative to the total energy, describing how much energy does not come from any particular direction. The sum of the (multiple) direct-to-total energy ratios and the diffusion-to-total energy ratio is 1 (if no residual energy exists). The surround coherence describes the coherence of non-directional sound in surrounding directions. The residual-to-total energy ratio defines the energy ratio of residual (such as microphone noise) acoustic energy and satisfies the requirement that the sum of the energy ratios is 1. The “Distance” parameter defines the distance of the sound originating from this direction index. It can be defined based on time frequency subframes and in meters on a logarithmic scale, and the range of values ​​can be defined, for example, from 0 to 100m.

[0079] However, the MASA format can also include other parameters, such as:

[0080] Version, which describes the incremental version number of the MASA metadata format.

[0081] The channel audio format describes the following fields (and can be stored as two bytes):

[0082] The number of directions, which indicates the number of directions in the metadata, where each direction is associated with a set of direction-related spatial metadata;

[0083] The number of channels, which indicates the number of transmission channels in the specified format;

[0084] The transmission channel definition describes the transmission channel.

[0085] Source format, which describes the original format used to create the audio signal;

[0086] Source format description, which can provide further description of a specific source format; and

[0087] Channel distance, which describes the distance between channels.

[0088] The IVAS codec is an extension of the 3GPP EVS codec and is designed to provide new immersive voice and audio services over 4G / 5G. These immersive services include, for example, immersive voice and audio for virtual reality (VR). The multi-purpose audio codec is expected to handle the encoding, decoding, and rendering of voice, music, and general audio. It is expected to support channel-based and scene-based audio inputs, including spatial information about the sound field and sound sources. It is also expected to enable low-latency operation for session services and support high error robustness under various transmission conditions.

[0089] Because the IVAS codec is expected to operate at a wide range of bit rates, from very low (13 kb / s) to relatively high (500 kb / s), various strategies are required to compress spatial metadata. The raw bit rate of MASA metadata is relatively high (approximately 310 kb / s for one direction and approximately 500 kb / s for two directions), so at lower bit rates, it is expected that only the most critical parts of the metadata will be transmitted from the encoder to the decoder. In practice, it is not possible to send parameter values ​​for every frequency band, time subframe, and direction (at least for most practical bit rates). Instead, some values ​​must be combined (e.g., sending only one direction instead of two, and / or sending (multiple) of the same direction for multiple frequency bands and / or time subframes). At the absolute lowest bit rates, a significant reduction is required due to the very limited number of bits available to describe the metadata.

[0090] For example, at very low audio bitrates (13.2 kb / s to 32 kb / s), very few bits are available for encoding metadata. For instance, at 16.4 kb / s stereo MASA, the available bitrate for metadata might be as low as 3 kb / s to maintain the quality of (multiple) audio signals or (multiple) transmission signals. Since even the raw bitrate of MASA metadata in one direction is approximately 310 kb / s, this reduction is significant.

[0091] While the number of frequency bands and subframes can be reduced to a low number, even with reasonable precision and TF resolution (e.g., 5 frequency bands and 4 subframes, i.e., 20 time-frequency tiles) transmitting only the directional and direct-to-total-energy ratio parameters, encoding these parameters (depending on the metadata values) to approximately 60 bits per frame at the aforementioned bit rate may not always provide good quality, depending on the content of the spatial audio. Apparatus and methods for obtaining these significant reductions without loss of quality are currently being investigated. The concept discussed in the following embodiments is to provide apparatus and methods configured to control and select reduction methods for each metadata frame in order to obtain a good quality output.

[0092] Therefore, for example, in some embodiments, apparatus and methods are provided configured to select between a single subframe and a single-band metadata representation. In some embodiments, the control mechanism or selection is based on an initial detection or determination operation. The initial detection or determination operation is implemented based on a formation metric that can be compared with a threshold. In some embodiments, the metric itself may be formed based on the analysis of parameterized parameters, such as (multiple) directly related to the total energy ratio and (multiple) signal energy.

[0093] about Figure 1 Example apparatus and system for implementing embodiments of this application are shown. System 100 is shown having an "analysis" section 121 and a "synthesis" section 131. The "analysis" section 121 is the section from receiving multi-channel signals to encoding spatial metadata and transmission signals, and the "synthesis" section 131 is the section from decoding the encoded spatial metadata and transmission signals to the presentation of the regenerated signal (e.g., in the form of a multi-channel loudspeaker).

[0094] In the following description, the “analysis” section 121 is described as a series of parts; however, in some embodiments, this section may be implemented as a function of the same functional means or within a section. In other words, in some embodiments, the “analysis” section 121 is an encoder that includes at least one of the following: a transmission signal generator or an analysis processor.

[0095] The input to system 100 and the "analysis" section 121 is a multi-channel signal 102. The "analysis" section 121 may include a transmission signal generator 103, an analysis processor 105, and an encoder 107. In the following example, a microphone channel signal input is described, which may be two or more microphones integrated into or connected to a mobile device (e.g., a smartphone). However, in other embodiments, any suitable input (or synthesized multi-channel) format can be implemented. For example, other suitable audio signal format inputs may be microphone arrays, such as B-format microphones, planar microphone arrays, or Eigenmike, Ambisonic signals (e.g., first-order Ambisonics (FOA), higher-order Ambisonics (HOA)), speaker surround mixes and / or objects, artificially created spatial mixes (e.g., from an audio or VR teleconferencing bridge), or combinations of the above.

[0096] The multi-channel signal is transmitted to the transmission signal generator 103 and the analysis processor 105.

[0097] In some embodiments, the transmission signal generator 103 is configured to receive multi-channel signals and generate a suitable audio signal format for encoding. The transmission signal generator 103 may, for example, generate stereo or single-channel audio signals. The transmission audio signal generated by the transmission signal generator can be any known format. For example, when the input is an audio signal from a mobile phone microphone array, the transmission signal generator 103 may be configured to select a left and right microphone pair and apply any suitable processing to the audio signal pair, such as automatic gain control, microphone noise removal, wind noise removal, and equalization. In some embodiments, when the input is a first-order Ambisonic / higher-order Ambisonic (FOA / HOA) signal, the transmission signal generator may be configured to generate directional beam signals directed to the left and right, such as two opposing cardioid signals. Furthermore, in some embodiments, when the input is a speaker surround mix and / or an object, the transmission signal generator 103 can be configured to generate a downmix signal that combines the left channel into a left downmix channel, combines the right channel into a right downmix channel, and adds the center channel to both transmission channels with appropriate gain.

[0098] In some embodiments, the transmission signal generator is bypassed (or, in other words, optional). For example, in some cases where analysis and synthesis occur at the same device in a single processing step, no transmission signal is generated without intermediate processing, and the input audio signal is transmitted unprocessed. The number of transmission channels generated can be any suitable number, rather than, for example, one or two channels.

[0099] The output of the transmission signal generator 103 can be transmitted to the encoder 107.

[0100] In some embodiments, the analysis processor 105 is further configured to receive the multichannel signal and analyze the signal to generate spatial metadata 106 associated with the multichannel signal and therefore with the transmission signal 104. In some embodiments, the spatial metadata associated with the audio signal may be provided to the encoder as a separate bitstream. In some embodiments, the multichannel signal 102 input includes spatial metadata and is passed directly to the encoder 107.

[0101] The analysis processor 105 can be configured to generate spatial metadata parameters, which, for each time-frequency analysis interval, may include at least one orientation parameter 108 and at least one energy ratio parameter 110 (and in some embodiments, may also include other parameters such as those previously described, and a non-exhaustive list thereof includes the number of orientations, circumferential coherence, diffusion-to-total energy ratio, residual-to-total energy ratio, propagation coherence parameter, and distance parameter). The orientation parameter can be represented in any suitable manner, for example, as spherical coordinates, which are expressed as azimuth angles. And the elevation angle θ(k,n).

[0102] In some embodiments, the number of spatial metadata parameters may vary depending on the time-frequency tile. Thus, for example, in band X, all spatial metadata parameters are acquired (generated) and transmitted, while in band Y, only one of the spatial metadata parameters is acquired and transmitted; furthermore, in band Z, no parameters are acquired or transmitted. A practical example of this could be that for some time-frequency tiles corresponding to the highest frequency band, some spatial metadata parameters are unnecessary for perceptual reasons. Spatial metadata 106 can be passed to encoder 107.

[0103] In some embodiments, the analysis processor 105 is configured to apply a time-frequency transformation to the input signal. Then, for example, in a time-frequency tile, when the input is a mobile phone microphone array, the analysis processor can be configured to estimate a delay value between microphone pairs that maximizes the inter-microphone correlation. Based on these delay values, the analysis processor can then be configured to determine corresponding orientation values ​​for spatial metadata. Furthermore, the analysis processor can be configured to determine a direct-to-total ratio parameter based on the correlation values.

[0104] In some embodiments, for example, when the input is a FOA signal, the analysis processor 105 can be configured to determine an intensity vector. The analysis processor can then be configured to determine the direction parameter values ​​of the spatial metadata based on the intensity vector. The spread-to-total ratio can then be determined, thereby determining the direct-to-total ratio parameter values ​​of the spatial metadata. This analysis method is referred to in the literature as Directional Audio Coding (DirAC).

[0105] In some examples, such as when the input is a HOA signal, the analysis processor 105 can be configured to divide the HOA signal into multiple sectors, using the method described above in each of these sectors. This sector-based approach is referred to in the literature as Higher-Order DirAC (HO-DirAC). In these examples, each time-frequency tile corresponding to the multiple sectors has more than one simultaneous directional parameter value.

[0106] Furthermore, in some embodiments where the input is a speaker surround mix and / or an audio object-based signal, the analysis processor can be configured to convert the signal into a FOA / HOA signal format and obtain the direction and direct-to-total ratio parameter values ​​as described above.

[0107] Encoder 107 may include an audio encoder core 109 configured to receive transmitted audio signals 104 and generate suitable encodings of these audio signals. In some embodiments, encoder 107 may be a computer (running suitable software stored in memory and at least one processor), or alternatively, a specific device utilizing, for example, an FPGA or ASIC. Audio encoding may be implemented using any suitable scheme.

[0108] Encoder 107 may further include a spatial metadata encoder / quantizer 111 configured to receive spatial metadata and output information in an encoded or compressed form. In some embodiments, encoder 107 may further interleave the spatial metadata, multiplex the spatial metadata into a single data stream, or embed the spatial metadata into an encoded downmixer signal before transmission or storage, such as... Figure 1 As shown by the dashed line in the diagram. Multiplexing can be implemented using any suitable scheme.

[0109] In some embodiments, the transmission signal generator 103 and / or analysis processor 105 may be located on a device separate from the encoder 107 (or otherwise separated). For example, in such embodiments, spatial metadata (and associated non-spatial metadata) parameters associated with the audio signal may be provided to the encoder as a separate bitstream.

[0110] In some embodiments, the transmission signal generator 103 and / or analysis processor 105 may be part of the encoder 107, i.e., located inside the encoder and on the same device.

[0111] In the following description, the “composite” portion 131 is described as a series of portions; however, in some embodiments, the portion may be implemented as the same functional device or a function within the portion.

[0112] On the decoder side, the received or retrieved data (stream) can be received by decoder / demultiplexer 133. Decoder / demultiplexer 133 can demultiplex the encoded stream and pass the audio encoded stream to transport signal decoder 135, which is configured to decode the audio signal to obtain the transport audio signal. Similarly, decoder / demultiplexer 133 may include metadata extractor 137, which is configured to receive encoded spatial metadata (e.g., a direction index representing a direction parameter value) and generate spatial metadata.

[0113] In some embodiments, the decoder / demultiplexer 133 may be a computer (running suitable software stored in memory and at least one processor), or alternatively, a specific device utilizing, for example, an FPGA or ASIC.

[0114] The decoded metadata and transmitted audio signals can be passed to the synthesis processor 139.

[0115] The “synthesis” section 131 of system 100 also shows a synthesis processor 139, which is configured to receive transmitted audio signals and spatial metadata, and recreate the synthesized spatial audio in the form of a multi-channel signal 110 in any suitable format (these may be multi-channel speaker formats, or in some embodiments, depending on the use case, any suitable output format, such as binaural or binaural signals).

[0116] The synthesis processor 139 therefore creates the output audio signal based on any suitable known method, such as a multi-channel speaker signal or a binaural signal. This will not be explained in detail here. However, as a simplified example, the speaker output can be rendered according to any of the following methods. For example, the transmitted audio signal can be divided into a direct stream and an ambient stream based on the direct-to-total energy ratio and the diffusion-to-total energy ratio. The direct stream can then be rendered using amplitude translation based on (multiple) directional parameters. The ambient stream can be further rendered using decorrelation. The direct stream and the ambient stream can then be combined.

[0117] The output signal can be reproduced using a multi-channel speaker setup or headphones.

[0118] It should be noted that Figure 1 The processing blocks can reside in the same or different processing entities. For example, in some embodiments, microphone signals from a mobile device are processed using a spatial audio capture system (including an analysis processor and a transmission signal generator), and the resulting spatial metadata and transmission audio signals (e.g., in the form of a MASA stream) are forwarded to an encoder (e.g., an IVAS encoder) that includes the aforementioned encoder. In other embodiments, input signals (e.g., 5.1-channel audio signals) are forwarded directly to an encoder (e.g., an IVAS encoder) that includes the aforementioned analysis processor, transmission signal generator, and encoder.

[0119] In some embodiments, there may be two (or more) input audio signals, wherein the first audio signal is generated by... Figure 1The apparatus shown processes (to generate data as input to the encoder), and the second audio signal is directly forwarded to the encoder (e.g., an IVAS encoder), which includes the aforementioned analysis processor, transmission signal generator, and encoder. The audio input signal can then be encoded independently in the encoder, or it can be combined in the parameter domain, for example, according to so-called MASA mixing.

[0120] In some embodiments, a compositing portion may exist, comprising separate decoder and compositing processor entities or devices, or the compositing portion may comprise a single entity comprising both a decoder and a compositing processor. In some embodiments, the decoder block may process more than the above input data streams in parallel. In applications, the term compositing processor may be interpreted as an internal or external renderer.

[0121] Therefore, in general, firstly, the system (analysis section) is configured to receive multi-channel audio signals. Then, the system (analysis section) is configured to generate suitable transmission audio signals (e.g., by selecting some of the audio signal channels). Next, the system is configured to encode the transmission audio signals for storage / transmission. After this, the system can store / transmit the encoded transmission audio signals and metadata. The system can retrieve / receive the encoded transmission audio signals and metadata. Then, the system is configured to extract transmission audio signals and metadata parameters from the encoded transmission audio signals and metadata parameters, such as by demultiplexing and decoding the encoded transmission audio signals and metadata parameters.

[0122] The system (synthesis part) is configured to synthesize the output multi-channel audio signal based on the extracted transmitted audio signal and metadata.

[0123] about Figure 2 Further detailed description of an example spatial metadata encoder / quantizer 111 according to some embodiments (e.g. Figure 1 (As shown).

[0124] In some embodiments, the input to the metadata encoder / quantizer 111 includes spatial metadata 106 and energy parameters. In other words, the spatial metadata (containing at least the direct total energy ratio r(k,n)) and the energy E(k,n) are acquired at the same resolution as the metadata (k is the frequency band index, and n is the time subframe index).

[0125] In some embodiments, the energy E(k,n) can already be calculated in the analysis processor 105 based on the time-frequency domain multichannel signal as follows.

[0126]

[0127] Because this process is designed for very low bit rates, the spatial metadata can already be in a relatively low-resolution form. For example, in some embodiments, the spatial metadata is formatted as spatial metadata parameters, which are parameters for each directional component across 5 frequency bands and 4 subframes. In some embodiments, the energy ratio (such as the direct-to-total ratio) can already be represented in full-frame temporal resolution instead of 4-subframe resolution. In other words, for the energy ratio, there is a single parameter value for the entire frame, rather than individual subframe parameter values.

[0128] In some embodiments, the input spatial metadata and energy parameters can be passed to the spatial metadata reduction optimization controller (or more generally referred to as the controller) 201.

[0129] In some embodiments, controller 201 includes a start determiner 211. The start determiner 211 is configured to determine when the short-term energy is significantly higher than the long-term energy. The determination that the short-term energy is significantly higher than the long-term energy indicates the potential start of a sound event, and is therefore perceptually important in terms of the perceived direction and timbre of the sound event.

[0130] Without a defined starting point, the soundscape typically changes "slowly," making fast temporal resolution less important. This means that better frequency resolution can be traded for temporal resolution, and merging can be achieved temporally rather than frequency-wise.

[0131] However, when a start is established, a faster temporal resolution is important for capturing and characterizing changes in the sound scene as well as possible. In this case, frequency resolution (if the start can be represented well using only a single frequency band) can be traded for temporal resolution.

[0132] about Figure 4 and Figure 5 The operation of example start determiner 211 and example start determiner 211 are illustrated respectively. In some other embodiments, different but suitable start determiners may be implemented to detect or determine the occurrence of a start.

[0133] In some embodiments, the start determiner 211 is configured to acquire spatial metadata and energy parameters, such as Figure 5 As shown in step 501.

[0134] In some embodiments, the start determiner 211 includes a total energy determiner 401. The total energy determiner is configured to sum the energy parameters over all frequency bands and time subframes to produce the total energy of the frame (in this example, m is the index of the time frame, which contains 4 subframes).

[0135]

[0136] Then, the total energy value can be forwarded to the signal envelope determiner 403. The operation of obtaining the total energy value is as follows: Figure 5 As shown in step 503.

[0137] In some embodiments, the start determiner 211 includes a signal envelope determiner 403. The signal envelope determiner 403 is configured to determine two signal envelopes, one with a fast decay time and the other with a slow decay time. For example, the signal envelopes could be:

[0138] E α (m)=max(αE α (m-1), E tot (m))

[0139] E β (m)=γmin(E α (m), βE β (m-1)+(1-β)E α (m))

[0140] Where α and β are coefficients that determine the exponential decay rate (between 0 and 1), and γ is a gain (>1) used to prevent erroneous detection of the start of a steady signal.

[0141] In these examples, the envelope E β (m) Response to change is greater than E a (m) Slow. The envelope can be passed to the starting filter 405.

[0142] Determining the signal envelope, as follows: Figure 5 As shown in step 505.

[0143] In some embodiments, the start determiner 211 includes a start filter 405. The start filter 405 can be configured to receive an envelope and can be implemented as follows:

[0144]

[0145] The output of the start filter can then be used to determine whether a start is occurring. For example, if the start filter o(m) has a value less than 1, then frame m can be determined to contain a start. Otherwise, the frame can be determined not to contain a start. The envelope is compared to determine the start value, such as... Figure 5 Step 507 is shown in the diagram.

[0146] Then, the starting value (or defined value) can be further configured to be output, such as... Figure 5 As shown in step 509.

[0147] Determine the starting metric and further determine if there is an initial operation such as... Figure 3 As shown in step 303.

[0148] In some embodiments, controller 201 includes a frequency band selector 213. In some embodiments, frequency band selector 213 is configured to attempt to find a suitable single frequency band of spatial metadata to represent the metadata of all frequency bands when the presence of a start is determined or detected.

[0149] about Figure 6 An example band selector is shown, and in addition... Figure 7 and Figure 8 A flowchart illustrating the operation of the example band selector 213 is shown.

[0150] In some embodiments, the frequency band selector 213 is configured to acquire spatial metadata, such as... Figure 7 Step 701 is shown in the diagram.

[0151] In some embodiments, the band selector 213 includes a threshold determiner 601. In some embodiments, the threshold determiner is configured to determine a threshold w. thr For example, the threshold can be found as follows:

[0152]

[0153] The threshold is determined as follows: Figure 7 Step 703 is shown in the diagram.

[0154] In some embodiments, the band selector 213 further includes a weighting ratio determiner 603. In some embodiments, the weighting ratio determiner is configured to determine a weighting ratio for the determined band. In some embodiments, the weighting ratio is determined in order from the highest frequency band K to the lowest frequency band. In some embodiments, the weighting ratio is determined as follows:

[0155]

[0156] The operation of calculating / determining the weighting ratio is as follows: Figure 7 Step 705 is shown in the diagram.

[0157] In some embodiments, the band selector 213 further includes a comparator 605. The comparator 605 is configured to perform a weighted ratio check / band selection operation, such as... Figure 7 Step 707 is shown in the diagram.

[0158] also, Figure 8 The comparator / selection operation is shown in more detail.

[0159] The first operation is to initiate and receive input, such as weighting ratios / thresholds, etc. Figure 8Step 801 is shown in the diagram.

[0160] Then, generate or determine the threshold or weight limit w. thr ,like Figure 8 Step 802 is shown in the diagram.

[0161] The next operation is to set the index i = K (highest frequency band), such as... Figure 8 Step 803 is shown in the diagram.

[0162] Then, generate the index weight factor w(i), such as Figure 8 Step 804 is shown in the diagram.

[0163] The next step is to compare the weight limit w. thr To test the index weight factor w(i), such as Figure 8 Step 805 is shown in the diagram.

[0164] If w(i)>w thr The next operation is to determine if i is the selected frequency band, such as... Figure 8 As shown in step 809, and then the operation ends, as follows. Figure 8 Step 813 is shown in the diagram.

[0165] If w(i) is not greater than w thr The next operation is to decrement i by 1, such as... Figure 8 Step 807 is shown in the diagram.

[0166] After subtracting 1 from i, the next operation is to check if i = 1, such as... Figure 8 Step 811 is shown in the diagram.

[0167] When i = 1, the next operation is to determine that i is the selected frequency band, such as... Figure 8 As shown in step 809, and then the operation ends, as follows. Figure 8 Step 813 is shown in the diagram.

[0168] If i is not equal to 1, the operation can loop back to step 804 as shown by the arrow and generate a new weighting factor and compare it with the weight limit w. thr To test the new index weight factor w(i), this process can continue until the index w(i) > w. thr Or index = 1.

[0169] The above assumes that the frequency band index starts from 1. This can be modified to suit any other indexing system (such as starting from 0).

[0170] The operation of outputting the selection (or identifying the selection index) is as follows: Figure 7 Step 709 is shown in the diagram.

[0171] This method is based on the method described in GB1814227.3, however, it can implement any suitable single-band selection method.

[0172] The determination of the optimal frequency band for representing all frequency bands is as follows: Figure 3 As shown in step 304.

[0173] In some embodiments, controller 201 includes a ratio comparator 215. The ratio comparator 215 is configured to check whether the selected single frequency band is good enough to provide benefit in time-of-passage combining. This can be done by directly comparing the selected single frequency band b with the total ratio r. dir (b) This is achieved by directly comparing the energy-weighted average of all frequency bands with the total ratio. In some embodiments, the energy-weighted average of all frequency bands is obtained by directly comparing the total ratio with the total ratio as follows:

[0174]

[0175] Where r dir It is the direct to the total ratio, and E is energy.

[0176] In these embodiments, where the direct-to-total ratio of a selected single frequency band is higher than the average ratio (and a start exists), then the selected single frequency band should be used to represent all metadata. Otherwise, controller 201 is configured to signal that parameters for time-merging are to be used.

[0177] In other words, regarding ratio comparators

[0178] If r dir (b)>r mean Then a single-band strategy will be used.

[0179] Otherwise, use a time merging strategy.

[0180] The controller 201 can then be configured to control the subframe merging 203 and the band filter 205 to implement the determined strategy.

[0181] In some embodiments, the metadata encoder 111 includes a subframe merger 203. The subframe merger 203 may be controlled by a controller to implement (or not implement) based on the subframe merging operation described above. For example, the subframe merger 203 may be configured to merge all subframe parameters into a single (sub)frame parameter, i.e., merge parameters by time.

[0182] This can be achieved through any suitable process. For example, it can be achieved using the merging method proposed in UKIPO patent applications 1919130.3 and 1919131.1. In some embodiments, the direction and direct-to-total ratio is merged using the sum of the direction vectors above the subframe, where the vectors have been weighted with corresponding direct-to-total ratios and energy. This sum vector is then pointed toward the merging direction, and the merged direct-to-total ratio is the length of the sum vector divided by the sum energy.

[0183] In some embodiments, no additional calculation is required to combine the direct to total energy ratio, as the direct to total ratio can already be combined in time (however, the direction of combination may still need to be calculated). Alternatively, energy weighting can be used to average the ratio. In some embodiments, the subframe combiner is configured to combine other parameters (e.g., propagation coherence and surround coherence) with a direct energy-weighted average of the parameters above the subframe.

[0184] The subframe merging parameter 204 can then be output to encoder 207.

[0185] In some embodiments, the metadata encoder 111 includes a band filter 205. The band filter 205 may be controlled by a controller to implement (or not implement) parameter selection based on the above-described band selection.

[0186] For example, bandpass filter 205 can be configured to represent all frequency bands using a single selected frequency band. In other words, parameters associated with the selected frequency band can be output as parameters to be encoded by encoder 207. This can be performed, for example, as described in GB1814227.3, where it is noted that this approach can achieve better perceived quality than a simple averaging over frequency. In some embodiments, an energy-weighted direct-to-total ratio is calculated for the frequency band.

[0187] Therefore, the parameters for frequency band selection can be selected and passed to the encoder (when controlled by the controller based on the above).

[0188] Therefore, as Figure 3 As outlined in the document, in the absence of a determined / detected start, spatial metadata is temporally merged for a subframe, such as... Figure 3 As shown in step 307.

[0189] Upon determining / detecting the start, the optimal single frequency band for representing all frequency bands is determined, such as... Figure 3 As shown in step 304.

[0190] Then, individual frequency bands are tested to determine if the bandwidth ratio is higher than the weighted average ratio of all frequency bands, such as... Figure 3 As shown in step 305.

[0191] When the single-band ratio is lower than the weighted average ratio of all bands, for a subframe, spatial metadata is merged temporally, such as... Figure 3 As shown in step 307.

[0192] When the single-band ratio is higher than the weighted average ratio of all bands, single-band spatial metadata is used to represent all spatial metadata of the subframe, such as... Figure 3 As shown in step 306.

[0193] In some embodiments, the metadata encoder 111 includes encoder 207. The encoder can be any suitable metadata parameter encoder. In some embodiments, encoder 207 can be configured to perform further quantization or encoding of parameters. Therefore, the reduced amount of metadata can be quantized and encoded according to any suitable method.

[0194] In some embodiments, the encoder further generates appropriate signaling to indicate which option has been selected. This can be achieved, for example, with a single bit. The use of this low bitrate metadata reduction mode can be based on the available bit budget configured per frame, therefore it does not require explicit signaling in use, as it can be implicitly known from the codec configuration.

[0195] In some embodiments, the low bitrate metadata reduction operating mode can be determined at the decoder based on some other information or signaling. In such embodiments, the use of this low bitrate metadata reduction operating mode can be signaled or indicated to the decoder by appropriate indicators or signals. For example, signaling bits can be used to indicate whether the mode is operable, and then another signaling bit can be used to indicate which merging option is active.

[0196] The operation of encoding the reduced metadata parameters (and signaling the reduction mode) is as follows: Figure 3 As shown in step 309.

[0197] For the example input metadata above, the time-merging strategy will result in an output of 5 frequency bands and 1 subframe for encoding, while the single-band selection strategy will result in an output of 1 frequency band and 4 subframes. Therefore, the original data reduction is approximately 25% of the original metadata.

[0198] In some embodiments, in decoder 133, metadata extractor 137 is configured to determine whether a low bit rate combining system is in use. As described above, in some embodiments, this may be based on appropriate signaling or indicators received from the encoder.

[0199] When the decoder determines that a merging system is being used, signaling (bits) is decoded to determine which merging strategy is being used. Based on the determined merging strategy, the reduced metadata can be copied or separated (or de-merged) in time (for time-merging strategies) or frequency (for single-band selection strategies) to achieve the desired time-frequency resolution (e.g., 5 bands and 4 subframes). This metadata can then be used normally in presentation or output as part of the MASA format.

[0200] In implementing these embodiments, the metadata bit rate can be significantly reduced while maintaining good spatial quality, thanks to the sufficiently good quantization resolution for the remaining parameters. Furthermore, the embodiments can provide better perceived quality than using only one merging strategy.

[0201] The embodiments given above are specifically designed for low bit rates and are most efficient when the input metadata is already in a relatively low time-frequency (TF) resolution format. However, the embodiments discussed above can be applied to any input TF resolution. This also applies to the energy ratio resolution, and in some embodiments, it can be extended to a 4-subframe resolution energy ratio.

[0202] In some embodiments, other merging strategies and related metrics can be implemented in conjunction with the examples described above. The embodiments shown above introduce a simple solution that works well and does not require complex signaling and metadata codec implementations. For example, these merging strategies could be normal merging of frequencies, as well as partial merging of both time and frequency.

[0203] A single-band selection method has been proposed that selects the same frequency band for all subframes. However, it is also possible to select a different frequency band for each subframe and construct a combined single frequency band from the frequency bands separated from the subframes. In some embodiments, this can provide quality improvements.

[0204] Like most encoder-based metadata reduction processes, this process can be performed before encoding or during metadata generation (analysis operations).

[0205] about Figure 9 The illustration shows an example electronic device that can be used as an analysis or synthesis device. This device can be any suitable electronic device or apparatus. For example, in some embodiments, device 1400 is a mobile device, user equipment, tablet computer, computer, audio playback device, etc.

[0206] In some embodiments, device 1400 includes at least one processor or central processing unit 1407. Processor 1407 may be configured to execute various program codes, such as the methods described herein.

[0207] In some embodiments, device 1400 includes memory 1411. In some embodiments, at least one processor 1407 is coupled to memory 1411. Memory 1411 can be any suitable storage component. In some embodiments, memory 1411 includes a program code portion for storing program code implementable on processor 1407. Furthermore, in some embodiments, memory 1411 may also include a storage data portion for storing data, such as data that has been processed or will be processed according to the embodiments described herein. The implemented program code stored in the program code portion and the data stored in the storage data portion can be retrieved by processor 1407 via memory processor coupling when needed.

[0208] In some embodiments, device 1400 includes a user interface 1405. In some embodiments, user interface 1405 may be coupled to processor 1407. In some embodiments, processor 1407 may control the operation of user interface 1405 and receive input from user interface 1406. In some embodiments, user interface 1405 may enable a user to input commands to device 1400, for example, via a keypad. In some embodiments, user interface 1405 may enable a user to obtain information from device 1400. For example, user interface 1405 may include a display configured to display information from device 1400 to a user. In some embodiments, user interface 1405 may include a touchscreen or touch interface that enables information to be input to device 1400 and further displayed to a user of device 1400. In some embodiments, user interface 1405 may be a user interface for communicating with a location determiner as described herein.

[0209] In some embodiments, device 1400 includes an input / output port 1409. In some embodiments, input / output port 1409 includes a transceiver. In such embodiments, the transceiver may be coupled to processor 1407 and configured to enable communication with other devices or electronic devices, such as via a wireless communication network. In some embodiments, the transceiver or any suitable transceiver or transmitter and / or receiver component may be configured to communicate with other electronic devices or devices via wire or wired coupling.

[0210] The transceiver can communicate with other devices using any suitable known communication protocol. For example, in some embodiments, the transceiver may use a suitable Universal Mobile Telecommunications System (UMTS) protocol, a Wireless Local Area Network (WLAN) protocol (e.g., IEEE 802.X), a suitable short-range radio frequency communication protocol (such as Bluetooth), or an Infrared Data Communication Path (IRDA).

[0211] Transceiver input / output port 1409 can be configured to receive signals, and in some embodiments, processor 1407 executing appropriate code is used to determine the parameters described herein.

[0212] It is also noted in this document that, although exemplary embodiments have been described above, several changes and modifications may be made to the disclosed solutions without departing from the scope of the invention.

[0213] Generally, various embodiments can be implemented using hardware or dedicated circuitry systems, software, logic, or any combination thereof. Some aspects of this disclosure can be implemented in hardware, while others can be implemented in firmware or software, which can be executed by a controller, microprocessor, or other computing device, although this disclosure is not limited thereto. While various aspects of this disclosure may be shown and described as block diagrams, flowcharts, or using some other illustrations, it should be clearly understood that, by way of non-limiting example, the blocks, apparatuses, systems, techniques, or methods described herein can be implemented in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0214] The term "circuit system" as used in this application may refer to one or more or all of the following:

[0215] (a) Hardware circuit implementation only (such as implementation only in analog and / or digital circuit systems) and

[0216] (b) A combination of hardware circuitry and software, such as (if applicable):

[0217] (i) A combination of (multiple) analog and / or digital hardware circuits and software / firmware, and

[0218] (ii) Any part of (multiple) hardware processors having software (including (multiple) digital signal processors, software, and (multiple) memories), which work together to enable a device (such as a mobile phone or server) to perform various functions, and

[0219] (c) (Multiple) hardware circuits and / or (multiple) processors, such as (multiple) microprocessors or a portion thereof, which require software (such as firmware) to operate, but may be absent when the software is not required to operate.

[0220] This definition of circuit system applies to all uses of the term in this application, including in any claim. As another example, as used in this application, the term circuit system also covers only the implementation of hardware circuitry or processor (or processors) or a portion thereof and its accompanying software and / or firmware.

[0221] For example, if applicable to a particular claim element, the term circuit system also covers baseband integrated circuits or processor integrated circuits for mobile devices, or similar integrated circuits in servers, cellular network devices or other computing or network devices.

[0222] Embodiments of this disclosure can be implemented by computer software executable by the data processor of a mobile device, such as in a processor entity, or by hardware, or by a combination of software and hardware. The computer software or program (also referred to as a program product, including software routines, applets, and / or macros) can be stored in any device-readable data storage medium, and they include program instructions for performing a specific task. The computer program product may include one or more computer-executable components that, when the program is run, are configured to perform the embodiment. The one or more computer-executable components may be at least one piece of software code or a portion thereof.

[0223] Furthermore, it should be noted that any block of the logic flow shown in the figure can represent a program step, or interconnected logic circuits, blocks and functions, or a combination of program steps and logic circuits, blocks and functions. Software can be stored on physical media, such as memory chips or memory blocks implemented within a processor, magnetic media such as hard disks or floppy disks, and optical media such as DVDs and their data variants, CDs. Physical media are non-transitory media.

[0224] The memory can be of any type suitable for the local technical environment and can be implemented using any suitable data storage technology, such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory, and removable memory. The data processor can be of any type suitable for the local technical environment and, by way of non-limiting example, can include one or more of the following: general-purpose computers, special-purpose computers, microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), FPGAs, gate-level circuits, and processors based on multi-core processor architectures.

[0225] The embodiments of this disclosure can be practiced in a variety of components, such as integrated circuit modules. In general, integrated circuit design is a highly automated process. Complex and powerful software tools can be used to transform logic-level designs into semiconductor circuit designs ready to be etched and formed on semiconductor substrates.

[0226] The scope of protection sought by the various embodiments of this disclosure is defined by the independent claims. Embodiments and features described in this specification that are not within the scope of the independent claims (if any) are to be interpreted as examples that aid in understanding the various embodiments of this disclosure.

[0227] The foregoing description has provided a complete and informative description of exemplary embodiments of the present disclosure by way of non-limiting example. However, various modifications and adaptations will become apparent to those skilled in the art when read in conjunction with the accompanying drawings and appended claims, given the foregoing description. Nevertheless, all such and similar modifications to the teachings of this disclosure will still fall within the scope of the invention as defined in the appended claims. In fact, other embodiments exist that may include combinations of one or more embodiments with any other embodiments previously discussed.

Claims

1. An apparatus for spatial audio coding, comprising components configured to: For at least one audio signal, spatial audio signal parameter values ​​are obtained, wherein the spatial audio signal parameter values ​​are distributed within the time frequency domain of the time period of the at least one audio signal; A merging metric is determined to control the merging of spatial audio signal parameter values ​​in the time-frequency domain, wherein the component configured to determine the merging metric is configured to: determine an initial metric for detecting the start of a sound event; as well as Based on the merging metric, the spatial audio signal parameter values ​​are merged into a smaller number of spatial audio signal parameter values ​​at frequency within the time-frequency domain, wherein the component configured to merge the spatial audio signal parameter values ​​is configured as follows: When the initial metric indicates the start of the sound event, determine the spatial audio signal parameter value frequency band that best represents the spatial audio signal parameter value frequency band of the time period; For the determined frequency band of the spatial audio signal parameter value, determine whether the energy ratio of the frequency band is greater than the weighted average of the energy ratios of the frequency bands within the time period; as well as When the energy ratio of the determined spatial audio signal parameter value frequency band is greater than the weighted average of the energy ratio of the frequency band of the time period, the spatial audio signal parameter values ​​are merged into a smaller number of spatial audio signal parameter values ​​in the frequency domain of the time frequency domain.

2. The apparatus of claim 1, wherein the component configured to determine the initial metric is configured to: Determine the energy parameters for the at least one audio signal within the said time period; The envelope of the slow audio signal is determined based on the energy parameters and the slow decay time. The fast audio signal envelope is determined based on the energy parameters and fast decay time. The initial metric is determined based on the slow audio signal envelope and the fast audio signal envelope.

3. The apparatus of claim 1, wherein the apparatus further comprises a component configured to: merge the spatial audio signal parameter values ​​into a smaller number of spatial audio signal parameter values ​​in time when the energy ratio of the determined frequency band of the spatial audio signal parameter value is less than the weighted average of the energy ratios of the frequency band of the time period.

4. The apparatus according to any one of claims 1 to 3, wherein the component is further configured to encode the combined spatial audio signal parameter values.

5. The apparatus of claim 4, wherein the component configured to encode the merged spatial audio signal parameter values ​​is configured to quantize the merged spatial audio signal parameter values.

6. The apparatus of claim 4, wherein the component configured to encode the merged spatial audio signal parameter values ​​is configured to: entropy encode the merged spatial audio signal parameter values.

7. A method for spatial audio coding, comprising: For at least one audio signal, spatial audio signal parameter values ​​are obtained, wherein the spatial audio signal parameter values ​​are distributed within the time frequency domain of the time period of the at least one audio signal; Determine a merging metric to control the merging of spatial audio signal parameter values ​​in the time-frequency domain, wherein determining the merging metric includes: determining a start metric for detecting the beginning of a sound event; and Based on the merging metric, within the time-frequency domain, the spatial audio signal parameter values ​​are merged into a smaller number of spatial audio signal parameter values ​​at different frequencies, wherein merging the spatial audio signal parameter values ​​includes: When the initial metric indicates the start of the sound event, determine the spatial audio signal parameter value frequency band that best represents the spatial audio signal parameter value frequency band of the time period; For the determined frequency band of the spatial audio signal parameter value, determine whether the energy ratio of the frequency band is greater than the weighted average of the energy ratios of the frequency bands within the time period; and When the energy ratio of the determined spatial audio signal parameter value frequency band is greater than the weighted average of the energy ratio of the frequency band of the time period, the spatial audio signal parameter values ​​are merged into a smaller number of spatial audio signal parameter values ​​in the frequency domain of the time frequency domain.

8. The method of claim 7, wherein determining the initial metric comprises: Determine the energy parameters for the at least one audio signal within the said time period; The envelope of the slow audio signal is determined based on the energy parameters and the slow decay time. The fast audio signal envelope is determined based on the energy parameters and fast decay time. The initial metric is determined based on the slow audio signal envelope and the fast audio signal envelope.

Images