Determination of the importance of spatial audio parameters and associated coding

Abstract

In particular, a device for spatial audio coding is disclosed, which can receive or determine a sub-band-based spatial audio parameter for providing spatial audio reproduction for one or more audio signals (102), the spatial audio parameter including a coherence value (112) for each of a plurality of sub-bands (202) of a frame. The device further determines an importance metric for the coherence values ​​(401) of the plurality of sub-bands of the frame, and uses the importance metric to determine whether to encode the coherence values ​​of the plurality of sub-bands of the frame (403).

Description

Technical Field

[0001] The present application relates to an apparatus and method for sound field related parameter encoding, but does not exclusively relate to time-frequency domain direction related parameter encoding for audio encoders and decoders. Background Art

[0002] Parametric spatial audio processing is the field of audio signal processing that uses a set of parameters to describe the spatial aspects of sound. For example, in parametric spatial audio capture from a microphone array, estimating a set of parameters from the microphone array signal, such as the direction of the sound in the frequency band, and the ratio of the directional to non-directional parts of the captured sound in the frequency band, is a typical and effective choice. It is well known that these parameters describe the perceived spatial characteristics of the captured sound at the location of the microphone array. These parameters can be used accordingly in the synthesis of spatial sound for binaural headphones, speakers, or other formats such as Ambisonics.

[0003] Therefore, the directionality in a frequency band and the direct-to-total energy ratio are particularly effective parameterizations for spatial audio capture.

[0004] A parameter set including directional parameters in a frequency band and energy ratio parameters in a frequency band (indicating the directionality of the sound) can also be used as spatial metadata for an audio codec (which can also include other parameters such as surround coherence, spread coherence, number of directions, distance, etc.). For example, these parameters can be estimated from an audio signal captured by a microphone array, and, for example, a stereo or mono signal can be generated from the microphone array signal to be transmitted together with the spatial metadata. The stereo signal can, for example, be encoded with an AAC encoder, while the mono signal can be encoded with an EVS encoder. The decoder can decode the audio signal into a PCM signal and process the sound in the frequency band (using the spatial metadata) to obtain a spatial output, for example, a binaural output.

[0005] The aforementioned solution is particularly suitable for encoding captured spatial sound from a microphone array (e.g., in a mobile phone, a VR camera, a stand-alone microphone array). However, it may be desirable for such an encoder to have other input types in addition to the signal captured by the microphone array, such as a speaker signal, an audio object signal, or an Ambisonic signal.

[0006] Analysis of First Order Ambisonics (FOA) input for spatial metadata extraction has been well documented in the scientific literature related to Directional Audio Coding (DirAC) and Harmonic Plane Wave Expansion (Harpex). This is because there are microphone arrays that directly provide FOA signals (more precisely: its variant, B-format signals), and therefore analysis of such input has become a research focus in this field.

[0007] Another input to the encoder is also a multi-channel speaker input, such as a 5.1 or 7.1 channel surround sound input.

[0008] However, for the components of spatial metadata, compression and encoding of coherence parameters is of considerable interest in order to minimize the total number of bits required to represent the spatial audio parameters. Summary of the invention

[0009] According to a first aspect, an apparatus for spatial audio coding is provided, comprising: means for receiving or determining, for one or more audio signals, subband-based spatial audio parameters for providing spatial audio reproduction, wherein the spatial audio parameters include a coherence value for each of a plurality of subbands of a frame; means for determining a significance measure for the coherence values ​​of the plurality of subbands of the frame; and means for using the significance measure to determine whether to encode the coherence values ​​of the plurality of subbands of the frame.

[0010] The apparatus may also include a component for receiving an energy ratio value for each of a plurality of subbands, wherein the component for determining an importance metric for the coherence values ​​of the plurality of subbands of the frame may include: a component for determining a coherent non-directional energy ratio for each of the plurality of subbands, wherein the coherent non-directional energy ratio of the subband is determined as the non-directional energy ratio of the subband multiplied by the coherence value of the subband, wherein the non-directional energy ratio is related to an energy ratio remaining in the subband after the energy ratio value of the subband has been removed; and a component for summing the determined coherent non-directional energy ratios for all the plurality of subbands.

[0011] The component for determining the importance measure for the coherence values ​​of multiple subbands of the frame may also include: a component for multiplying the coherent non-directional energy ratio by the non-directional energy ratio for each subband; a component for summing the products of the coherent non-directional energy ratio and the non-directional energy ratio of each subband for all multiple subbands; and a component for normalizing the sum of the products of the coherent non-directional energy ratio and the non-directional energy ratio of each subband by the sum of the non-directional energy ratio of each subband.

[0012] The importance measure of the coherence values ​​of the plurality of subbands of the frame may be a first importance measure of the coherence values ​​of the plurality of subbands of the frame.

[0013] The importance measure of the coherence values ​​of the plurality of subbands of the frame may be a second importance measure of the coherence values ​​of the plurality of subbands of the frame.

[0014] The means for determining the importance measure for the coherence values ​​of the plurality of subbands of the frame may include means for determining the importance measure for the coherence values ​​of the plurality of subbands of the frame as a maximum measure of the first importance measure and the second importance measure.

[0015] The means for using the importance metric to determine whether to encode the coherence values ​​of the plurality of subbands of the frame may include: means for comparing the importance metric to a threshold; and means for encoding the coherence values ​​of the plurality of subbands of the frame when the importance metric exceeds the threshold.

[0016] The coherence value of each subband of the plurality of subbands may be a surround coherence value, wherein the surround coherence value may be determined based on an inter-channel coherence between the two or more channel audio signals.

[0017] The surround coherence value of each subband in a plurality of subbands may be determined by the apparatus including means for performing the following operations: calculating a covariance matrix associated with two or more channel audio signals; monitoring a channel audio signal having maximum energy determined based on the covariance matrix and a subset of other channel audio signals, wherein the subset is a number determined between 1 and a number less than the total number of channel audio signals having second-maximum energy; and generating surround covariance parameters based on selecting a minimum value among normalized coherences determined between the channel audio signal having maximum energy and each second-maximum energy channel audio signal.

[0018] The energy ratio value of each sub-band may be a direct to total energy ratio value of each sub-band.

[0019] The non-directional energy ratio for a sub-band is determined as 1 minus the direct to total energy ratio for that sub-band.

[0020] According to a second aspect, there is a method for spatial audio coding, comprising: for one or more audio signals, receiving or determining subband-based spatial audio parameters for providing spatial audio reproduction, wherein the spatial audio parameters include coherence values ​​for each of a plurality of subbands of a frame; determining an importance metric for the coherence values ​​of the plurality of subbands of the frame; and using the importance metric to determine whether to encode the coherence values ​​of the plurality of subbands of the frame.

[0021] The method may also include receiving an energy ratio value for each of a plurality of subbands, wherein determining an importance metric for the coherence values ​​of the plurality of subbands of the frame may include: determining a coherent non-directional energy ratio for each of the plurality of subbands, wherein the coherent non-directional energy ratio of the subband is determined as the non-directional energy ratio of the subband multiplied by the coherence value of the subband, wherein the non-directional energy ratio is related to an energy ratio remaining in the subband after the energy ratio value of the subband has been removed; and summing the determined coherent non-directional energy ratios for all plurality of subbands.

[0022] Determining an importance metric for the coherence values ​​of multiple subbands of the frame may also include: for each subband, multiplying the coherent non-directional energy ratio by the non-directional energy ratio; for all multiple subbands, summing the products of the coherent non-directional energy ratio and the non-directional energy ratio of each subband; and normalizing the sum of the products of the coherent non-directional energy ratio and the non-directional energy ratio of each subband by the sum of the non-directional energy ratio of each subband.

[0023] The importance measure of the coherence values ​​of the plurality of subbands of the frame may be a first importance measure of the coherence values ​​of the plurality of subbands of the frame.

[0024] The importance measure of the coherence values ​​of the plurality of subbands of the frame may be a second importance measure of the coherence values ​​of the plurality of subbands of the frame.

[0025] The means for determining the importance metric for the coherence values ​​of the plurality of subbands of the frame may include determining the importance metric for the coherence values ​​of the plurality of subbands of the frame as a maximum metric of the first importance metric and the second importance metric.

[0026] Using the importance metric to determine whether to encode the coherence values ​​of the plurality of subbands of the frame may include: comparing the importance metric to a threshold; and encoding the coherence values ​​of the plurality of subbands of the frame when the importance metric exceeds the threshold.

[0027] The coherence value of each subband of the plurality of subbands may be a surround coherence value, wherein the surround coherence value may be determined based on an inter-channel coherence between the two or more channel audio signals.

[0028] A surround coherence value for each of a plurality of subbands may be determined by an apparatus comprising the following operations: calculating a covariance matrix associated with two or more channel audio signals; monitoring a channel audio signal having maximum energy determined based on the covariance matrix and a subset of other channel audio signals, wherein the subset is a number determined between 1 and a number less than a total number of channel audio signals having second-maximum energy; and generating surround covariance parameters based on selecting a minimum value among normalized coherences determined between the channel audio signal having maximum energy and each second-maximum energy channel audio signal.

[0029] The energy ratio value of each sub-band may be a direct to total energy ratio value of each sub-band.

[0030] The non-directional energy ratio of a sub-band may be determined as 1 minus the direct to total energy ratio for that sub-band.

[0031] According to a third aspect, there is provided an apparatus for spatial audio coding, the apparatus comprising at least one processor and at least one memory comprising computer program code, the at least one memory and the computer code being configured to: for one or more audio signals, receive or determine subband-based spatial audio parameters for providing spatial audio reproduction, wherein the spatial audio parameters comprise a coherence value for each of a plurality of subbands of a frame; determine an importance metric for the coherence values ​​of the plurality of subbands of the frame; and use the importance metric to determine whether to encode the coherence values ​​of the plurality of subbands of the frame.

[0032] The device may further be caused to receive an energy ratio value for each of a plurality of subbands, wherein the device caused to determine an importance metric for the coherence values ​​of a plurality of subbands of the frame may be caused to: determine a coherent non-directional energy ratio for each of a plurality of subbands, wherein the coherent non-directional energy ratio of the subband is determined as the non-directional energy ratio of the subband multiplied by the coherence value of the subband, wherein the non-directional energy ratio is related to the energy ratio remaining in the subband after the energy ratio value of the subband has been removed; and sum the determined coherent non-directional energy ratios for all plurality of subbands.

[0033] The device, which is enabled to determine the importance measure for the coherent values ​​of multiple subbands of the frame, can be further enabled to: for each subband, multiply the coherent non-directional energy ratio by the non-directional energy ratio; for all multiple subbands, sum the products of the coherent non-directional energy ratio and the non-directional energy ratio of each subband; and normalize the sum of the products of the coherent non-directional energy ratio and the non-directional energy ratio of each subband by the sum of the non-directional energy ratio of each subband.

[0034] The importance measure of the coherence values ​​of the plurality of subbands of the frame may be a first importance measure of the coherence values ​​of the plurality of subbands of the frame.

[0035] The importance measure of the coherence values ​​of the plurality of subbands of the frame may be a second importance measure of the coherence values ​​of the plurality of subbands of the frame.

[0036] The apparatus caused to determine the importance measure for the coherence values ​​of the plurality of subbands of the frame may be caused to determine the importance measure for the coherence values ​​of the plurality of subbands of the frame as a maximum measure of the first importance measure and the second importance measure.

[0037] The apparatus caused to use the importance metric to determine whether to encode coherence values ​​of multiple subbands of the frame may be caused to: compare the importance metric to a threshold; and encode coherence values ​​of multiple subbands of the frame when the importance metric exceeds the threshold.

[0038] The coherence value of each subband of the plurality of subbands may be a surround coherence value, wherein the surround coherence value may be determined based on an inter-channel coherence between the two or more channel audio signals.

[0039] The surround coherence value of each subband in the plurality of subbands may be determined by the apparatus being caused to perform the following operations: calculating a covariance matrix associated with two or more channel audio signals; monitoring a channel audio signal having maximum energy determined based on the covariance matrix and a subset of other channel audio signals, wherein the subset is a number determined between 1 and a number less than the total number of channel audio signals having second-maximum energy; and generating surround covariance parameters based on selecting a minimum value among normalized coherences determined between the channel audio signal having maximum energy and each second-maximum energy channel audio signal.

[0040] The energy ratio value of each sub-band may be a direct to total energy ratio value of each sub-band.

[0041] The non-directional energy ratio of a sub-band may be determined as 1 minus the direct to total energy ratio for that sub-band.

[0042] A computer program includes program instructions for causing a computer to execute the method described above.

[0043] A computer program product stored on a medium may cause an apparatus to perform the method described herein.

[0044] An electronic device may include an apparatus as described herein.

[0045] A chipset may include the apparatus as described herein.

[0046] The embodiments of the present application are intended to solve the problems associated with the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] For a better understanding of the present application, reference will now be made by way of example to the accompanying drawings, in which:

[0048] Figure 1 schematically illustrates a system suitable for implementing the apparatus of some embodiments;

[0049] Figure 2 schematically illustrates a metadata encoder according to some embodiments;

[0050] Figure 3 According to some embodiments, Figure 2 A flowchart of the operation of the metadata encoder shown in;

[0051] Figure 4 Schematically illustrating how Figure 2 The coherent encoder shown in ;

[0052] Figure 5 According to some embodiments Figure 4 A flowchart of the operation of the coherence value significance determiner shown in;

[0053] Figure 6 According to some embodiments Figure 4 A flowchart of the operation of the coherence value code determiner shown in;

[0054] Figure 7 An example device suitable for implementing the illustrated means is schematically illustrated. DETAILED DESCRIPTION

[0055] Suitable devices and possible mechanisms for providing metadata parameters derived from effective spatial analysis are described in more detail below. In the following discussion, multi-channel systems will be discussed with respect to multi-channel microphone implementations. However, as described above, the input format can be any suitable input format, such as multi-channel speakers, Ambisonic (FOA / HOA), etc. It should be understood that in some embodiments, the channel position is based on the position of the microphone, or is based on a virtual position or direction. In addition, the output of the example system is a multi-channel speaker arrangement. However, it should be understood that the output can be rendered to the user via means other than speakers. In addition, the multi-channel speaker signal can be summarized as two or more playback audio signals.

[0056] For each considered time-frequency (TF) block or tile, in other words, for a time / frequency subband, the metadata includes at least the direction (elevation, azimuth), the energy ratio of the resulting direction, and the surround coherence independent of that direction. In addition, for each TF block, the extended coherence can be determined and included. All this data is encoded and sent (or stored) by the encoder so that the spatial signal can be reconstructed at the decoder.

[0057] The typical overall operating bit rate of the codec may leave 2 to 10 kbps for transmission / storage of spatial metadata. However, some other implementations may allow up to 20 kbps for transmission / storage of spatial metadata. The encoding of directional parameters and energy ratio components has been examined previously. However, the encoding of coherent data requires further study in order to minimize the overall operating bit rate required to encode spatial metadata.

[0058] The concept discussed below is to encode, for each time-frequency block, coherence parameters as well as direction and energy ratio parameters. In certain scenarios, the encoding of the coherence parameters for a frame may be adjusted based on the subjective importance of the coherence parameters to the listener. For example, for an audio frame, it may be determined that the coherence parameters for each subband may not contribute subjective importance relative to the surrounding subbands. In this case, a savings in operating bit rate may be achieved by not encoding the coherence parameters for that frame.

[0059] Therefore, the present invention is based on the following considerations: for certain cases, an overall saving in the number of bits used to encode the coherence parameters can be achieved by having a metric that can provide an indication of the subjective importance of the coherence value of each subband with respect to the surrounding subbands, and using the result of this metric to determine whether the coherence parameters should be encoded for this frame.

[0060] in this regard, Figure 1 Depicted are example devices and systems for implementing the embodiments shown in the present application. System 100 is shown as having an "analysis" part 121 and a "synthesis part 131. The "analysis" part 121 is the part from receiving a multi-channel loudspeaker signal to encoding of metadata and downmix signals, while the "synthesis" part 131 is the part from decoding of the encoded metadata and downmix signals to rendering of the regenerated signal (e.g., in the form of a multi-channel loudspeaker).

[0061] The input to the system 100 and the "analysis" section 121 is a multi-channel signal 102. In the following example, a microphone channel signal input is described, however, in other embodiments, any suitable input (or synthesized multi-channel) format may be implemented. For example, in some embodiments, the spatial analyzer and spatial analysis may be implemented outside the encoder. For example, in some embodiments, spatial metadata associated with the audio signal may be provided to the encoder as a separate bitstream. In some embodiments, the spatial metadata may be provided as a set of spatial (directional) index values.

[0062] The multi-channel signal is passed to the transmission signal generator 103 and the analysis processor 105 .

[0063] In some embodiments, the transmission signal generator 103 is configured to receive a multi-channel signal, generate a suitable transmission signal including a determined number of channels, and output a transmission signal 104. For example, the transmission signal generator 103 can be configured to generate a 2-audio channel down-mix of the multi-channel signal. The determined number of channels can be any suitable number of channels. In some embodiments, the transmission signal generator is configured to select or combine the input audio signal to the determined number of channels in other ways, such as by beamforming technology, and output it as a transmission signal.

[0064] In some embodiments, the transmission signal generator 103 is optional and the multi-channel signal is passed to the encoder 107 without processing in the same way as the transmission signal in this example.

[0065] In some embodiments, the analysis processor 105 is further configured to receive the multichannel signal and analyze the signal to generate metadata 106 associated with the multichannel signal and therefore associated with the transmission signal 104. The analysis processor 105 can be configured to generate metadata that can include a direction parameter 108 and an energy ratio parameter 110 and a coherence parameter 112 (and in some embodiments, a diffuseness parameter) for each time-frequency analysis interval. In some embodiments, the direction, energy ratio and coherence parameters can be considered spatial audio parameters. In other words, the spatial audio parameters include parameters intended to characterize the sound field created by the multichannel signal (or typically two or more playback audio signals).

[0066] In some embodiments, the parameters generated may differ from band to band. Thus, for example, in band X, all parameters are generated and transmitted, while in band Y, only one of the parameters is generated and transmitted, and furthermore, in band Z, no parameters are generated or transmitted. A practical example of this may be that for some bands, such as the highest band, some parameters are not required for perceptual reasons. The transmission signal 104 and the metadata 106 may be passed to the encoder 107.

[0067] The encoder 107 may include an audio encoder core 109 configured to receive the transport (e.g., downmix) signals 104 and generate a suitable encoding of these audio signals. In some embodiments, the encoder 107 may be a computer (running suitable software stored on a memory and on at least one processor), or alternatively may be a specific device, such as using an FPGA or ASIC. The encoding may be implemented using any suitable scheme. In addition, the encoder 107 may include a metadata encoder / quantizer 111 configured to receive metadata and output information in an encoded or compressed form. In some embodiments, Figure 1 The encoder 107 may further interleave, multiplex into a single data stream or embed metadata into the encoded downmix signal before transmission or storage as indicated by the dashed line in FIG. Multiplexing may be achieved using any suitable scheme.

[0068] On the decoder side, the received or acquired data (stream) may be received by a decoder / demultiplexer 133. The decoder / demultiplexer 133 may demultiplex the encoded stream and pass the audio encoded stream to a transport extractor 135 configured to decode the audio signal to obtain a transport signal. Similarly, the decoder / demultiplexer 133 may include a metadata extractor 137 configured to receive the encoded metadata and generate the metadata. In some embodiments, the decoder / demultiplexer 133 may be a computer (running appropriate software stored on a memory and on at least one processor), or alternatively may be a specific device, for example using an FPGA or ASIC.

[0069] The decoded metadata and transport audio signal may be passed to a synthesis processor 139 .

[0070] The “Synthesis” portion 131 of the system 100 also shows a synthesis processor 139 that is configured to receive the transmission and metadata and recreate synthesized spatial audio in the form of the multi-channel signal 110 in any suitable format based on the transmission signal and the metadata (which may be a multi-channel speaker format, or in some embodiments any suitable output format such as a binaural or surround sound signal, depending on the use case).

[0071] Therefore, to summarize the above, firstly, the system (analysis part) is configured to receive a multi-channel audio signal.

[0072] Furthermore, the system (analysis part) is configured to generate a suitable transmission audio signal (eg by selecting or downmixing some audio signal channels).

[0073] The system is further configured to encode the transmission signal and metadata for storage / transmission.

[0074] Thereafter, the system may store / send the encoded transmission and metadata.

[0075] The system can acquire / receive encoded transmissions and metadata.

[0076] Furthermore, the system is configured to extract the transport and metadata from the encoded transport and metadata parameters, for example, demultiplex and decode the encoded transport and metadata parameters.

[0077] The system (synthesis part) is configured to synthesize an output multi-channel audio signal based on the extracted transmission audio signal and metadata.

[0078] about Figure 2 , describes in more detail an example analysis processor 105 and metadata encoder / quantizer 111 (eg, Figure 1 ).

[0079] In some embodiments, the analysis processor 105 comprises a time-frequency domain transformer 201 .

[0080] In some embodiments, the time-frequency domain converter 201 is configured to receive the multi-channel signal 102 and apply a suitable time-frequency domain transform such as a short-time Fourier transform (STFT) to convert the input time domain signal into a suitable time-frequency signal. These time-frequency signals can be passed to the spatial analyzer 203.

[0081] Thus, for example, in the time-frequency domain representation, the time-frequency signal 202 may be represented as:

[0082] s i (b,n)

[0083] Where b is the frequency bin index, n is the time-frequency block (frame) index, and i is the channel index. In another expression, n can be considered as a time index with a sampling rate lower than the sampling rate of the original time domain signal. These frequency points can be grouped into multiple subbands, which group one or more points into subbands with frequency band index k=0,...,K-1. Each subband k has the lowest point b k,low and the highest point b k,high , and this subband contains k,low to b k,high The width of the subband can be approximated by any suitable distribution, for example, the equivalent rectangular bandwidth (ERB) scale or the Bark scale.

[0084] In some embodiments, the analysis processor 105 comprises a spatial analyzer 203. The spatial analyzer 203 may be configured to receive the time-frequency signals 202 and, based on these signals, estimate the directional parameters 108. The directional parameters may be determined based on any audio-based determination of "direction".

[0085] For example, in some embodiments, spatial analyzer 203 is configured to estimate direction with two or more signal inputs. This represents the simplest configuration for estimating "direction", more complex processing may be performed with even more signals.

[0086] Therefore, the spatial analyzer 203 may be configured to provide, for each frequency band and time-frequency block within a frame of the audio signal, at least one azimuth and elevation angle, denoted as azimuth and elevation, respectively. and the elevation angle θ(k,n). The direction parameter 108 may also be passed to the direction encoder 205.

[0087] The spatial analyzer 203 may also be configured to determine an energy ratio parameter 110. The energy ratio may be considered as a determination of the energy of an audio signal that may be considered to arrive from a direction. The direct to total energy ratio r(k,n) may be estimated, for example, using a stability measure of the directivity estimate, or using any correlation measure, or any other suitable method for obtaining a ratio parameter. Each direct to total energy ratio corresponds to a specific spatial direction and describes how much energy there is from that specific spatial direction compared to the total energy. This value may also be expressed separately for each time-frequency block. The spatial direction parameters and the direct to total energy ratio describe how much of the total energy of each time-frequency block comes from a specific direction. In general, the spatial direction parameters may also be considered as the direction of arrival (DOA).

[0088] In an embodiment, the direct to total energy ratio parameter may be estimated based on a normalized cross-correlation parameter cor′(k,n) between microphone pairs at frequency band k, the value of which is between −1 and 1. By dividing the normalized cross-correlation parameter by the diffuse field normalized cross-correlation parameter cor′ D (k,n) and the energy ratio parameter r(k,n) can be directly determined as:

[0089]

[0090] This direct to total energy ratio is further explained in PCT Publication WO 2017 / 005978, which is incorporated herein by reference in its entirety.

[0091] The energy ratio may be passed to an energy ratio encoder 207 .

[0092] Furthermore, the spatial analyzer 203 may be configured to determine a plurality of coherence parameters 112 , which may include surround coherence (γ(k,n)) and extended coherence (ζ(k,n)), both of which are analyzed in the time-frequency domain.

[0093] Each of the aforementioned coherence problem parameters is discussed next. All processing is performed in the time-frequency domain, so for the sake of brevity, the time-frequency indices k and n are removed where necessary.

[0094] Let us first consider the case where two spaced loudspeakers (eg, front left and front right) are used instead of a single loudspeaker to reproduce the sound coherently. A coherence analyzer can be configured to detect that this approach has been applied to surround mixing.

[0095] Therefore, in some embodiments, the spatial analyzer 203 may be configured to calculate the covariance matrix C for a given analysis interval including one or more time indexes n and frequency points b. The size of the matrix is ​​N L x N L , whose elements are denoted by c ij , where N L is the number of speaker channels, i and j are the speaker channel indices.

[0096] Next, the spatial analyzer 203 may be configured to determine the speaker channel i closest to the estimated direction (in this example, the azimuth angle θ) c :

[0097] i c =arg(min(|θ-α i |))

[0098] Among them, α i is the angle of speaker i.

[0099] Furthermore, in such an embodiment, the spatial analyzer 203 is configured to determine the c The left side l and right side i r The closest speaker.

[0100] The normalized coherence between loudspeakers i and j is expressed as:

[0101]

[0102] Using this equation, spatial analyzer 203 may be configured to calculate i l with i r The normalized coherence c′ between lr . In other words, calculate:

[0103]

[0104] Furthermore, the spatial analyzer 203 may be configured to use the diagonal elements of the covariance matrix to determine the energy of the loudspeaker channel i:

[0105] E i =c ii

[0106] And determine the speaker i l and i r With speaker i l 、i r and i c The energy ratio between them is:

[0107]

[0108] In turn, the spatial analyzer 203 may use these determined variables to generate a "stereoness" parameter:

[0109] μ=c′ lr ξ lr / lrc

[0110] This "stereoscopic" parameter has a value between 0 and 1. A value of 1 means that speaker i l and i r , and this sound dominates the energy of this sector. The reason for this may be, for example, that a speaker mixer uses amplitude panning techniques to create an "airy" perception of the sound. A value of 0 means that no such technique has been applied, and, for example, the sound may simply be localized to the closest speaker.

[0111] In addition, the spatial analyzer 203 may be configured to detect or at least identify situations where three (or more) speakers are used to coherently reproduce sound to create a "close" perception (e.g., using left front, right front, and center instead of just center). This may be because the mixing engineer created this situation when performing a surround mix on a multi-channel speaker mix.

[0112] In this embodiment, the coherence analyzer uses the same speaker i previously identified. l 、i r and i c The normalized coherence value c′ can be determined using the normalized coherence determination discussed previously. cl and c′ cr . In other words, the following values ​​are calculated:

[0113]

[0114] Furthermore, the spatial analyzer 203 can determine the normalized coherence value c′ describing the coherence between the loudspeakers using the following equation: clr :

[0115] c′ clr =min(c′ cl ,c′ cr )

[0116] In addition, the spatial analyzer 203 may be configured to determine a spatial distribution that describes how (ie, how uniform) the energy is in channel i. l 、i r and i c Parameters evenly distributed between:

[0117]

[0118] Using these variables, spatial analyzer 203 can determine a new coherence shift parameter κ as:

[0119] κ=c′ clr ξ clr

[0120] The coherence pan parameter κ has a value between 0 and 1. A value of 1 means that l 、i r and i c , and the energy of that sound is evenly distributed between those speakers. The reason for this could be, for example, because the speaker mix was generated using recording mixing techniques for creating the perception of closer sound sources. A value of 0 means that no such techniques have been applied, for example, the sound may simply be localized to the closest speaker.

[0121] Determine the metric i l and i r in (but not in i c The stereo parameter μ of the coherent sound volume and the measure of all i l 、i r and i c The spatial analyser 203 which determines the coherence panning parameters κ of the coherent sound volume in is configured to use these parameters to determine a coherence parameter to be output as metadata.

[0122] Therefore, the spatial analyzer 203 is configured to combine the stereo parameter μ and the coherence translation parameter κ to form an extended coherence ζ parameter with values ​​ranging from 0 to 1. An extended coherence ζ value of 0 indicates a point source, in other words, the source should be detected with as few loudspeakers as possible (e.g., only loudspeaker i c ) to reproduce the sound. As the value of the extended coherence ζ increases, more energy is extended to the speaker i csurrounding speakers; until the value is 0.5, the energy is in speaker i l 、i r and i c When the value of the extended coherence ζ exceeds 0.5, the loudspeaker i c The energy in decreases until the value is 1, at speaker i c There is no energy in the speaker, and all the energy is in the speaker l and i r Place.

[0123] In some embodiments, using the above parameters μ and κ, spatial analyzer 203 is configured to determine the extended coherence parameter ζ using the following expression:

[0124]

[0125] The above expressions are merely examples, and it should be noted that the spatial analyzer 203 may adopt any other manner to estimate the extended coherence parameter ζ as long as it complies with the above parameter definition.

[0126] In addition to being configured to detect the previous situations, the spatial analyzer 203 may also be configured to detect or at least identify situations in which sound is reproduced coherently from all (or nearly all) speakers to create a perception of "inside-the-head" or "above".

[0127] In some embodiments, the spatial analyzer 203 may be configured to determine the energy E having the maximum value. i and speaker channel i e to classify.

[0128] Furthermore, the spatial analyzer 203 can be configured to determine the channel and M L Normalized coherence c′ between the other loudest channels ij . Then, we can monitor the channel and M L These normalized coherences c′ between the other loudest channels ij In some embodiments, M L Can be N L -1, which would mean monitoring the coherence between the loudest channel and all other speaker channels. However, in some embodiments, M L It can be a smaller number, for example, N L -2. Using these normalized coherence values, the coherence analyzer can be configured to determine the surround coherence parameter γ using the following expression:

[0129]

[0130] in, is the loudest channel and M L Normalized coherence between the next loudest channels.

[0131] The surround coherence parameter γ has a value from 0 to 1. A value of 1 means that there is coherence between all (or almost all) of the loudspeaker channels. A value of 0 means that there is no coherence between all (or even almost all) of the loudspeaker channels.

[0132] The above expression is merely an example of estimating the surround coherence parameter γ, and any other manner may be used as long as it conforms to the above parameter definition.

[0133] Thereafter, the analysis processor 203 may apply directional analysis to determine direction and energy ratio parameters.

[0134] Furthermore, the analysis processor 203 may be configured to output the determined parameters.

[0135] Although the direction, energy ratio and coherence parameters are expressed for each time index n in this article, in some embodiments, these parameters can be combined over several time indexes. As described, the same applies to the frequency axis, and the direction of several frequency points b can be represented by a direction parameter in the frequency band k including the several frequency points b. The same applies to the spatial parameters discussed in this article.

[0136] In some embodiments, the direction data can be represented using 16 bits. In such an embodiment, the energy ratio parameter can be represented on 8 bits. For each frame, there can be 5 subbands and 4 time-frequency (TF) blocks. Therefore, in this example, (16+8)x4x5 bits are required to store the uncompressed direction and energy ratio metadata for each frame. The coherent data for each TF block can be a representation between 0 and 1 and can initially be represented on 8 bits.

[0137] Likewise Figure 2 As shown in , an example metadata encoder / quantizer 111 is shown according to some embodiments.

[0138] The metadata encoder / quantizer 111 may include a direction encoder 205. The direction encoder 205 is configured to receive a direction parameter (such as an azimuth angle). and elevation angles θ(k,n) 108) (and in some embodiments, the expected bit allocation), and thereby generate a suitable encoded output. In some embodiments, the encoding is based on an arrangement of spheres (forming a spherical grid arranged in a ring on a "surface" sphere), which are defined by a lookup table defined by a determined quantization resolution. In other words, the spherical grid uses the concept of covering a sphere with smaller spheres, and considering the centers of the smaller spheres as points defining a grid of almost equidistant directions. The smaller spheres thus define cones or solid angles about the center point, which can be indexed according to any suitable indexing algorithm. Although spherical quantization is described herein, any suitable quantization (linear or non-linear) may also be used.

[0139] Furthermore, in some embodiments, the direction encoder 205 is configured to determine the variance of the azimuth parameter values ​​and pass it to the coherence encoder 209 .

[0140] The encoded direction parameter may then be passed to the combiner 211 .

[0141] The metadata encoder / quantizer 111 may include an energy ratio encoder 207. The energy ratio encoder 207 is configured to receive the energy ratio and determine the appropriate encoding for compressing the energy ratio of the sub-band and the time-frequency block. For example, in some embodiments, the energy ratio encoder 207 is configured to use 3 bits to encode each energy ratio parameter value.

[0142] Furthermore, in some embodiments, instead of sending or storing all energy ratio values ​​for all TF blocks, only a weighted average per subband is sent or stored. The average can be determined by considering the total energy of each time block, thereby favoring the values ​​of subbands with more energy.

[0143] In such an embodiment, the quantized energy ratio value is the same for all TF blocks of a given sub-band.

[0144] In some embodiments, the energy ratio encoder 207 is further configured to pass the quantized (encoded) energy ratio values ​​to the combiner 211 and the coherent encoder 209 .

[0145] The metadata encoder / quantizer 111 may include a coherence encoder 209. The coherence encoder 209 is configured to receive the coherence values ​​and determine appropriate encoding of the coherence values ​​for compressed subbands and time-frequency blocks. A 3-bit precision value for the coherence parameter value has been shown to produce acceptable audio synthesis results, but even so, a total of 3x20 bits are required for the coherence data for all TF blocks (in this example, 8 subbands and 5 TF blocks per frame).

[0146] It is therefore an object of an embodiment to reduce the number of bits used to represent coherent data over the course of multiple TF blocks.

[0147] The encoded coherence parameter value may then be passed to the combiner 211 .

[0148] The metadata encoder / quantizer 111 may include a combiner 211. The combiner is configured to receive the encoded (or quantized / compressed) directionality parameters, energy ratio parameters, and coherence parameters, and combine these parameters to generate a suitable output (e.g., a metadata bitstream, which may be combined with a transmission signal or sent or stored separately from the transmission signal).

[0149] about Figure 3 , showing how according to some embodiments Figure 2 Example operation of the metadata encoder / quantizer shown in .

[0150] like Figure 3 As shown in step 301, the initial operation is to obtain metadata (such as azimuth value, elevation value, energy ratio, coherence, etc.).

[0151] like Figure 3 As shown in step 303, the direction values ​​(elevation angle, azimuth angle) may then be compressed or encoded (eg, by applying spherical quantization or any suitable compression).

[0152] like Figure 3 As shown in step 305, the energy ratio values ​​are compressed or encoded (eg, by generating weighted averages for each subband and then quantizing them into 3-bit values).

[0153] like Figure 3 As shown in step 307, the coherence value is also compressed or encoded.

[0154] like Figure 3 As shown in step 309, the encoded direction value, energy ratio, and coherence value are then combined to generate encoded metadata.

[0155] about Figure 4 , showing that Figure 2 An example coherent encoder 209 is shown in FIG.

[0156] As previously described, a technical benefit may be achieved by adapting the encoding of the coherence parameters, wherein for some audio frames the coherence parameters are encoded and thus form part of the encoded bitstream, whereas for other audio frames the coherence parameters are not encoded. The technical benefit is an overall reduction in the bitrate of the encoded bitstream.

[0157] The decision whether to encode the coherence parameters may be made based on the subjective importance of the coherence value of each subband with respect to the surrounding subbands. In other words, the decision to encode the coherence parameters may be made based on whether the coherence parameters contribute to the overall perceived listening experience of the end user.

[0158] Listening experiments have demonstrated that the perception of coherence of spatial audio signals can be at least partially attributed to the interactions between signals present in various frequency bands. This may seem contrary to the premise that for the perception of sound, human frequency resolution can be siloed / compartmentalized according to nonlinear frequency scales such as equivalent rectangular bandwidth (ERB).

[0159] For example, it has been noted through experiments (including perceptual listening tests) that if the coherence is low for most of the bands, then higher coherence in a single band may contribute negligibly to the perception of spatial audio. Therefore, in this case, not encoding the coherence parameter may be beneficial to the overall bitrate of the encoded bitstream.

[0160] Alternatively, it has also been noted through perceptual listening tests that if the energy ratio of a single frequency band is found to be significantly lower than other surrounding frequency bands, then coherence may be perceptually important at that frequency band. In this case, it has been found that the human auditory system may be particularly sensitive. Therefore, it would be prudent to send coherence parameters for the frame in order to enhance the overall listening experience of spatial audio.

[0161] Therefore, the results of the experimental observations indicate that it is possible to devise a metric (or measure) that represents the importance of coherence of a spatial audio signal.

[0162] These effects may be particularly prevalent for the perception of surround coherence of spatial audio signals. However, measuring the importance of other types of coherence parameters may also be helpful when deciding whether to encode coherence parameters. For example, techniques such as these may also be applied to extend coherence parameter values.

[0163] As mentioned above, these effects can be attributed to interactions between frequency bands when it comes to the perception of coherence.

[0164] In this regard, first refer to Figure 4 401 in which the coherence value importance determiner is described as receiving a coherence value for each subband k of a TF block n. In addition, 401 can also be configured to receive a direct-to-total energy ratio r(k,n) of TF block n (or subframe n), i.e., an energy ratio associated with the K subbands of the TF block with time index n. Similarly, the coherence values ​​are also transmitted to the coherence value importance determiner 401 as K coherence values ​​for the K subbands of the TF block with time index n.

[0165] In the first embodiment, the coherence value may include surrounding coherence (r(k,n)). Note that k is used to indicate a subband index, and n is an index of a TF block.

[0166] Also refer to Figure 5 , which depicts example processing steps for determining the significance of surround coherence values ​​for a TF block comprising K subbands.

[0167] Initially, processing block 401 may determine an estimate of the non-directional energy ratio for each subband k by using the direct total energy ratio r(k,n). In an embodiment, the non-directional (surround) energy ratio a(k,n) may be given by:

[0168] a(k,n)=1-r(k,n) for k=0 to K-1

[0169] Note that r(k,n) is the energy in a particular direction out of the total energy, with the direction for each frequency being given by the azimuth and elevation angles. As previously described, the direction is represented by the direction index from the direction encoder 205. The energy ratio r(k,n) can have a value between 0 and 1, where 1 means that the total energy is concentrated along one direction index for subband k. Thus, a(k,n) represents the ambient (or non-directional) energy ratio for a particular subband k, in other words, the energy is not concentrated along a particular direction index.

[0170] like Figure 5 501 in the figure shows a processing step: determining the non-directional (surrounding) energy ratio.

[0171] Therefore, the proportion of coherent non-directional (surrounding) energy for each subband k can be given by:

[0172] ρ(k,n)=a(k,n)γ(k,n) for k=0 to K-1

[0173] The processing steps for determining the proportion of non-directional (surround) energy for each subband are described as follows Figure 5 Processing step 503 in .

[0174] A first metric (or measure) of the importance of the ambient coherence value of TF block n (or subframe n) can be estimated by summing the proportion of non-directional energy over all subbands of subframe n:

[0175]

[0176] The first metric ξ1 may be viewed as giving an indication of whether non-directional (surround) energy is prominent in the multiple frequency sub-bands of sub-frame n.

[0177] The processing step of determining a first measure of the importance of the surrounding coherence value of subframe n is described as follows Figure 5 Processing step 505 in .

[0178] A second metric (or measure) ξ2 of the importance of the surround coherence value of subframe n can be estimated by taking the non-directional energy ratio ρ(k,n) and weighting it with the surround (or non-directional) energy ratio a(k,n) for a particular subband k. This can be summed across all subbands and normalized with the "total" ratio of the surround (or non-directional) energy. The second metric ξ2 can be expressed as:

[0179]

[0180] Here, μ is a tuning factor. Experimental observations have shown that a value of μ = 0.4 can produce favorable results.

[0181] The second metric (or measurement) ξ2 can be viewed as providing a measure of whether the ambient coherence energy is prominent within the predominantly non-directional sub-band, and therefore providing an indication of whether the ambient coherence value of the provided sub-frame may be perceptually significant and therefore whether the coherence value should be encoded and sent.

[0182] The processing step of determining a second measure of the importance of the surrounding coherence parameter of TF(subframe)n is described as follows Figure 5 Processing step 507 in .

[0183] In an embodiment, the first metric and the second metric may be combined to provide a single metric of the importance of the surrounding coherence values ​​of TF block (subframe) n.

[0184] This merging may take the form of taking the maximum of the first metric or the second metric to give a single metric ξ of the importance of the surrounding coherence values ​​of TF block (subframe) n:

[0185] ξ(n)=max(ξ1(n),ξ2(n))

[0186] The output of the coherence value significance determiner 401 may be a single measure ξ of the significance of the surrounding coherence values. This parameter may be passed to the coherence value encoding determiner 403. The processing block 403 may be arranged to determine whether the coherence value of the subframe should be encoded.

[0187] exist Figure 5 The processing step is shown by step 509 in : combining the first metric and the second metric to give a single metric indicating the importance of the surrounding coherence value of TF block n.

[0188] In some embodiments, the single measure ξ of the importance of the surrounding coherence values ​​may include only the first measure ξ1 of the importance of the surrounding values. Alternatively, in other embodiments, the single measure ξ of the importance of the surrounding coherence values ​​may include only the second measure ξ2 of the importance of the surrounding values. In other words, in both embodiments, there is no merging step 509. Therefore, for the operation of the coherence value importance determiner 401, the measure ξ of the importance of the surrounding coherence values ​​may be only the first measure ξ1, or only the second measure ξ2.

[0189] Figure 6 An example set of processing steps that may be performed by the coherence value coding determiner 403 is shown.

[0190] In an embodiment, determining whether the surrounding coherence value of a TF block should be encoded may take the form of comparing a single measure of the importance of the surrounding coherence value ξ with a threshold τ(n) for subframe n. This comparison may be expressed as:

[0191] e(n)=1, if ξ(n)>τ(n)

[0192] e(n)=0, if ξ(n)≤τ(n)

[0193] Among them, e(n)=1 means that the coherent value of TF block (or subframe) n is encoded (and sent), and e(n)=0 means that the coherent value of subframe n is not encoded.

[0194] The threshold may be a constant. For example, a value such as τ(n)=τ=0.15 may be taken. Alternatively, in some embodiments, the threshold may change over time. For example, the threshold may be selected based on a factor based on the encoding bit rate.

[0195] The step of comparing the single measure ξ of the importance of the surrounding coherence values ​​to a threshold is shown as Figure 6 Processing step 601 in .

[0196] Furthermore, the function block 403 may be arranged to output the coherence value of TF block (subframe) n to the coherence parameter encoder 405, provided that the result of the above comparison indicates that the coherence value is significant enough to warrant encoding. Figure 4 A connection path is depicted from the coherence parameter encoding determiner 403 to the coherence parameter encoder 405. However, if the single measure ξ of the importance surrounding the coherence value does not exceed a threshold, the coherence value is not encoded.

[0197] Figure 6 Processing step 603 in 400 depicts the case where a single measure ξ of the importance of surrounding coherence values ​​exceeds a threshold, resulting in the coherence value of the block being sent to the coherence value encoder 405 for encoding. Figure 6 Processing step 605 in illustrates the situation where the single measure ξ of the importance of the surrounding coherence values ​​does not exceed a threshold, resulting in the coherence values ​​of the TF block not being encoded.

[0198] The coherence parameter encoder 405 is configured to receive coherence values ​​that have been determined to be perceptually significant from the functional block 403. These coherence values ​​may be an 8-bit representation between 0 and 1.

[0199] The coherent value encoder 405 can be configured to encode the coherent values ​​associated with the subbands of the TF blocks (or subframes) on a per-TF block basis. The coherent value encoder 405 can adopt a method in which a specific quantization table from a plurality of quantization tables is first used to scalar quantize the coherent values ​​of each subband. The selection of a specific quantization table for quantizing the coherent values ​​can depend on the energy ratio value of each subband, and the number of entries in each quantization table can be different from each other. In essence, each quantization table in a plurality of quantization tables can include a series of pre-calculated quantized coherent values, which can be customized to specific energy ratio values.

[0200] Furthermore, the quantization table index associated with each coherence value may be encoded using a product type code (by which all quantization indices for the subbands of a TF block are concatenated together into a combined index) or a run length code such as a Golomb Rice code.

[0201] In case Golomb Rice codes are used to encode the quantization indices of the coherence values, their mean values ​​can be removed for the TF blocks.

[0202] A selection can be made between a joint combined index and a Golomb Rice code for encoding the subband coherent value quantization index for the TF block based on an estimate of the number of bits that the Golomb Rice encoder will use. If this is less than the bits of the joint combined index, the Golomb Rice encoder is used to encode the subband coherent value quantization index for the TF block. The selection between the joint combined index method and the Golomb Rice method can be signaled as a single bit within the bitstream.

[0203] In turn, the index values ​​of the codebook may form encoded coherence values, which in turn are combined with the encoded direction values ​​and the encoded energy ratios in the combiner 211 to form spatial audio metadata.

[0204] about Figure 7, shows an example electronic device that can be used as an analysis or synthesis device. The device can be any suitable electronic device or apparatus. For example, in some embodiments, the device 1400 is a mobile device, a user device, a tablet computer, a computer, an audio playback device, etc.

[0205] In some embodiments, device 1400 includes at least one processor or central processing unit 1407. Processor 1407 may be configured to execute various program codes, such as the methods described herein.

[0206] In some embodiments, the device 1400 includes a memory 1411. In some embodiments, at least one processor 1407 is coupled to the memory 1411. The memory 1411 can be any suitable storage component. In some embodiments, the memory 1411 includes a program code portion for storing program codes that can be implemented on the processor 1407. In addition, in some embodiments, the memory 1411 may also include a storage data portion for storing data (e.g., data that has been processed or will be processed according to the embodiments described herein). Whenever necessary, the processor 1407 can obtain the implementation program code stored in the program code portion and the data stored in the storage data portion via the memory-processor coupling.

[0207] In some embodiments, device 1400 includes a user interface 1405. In some embodiments, user interface 1405 may be coupled to processor 1407. In some embodiments, processor 1407 may control the operation of user interface 1405 and receive input from user interface 1405. In some embodiments, user interface 1405 may enable a user to input commands to device 1400, for example, via a keyboard. In some embodiments, user interface 1405 may enable a user to obtain information from device 1400. For example, user interface 1405 may include a display configured to display information from device 1400 to a user. In some embodiments, user interface 1405 may include a touch screen or touch interface that enables information to be input to device 1400 and also displays information to a user of device 1400. In some embodiments, user interface 1405 may be a user interface for communicating with a position determiner as described herein.

[0208] In some embodiments, the device 1400 includes an input / output port 1409. In some embodiments, the input / output port 1409 includes a transceiver. In such embodiments, the transceiver may be coupled to the processor 1407 and configured to enable communication with other devices or electronic devices, for example, via a wireless communication network. In some embodiments, the transceiver or any suitable transceiver or transmitter and / or receiver device may be configured to communicate with other electronic devices or devices via a wired or wired coupling.

[0209] The transceiver can communicate with other devices via any suitable known communication protocol. For example, in some embodiments, the transceiver can use a suitable Universal Mobile Telecommunications System (UMTS) protocol, a wireless local area network (WLAN) protocol such as, for example, IEEE 802.X, a suitable short-range radio frequency communication protocol such as Bluetooth, or an infrared data communication path (IRDA).

[0210] The transceiver input / output port 1409 may be configured to receive signals and, in some embodiments, determine parameters as described herein using the processor 1407 executing appropriate code. In addition, the device may generate appropriate downmix signals and parameter outputs to send to a synthesis device.

[0211] In some embodiments, the device 1400 may be used as at least part of a synthesis device. Thus, the input / output port 1409 may be configured to receive a downmix signal and, in some embodiments, receive parameters determined at a capture device or processing device as described herein, and generate a suitable audio signal format output using a processor 1407 executing suitable code. The input / output port 1409 may be coupled to any suitable audio output, such as to a multi-channel speaker system and / or headphones or the like.

[0212] Generally, various embodiments of the present invention may be implemented in hardware or dedicated circuits, software, logic, or any combination thereof. For example, some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device, but the present invention is not limited thereto. Although various aspects of the present invention may be shown and described as block diagrams, flow charts, or illustrated or described using some other graphical representation, it should be fully understood that the boxes, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general hardware or controllers or other computing devices, or some combination thereof as non-limiting examples.

[0213] Embodiments of the present invention may be implemented by computer software, or by hardware, or by a combination of software and hardware, the computer software being executable by a data processor of a mobile device, such as in a processor entity. In addition, in this regard, it should be noted that any box of the logic flow in the accompanying drawings may represent program steps, or interconnected logic circuits, blocks and functions, or a combination of program steps and logic circuits, blocks and functions. The software may be stored on a physical medium, such as a memory chip, or a memory block implemented within a processor, a magnetic medium such as a hard disk or a floppy disk, and an optical medium such as a DVD and its data variants, a CD.

[0214] The memory may be of any type suitable for the local technical environment and may be implemented using any suitable data storage technology, such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory, and removable memory. The data processor may be of any type suitable for the local technical environment and may include, as non-limiting examples, one or more of a general-purpose computer, a special-purpose computer, a microprocessor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a gate-level circuit, and a processor based on a multi-core processor architecture.

[0215] Embodiments of the present invention may be practiced in various components such as integrated circuit modules. The design of integrated circuits is essentially a highly automated process. Complex and powerful software tools are available to convert a logic level design into a semiconductor circuit design ready to be etched and formed on a semiconductor substrate.

[0216] The program can route conductors and position components on a semiconductor chip using well-established design rules and a library of pre-stored design modules. Once the design of a semiconductor circuit is complete, the design results in a standardized electronic format can be transmitted to a semiconductor manufacturing facility or "fab" for fabrication.

[0217] The foregoing description has provided a complete and informative description of example embodiments of the present invention by way of exemplary and non-limiting examples. However, various modifications and adaptations will become apparent to those skilled in the relevant art in view of the foregoing description when read in conjunction with the accompanying drawings and the appended claims. However, all such and similar modifications to the teachings of the present invention will still fall within the scope of the present invention as defined in the appended claims.

Claims

1. An apparatus for spatial audio coding, comprising: means for receiving or determining, for one or more audio signals, spatial audio parameters on a subband basis for providing spatial audio reproduction, wherein the spatial audio parameters comprise coherence values ​​for each of a plurality of subbands of a frame; means for receiving an energy ratio value for each subband of the plurality of subbands; means for determining a first importance metric for the coherence values ​​of the plurality of subbands of the frame, by including means for determining a coherent non-directional energy ratio for each of the plurality of subbands, wherein the coherent non-directional energy ratio for a subband is determined as a non-directional energy ratio for the subband multiplied by a coherence value for the subband, wherein the non-directional energy ratio is related to an energy ratio remaining in the subband after the energy ratio value for the subband has been removed, and means for summing the determined coherent non-directional energy ratios for all of the plurality of subbands; and Means for using the first importance metric to determine whether to encode coherent values ​​for the plurality of subbands of the frame.

2. The device according to claim 1, further comprising: The means for determining a second importance metric for the coherence values ​​of the plurality of subbands of the frame, further comprising: means for multiplying the coherent non-directional energy proportion by the non-directional energy ratio for each sub-band; means for summing the product of the coherent non-directional energy proportion and the non-directional energy ratio for each sub-band for all of the plurality of sub-bands; and Means for normalizing the sum of products of the coherent non-directional energy proportion and the non-directional energy ratio for each sub-band by the sum of the non-directional energy ratio for each sub-band.

3. The device according to claim 2, wherein: The apparatus further comprises means for determining an importance metric for the coherence values ​​of the plurality of subbands of the frame, by having: Means for determining the importance metric of the coherence values ​​of the plurality of subbands of the frame as a maximum metric of the first importance metric and the second importance metric.

4. The device according to claim 3, wherein: Means for using the importance metric to determine whether to encode coherent values ​​of the plurality of subbands of the frame include: means for comparing the importance metric to a threshold; and Means for encoding coherence values ​​for the plurality of subbands of the frame when the significance metric exceeds the threshold.

5. The device according to any one of claims 1 to 4, wherein: The coherence value of each subband of the plurality of subbands is a surround coherence value, wherein the surround coherence value is determined based on an inter-channel coherence between two or more channel audio signals.

6. The device according to claim 5, wherein: The ambient coherence value for each of the plurality of subbands is determined by the apparatus comprising means for: calculating a covariance matrix associated with the two or more channel audio signals; monitoring a channel audio signal having a maximum energy determined based on the covariance matrix and a subset of other channel audio signals, wherein the subset is a determined number between 1 and a number less than a total number of channel audio signals having a second maximum energy; as well as Surround covariance parameters are generated based on selecting a minimum value among normalized coherences determined between the channel audio signal having the maximum energy and each of the second-maximum-energy channel audio signals.

7. The device according to any one of claims 1 to 6, wherein: The energy ratio value of each of the sub-bands is a direct to total energy ratio of each of the sub-bands.

8. The device according to claim 7, wherein: The non-directional energy ratio for a sub-band is determined as 1 minus the direct to total energy ratio for the sub-band.

9. A method for spatial audio coding, comprising: For one or more audio signals, receiving or determining subband-based spatial audio parameters for providing spatial audio reproduction, wherein the spatial audio parameters include coherence values ​​for each of a plurality of subbands of a frame; receiving an energy ratio value for each subband of the plurality of subbands; determining a first importance metric for the coherence values ​​of the plurality of subbands of the frame by: determining, for each of the plurality of subbands, a coherent non-directional energy proportion, wherein the coherent non-directional energy proportion for a subband is determined as a non-directional energy ratio of the subband multiplied by a coherence value for the subband, wherein the non-directional energy ratio is related to an energy ratio remaining in the subband after the energy ratio value for the subband has been removed, and summing the determined coherent non-directional energy proportions for all of the plurality of subbands; and The first importance metric is used to determine whether to encode coherent values ​​for the plurality of subbands of the frame.

10. The method according to claim 9, further comprising: Determining the first importance metric further comprises determining a second importance metric for the coherence values ​​of the plurality of subbands of the frame by: For each subband, multiplying the coherent non-directional energy proportion by the non-directional energy ratio; For all of the plurality of sub-bands, summing the product of the coherent non-directional energy proportion and the non-directional energy ratio of each sub-band; as well as The sum of products of the coherent non-directional energy proportion and the non-directional energy ratio for each sub-band is normalized by the sum of the non-directional energy ratio for each sub-band.

11. The method according to claim 10, further comprising: Determining an importance metric for the coherence values ​​of the plurality of subbands of the frame by: The importance metric of the coherence values ​​of the plurality of subbands of the frame is determined as a maximum metric of the first importance metric and the second importance metric.

12. The method according to claim 11, wherein: Using the importance metric to determine whether to encode coherent values ​​of the plurality of subbands of the frame comprises: comparing the importance metric to a threshold; and When the importance metric exceeds the threshold, coherent values ​​of the plurality of subbands of the frame are encoded.

13. The method according to any one of claims 9 to 12, wherein: The coherence value of each subband of the plurality of subbands is a surround coherence value, wherein the surround coherence value is determined based on an inter-channel coherence between two or more channel audio signals.

14. The method according to claim 13, wherein: The ambient coherence value for each of the plurality of subbands is determined by an apparatus comprising the following operations: calculating a covariance matrix associated with the two or more channel audio signals; monitoring a channel audio signal having a maximum energy determined based on the covariance matrix and a subset of other channel audio signals, wherein the subset is a determined number between 1 and a number less than a total number of channel audio signals having a second maximum energy; as well as Surround covariance parameters are generated based on selecting a minimum value among normalized coherences determined between the channel audio signal having the maximum energy and each of the second-maximum-energy channel audio signals.

15. The method according to any one of claims 9 to 14, wherein: The energy ratio value of each of the sub-bands is a direct to total energy ratio of each of the sub-bands.

16. The method according to claim 15, wherein: The non-directional energy ratio for a sub-band is determined as 1 minus the direct to total energy ratio for the sub-band.

17. A computer program product embodied on a non-transitory computer readable medium, comprising computer program code configured to, when executed on at least one processor, cause an apparatus to: For one or more audio signals, spatial audio parameters for providing spatial audio reproduction on a sub-band basis are received or determined, wherein: The spatial audio parameters include a coherence value for each subband of a plurality of subbands of a frame; receiving an energy ratio value for each subband of the plurality of subbands; determining a significance metric for the coherence values ​​of the plurality of subbands of the frame by causing the apparatus to: determine, for each of the plurality of subbands, a coherent non-directional energy proportion, wherein the coherent non-directional energy proportion for a subband is determined as a non-directional energy ratio for the subband multiplied by a coherence value for the subband, wherein the non-directional energy ratio is related to an energy ratio remaining in the subband after the energy ratio value for the subband has been removed, and summing the determined coherent non-directional energy proportions for all of the plurality of subbands; and The importance metric is used to determine whether to encode coherent values ​​for the plurality of subbands of the frame.

Images