Apparatus, method and computer program for enabling spatial audio rendering - Patents.com

Abstract

Examples of the disclosure relate to an apparatus, method, and computer program for enabling rendering of spatial audio including both direct audio and indirect audio. The apparatus may be configured to obtain a spatial audio signal comprising one or more audio signals and associated spatial metadata. The associated spatial metadata is configured to enable rendering of spatial audio from the one or more audio signals. The spatial audio includes direct audio and indirect audio. The apparatus is also configured to determine directional distribution information for the indirect audio using at least the associated spatial metadata. The apparatus is also configured to determine rendering information corresponding to the determined directional distribution information, and enable rendering of the spatial audio using the determined rendering information, the one or more audio signals, and the associated spatial metadata.

Description

[Technical field]

[0001] Examples of the present disclosure relate to apparatus, methods, and computer programs for enabling rendering of spatial audio, some of which include both direct and indirect audio. [Background technology]

[0002] Spatial audio allows the spatial properties of a sound scene to be reproduced for a user so that the user can perceive the spatial properties, which can provide an immersive audio experience for the user, or can be used for other applications. Summary of the Invention

[0003] According to various, but not necessarily all, examples of the present disclosure, an apparatus may be provided that includes means for obtaining a spatial audio signal comprising one or more audio signals and associated spatial metadata, the associated spatial metadata configured to enable rendering of spatial audio from the one or more audio signals, the spatial audio including direct audio and indirect audio; means for determining directional distribution information for the indirect audio using at least the associated spatial metadata; means for determining rendering information corresponding to the determined directional distribution information; and means for enabling rendering of the spatial audio using the determined rendering information, the one or more audio signals, and the associated spatial metadata.

[0004] Indirect audio may include omnidirectional audio.

[0005] The indirect audio may include diffuse audio.

[0006] The determined directional distribution information may indicate one or more directions associated with the indirect audio.

[0007] The rendering information may include a target covariance matrix of the audio signal.

[0008] The rendering information may include diffusion audio gains for channels of a multi-channel loudspeaker arrangement.

[0009] The means may be for using at least the associated spatial metadata to determine directional information for the direct audio.

[0010] The associated spatial metadata may include information that enables mixing of audio signals to enable rendering of spatial audio in a selected audio format.

[0011] The associated spatial metadata may include information indicative of at least one of sound direction and sound directivity for one or more frequency subbands.

[0012] The associated spatial metadata may include one or more prediction coefficients for one or more frequency subbands.

[0013] The associated spatial metadata may comprise one or more coherence parameters.

[0014] According to various, but not necessarily all, examples of the present disclosure, there is provided an electronic device comprising an apparatus as described herein, the electronic device being at least one of a phone, a camera, a computing device, and a videoconferencing device.

[0015] In accordance with various, but not necessarily all, examples of the present disclosure, a method is provided that includes obtaining a spatial audio signal including one or more audio signals and associated spatial metadata, the associated spatial metadata configured to enable rendering of spatial audio from the one or more audio signals, the spatial audio including direct audio and indirect audio; determining directional distribution information for the indirect audio using at least the associated spatial metadata; determining rendering information corresponding to the determined directional distribution information; and enabling rendering of the spatial audio using the estimated target spatial features, the one or more audio signals, and the associated spatial metadata.

[0016] In accordance with various, but not necessarily all, examples of the present disclosure, a computer program is provided that causes a processing circuit to execute computer program instructions including: obtaining a spatial audio signal comprising one or more audio signals and associated spatial metadata, the associated spatial metadata configured to enable rendering of spatial audio from the one or more audio signals, the spatial audio including direct audio and indirect audio; determining directional distribution information for the indirect audio using at least the associated spatial metadata; determining rendering information corresponding to the determined directional distribution information; and enabling rendering of the spatial audio using the estimated target spatial features, the one or more audio signals, and the associated spatial metadata.

[0017] Some examples will now be described with reference to the accompanying drawings. [Brief description of the drawings]

[0018] [Figure 1] FIG. 1 illustrates an exemplary system. [Diagram 2] FIG. 1 illustrates an exemplary method. [Diagram 3]FIG. 2 illustrates an exemplary decoder. [Figure 4] FIG. 2 illustrates an exemplary spatial combiner. [Diagram 5] FIG. 1 illustrates an exemplary apparatus. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] Examples of the present disclosure enable rendering of spatial audio comprising both direct and indirect audio. In some circumstances, the finite time and / or frequency resolution of audio processing may lead to some audio being identified as indirect or diffuse audio even though this was not the case in the original sound scene.

[0020] For example, in Directional Audio Coding (DirAC), the direction and diffuseness of audio are analyzed in frequency bands. This analysis is based on intensity. If the original sound scene contains only one dominant source in a time-frequency tile, it is likely to be analyzed as non-diffuse. However, if the original sound scene contains two or more sound sources with similar dominance levels, there may be some time-frequency tiles where sounds from different sources have similar intensity levels. This may lead to sounds being treated as diffuse audio or indirect sound, even if this is not the case in the original sound scene. This leads to other sounds being analyzed as diffuse audio in addition to the true diffuse audio.

[0021] Examples of the present disclosure address this issue by estimating directional distribution information of the indirect audio based on spatial metadata associated with the audio signal. The directional distribution information can then be used to enable rendering of the audio signal and avoid analyzing non-diffuse audio as a purely diffuse sound with a surround spatial distribution. This leads to improved spatial audio.

[0022] 1 illustrates an example system 101 that may be used to implement examples of the present disclosure. The system includes an encoder 105 and a decoder 109. In some examples, the encoder 105 and the decoder 109 may be in different devices. In some examples, the encoder 105 and the decoder 109 may be in the same device.

[0023] The system 101 is configured such that the encoder 105 obtains an input including a spatial audio signal 103. In this example, the spatial audio signal 103 may be a First Order Ambisonics (FOA) signal. Other types of spatial audio signals 103 may be used in other examples of this disclosure.

[0024] The spatial audio signals 103 may be obtained from two or more microphones configured to capture spatial audio. In examples where the audio signals 103 include FOA audio signals, the FOA audio signals may be obtained from a dedicated Ambisonics microphone, such as an Eigenmike, or any other suitable means. The spatial audio signals represent a sound scene. The sound scene may include one or more sound sources. In some examples, the spatial audio signals 103 may be obtained from sources other than microphones, for example, they may comprise multi-channel loudspeaker signals, such as 5.1 signals.

[0025] The spatial audio signal 103 may include direct audio and indirect audio. Direct audio may include audio arriving at the microphone from a particular direction. Direct audio may include audio from one or more dominant sound sources in the sound scene. Indirect audio may include background or ambient noise that appears to be omnidirectional and / or arrives from a wide range of directions. Indirect audio may also include audio that may be presumed to be omnidirectional or diffuse, but is not actually diffuse in the original sound scene. For example, if there are multiple sound sources of similar intensity levels, they may be analyzed as omnidirectional, or there are reflections of one or more sound sources.

[0026] The encoder 105 may comprise any means that may be configured to encode the audio signal 103 and provide a bitstream 107 as an output. The encoder 105 may be configured to use a parametric method to encode the audio signal 103. The parametric method may include an Immersive Voice and Audio Services (IVAS) method or any other suitable type of method. The encoder 105 may be configured to determine a transport audio signal and spatial metadata with the audio signal 103. The transport audio signal and spatial metadata may be multiplexed to provide the bitstream 107.

[0027] In some examples, the bitstream 107 may be transmitted from a device comprising the encoder 105 to a device comprising the decoder 109. In some examples, the bitstream 107 may be stored in a device comprising the encoder 105 and retrieved and decoded by the decoder 109 as appropriate.

[0028] The decoder 109 is configured to receive a bitstream 107 as an input. The decoder 109 comprises means that may be configured to decode the bitstream 107. The decoder 109 may decode the bitstream to generate transport audio and spatial metadata. The decoder 109 may be configured to render a spatial audio output 111 using the decoded spatial metadata. The spatial audio output 111 may be provided in any suitable format, such as a binaural audio signal.

[0029] 2-4 illustrate example methods and portions of a system 101 that enable the determination of directional distribution information for indirect audio and that may be used to render improved spatial audio.

[0030] 2 illustrates an example method that may be used to enable rendering of spatial audio in different audio formats, which may be implemented in a system 101, such as the system 101 illustrated in FIG.

[0031] The method includes, in block 201, obtaining a spatial audio signal 103. The spatial audio signal 103 may include an encoded audio signal. The audio signal may include multiple channels of audio, or a single channel of audio. In some examples, the spatial audio signal 103 may comprise an unencoded audio signal, such as an audio signal based on a signal from a microphone integrated into the device, or any other suitable type of audio.

[0032] The spatial audio signal 103 may include one or more audio signals and associated spatial metadata configured to enable rendering of spatial audio from the one or more audio signals.

[0033] The spatial metadata may include information that enables mixing of audio signals to enable rendering of spatial audio in a selected audio format.

[0034] Spatial metadata may be provided in frequency subbands. The spatial metadata may include information indicative of the direction of the sound and information indicative of the directivity of the sound for one or more frequency subbands. Sound directivity may be an indication of how directional or omnidirectional the sound is. Sound directivity may provide an indication of whether the sound is ambient or provided from a point source. Sound directivity may be provided as an energy ratio of direct sound versus ambient sound, or in any other suitable format. In some examples, the spatial metadata includes one or more coherence parameters, or any other suitable parameters.

[0035] The spatial metadata may be provided in any suitable format corresponding to the audio signal and / or the format used for rendering. For example, if the format used is a FOA signal, the spatial metadata may include information indicating how to predict the FOA signal from the audio signal for one or more frequency subbands. The audio signal may include multiple channels of audio, or a single channel of audio. Such information may comprise prediction coefficients for predicting the FOA signal from the transport audio signal. The transport audio signal may include multiple channels of audio, or a single channel of audio. For example, the FOA omni-directional signal W may be used as the transport audio signal, and the prediction coefficients may be used to predict the dipole signals X, Y, and Z from the transmit signal W.

[0036] Spatial audio includes direct audio and indirect audio. Indirect audio may include omnidirectional audio. In some examples, indirect audio may include diffuse audio. Indirect audio may also include audio that is not truly indirect in the original sound scene, but may be analyzed as indirect due to limitations in frequency and / or temporal and / or spatial resolution of the processing.

[0037] Indirect audio may include audio that covers a wider range of angles than direct audio. Direct audio may come from a single angle or a narrow range of angles, while indirect audio may come from a wide range of angles. In some examples, indirect audio may originate from all directions rather than from a specific direction.

[0038] At block 203, the method includes using the associated spatial metadata to determine directional distribution information for the indirect audio. In some examples, information in addition to the spatial metadata may be used to determine the directional distribution information.

[0039] The determined directional distribution information indicates one or more directions associated with the indirect audio. For example, if the indirect audio includes audio from multiple sound sources of similar dominance levels, the directional distribution information may include information regarding the respective directions of the different sound sources.

[0040] The determined directional distribution information may comprise information that may be used to adjust the spatialization of the audio to render the indirect audio from the correct direction instead of rendering the indirect audio as ambient or completely ambient.

[0041] In block 205, the method includes determining rendering information corresponding to the determined directional distribution information. The rendering information may include target spatial characteristics that should enable a listener to perceive spatial audio with sound sources in correct positions corresponding to the original sound scene.

[0042] The rendering information may include parameters that indicate how the directional distribution information corresponds to spatial audio characteristics that may be perceived by a listener.

[0043] The rendering information may be determined in any suitable format. The format used for the rendering information may depend on the format used for the spatial audio output 111. For example, the rendering information may include a target covariance matrix of the audio signal or diffuse audio gains of the channels of a multi-channel loudspeaker array or any other suitable type of information. The diffuse audio gains may provide an approximation of how diffuse the sound is and how the sound should be distributed in space.

[0044] At block 207, the method includes enabling rendering of spatial audio using the determined rendering information, the one or more audio signals and associated spatial metadata.

[0045] In some examples, the method also includes determining directional distribution information for the direct audio as well as the indirect audio. The spatial metadata and any other suitable information may be used to determine the directional distribution information for the direct audio.

[0046] Figure 3 illustrates in schematic form an exemplary decoder 109. The exemplary decoder 109 may be provided in a system 101, such as the system of Figure 1. The exemplary decoder 109 may be configured to implement a method, such as the method of Figure 2, to enable improved rendering of spatial audio.

[0047] The decoder 109 receives as input a bitstream 107. The bitstream 107 may include spatial metadata and corresponding audio signals. The audio signals include direct and indirect audio.

[0048] The bitstream 107 is provided to a demultiplexer 301. The demultiplexer 301 is configured to demultiplex the bitstream 107 into multiple streams. In the example of Fig. 3, the demultiplexer 301 demultiplexes the bitstream 107 into a first stream and a second stream. The first stream comprises the encoded spatial metadata 303 and the second stream comprises the encoded transport audio signal 319.

[0049] The encoded transport audio signal 319 is provided to a transport audio signal decoder 321. The transport audio signal decoder 321 is configured to decode the encoded transport audio signal 319 and provide as output a decoded transport audio signal 323. The process used to decode the encoded transport audio signal 319 may include a corresponding process used by the encoder 105 to encode the audio signal. The transport audio signal decoder 321 may comprise an Enhanced Voice Services (EVS) decoder, an Advanced Audio Coding (AAC) decoder, or any other suitable type of decoder.

[0050] The decoded transport audio signal 323 is provided to a time-frequency transform block 325. The time-frequency transform block 325 is configured to modify a domain of the decoded transport audio signal 323. In some examples, the time-frequency transform block 325 is configured to convert the decoded transport audio signal 323 into a time-frequency representation. The time-frequency transform block 325 may be configured to use any suitable means for modifying the domain of the decoded transport audio signal 323. For example, the time-frequency transform block 325 may be configured to use a short-time Fourier transform (STFT), a complex-modulated quadrature mirror filter (QMF) bank, low-delay variants thereof, or any other suitable means.

[0051] The time-frequency transformation block 325 provides as output a time-frequency transport audio signal 327 .

[0052] The encoded first spatial metadata 303 is provided as an input to a metadata decoder 305. The metadata decoder 305 is configured to decode the encoded first spatial metadata 303 and provide as output decoded spatial metadata 307. The process used to decode the encoded spatial metadata 303 may include a corresponding process used by the encoder 105 to encode the first spatial metadata. The metadata decoder 305 may include any suitable type of decoder.

[0053] The decoded spatial metadata 307 may be provided in any suitable format. For example, if the audio signal is encoded for FOA rendering, the decoded spatial metadata 307 may be in a format that enables FOA rendering. For example, the decoded spatial metadata 307 may include FOA prediction coefficients. The FOA prediction coefficients include information that can be converted into rendering information, such as a mixing matrix. The rendering information or mixing matrix may include any information that indicates how the audio signals should be mixed and / or decorrelated to generate an audio output in an Ambisonic format. In other examples, different types of audio formats may be used.

[0054] The decoded first spatial metadata 307 is provided as an input to a rendering information determiner block 309. The rendering information determiner block 309 may be configured to determine rendering information 311. The rendering information 311 may include one or more mixing matrices and / or any other suitable rendering information. The rendering information determiner block 309 provides the rendering information 311, such as a mixing matrix, as an output. The rendering information 311 may be different from the rendering information obtained in block 205 of FIG. 2 corresponding to the directional distribution information.

[0055] The rendering information 311 can be determined based on the decoded spatial metadata 307 .

[0056] In examples where the audio signal is encoded for FOA rendering, the rendering information 311 may include mixing matrices, which may be denoted as A(i,j,k,n), where i is the output channel index, j is the input channel, k is the frequency band, and n is the time frame. The mixing matrices may be used to render the FOA signal by applying them to the transport audio signal and / or a decorrelated version of the transport audio signal.

[0057] In other examples, other types of rendering information may be obtained. In some examples, the decoded spatial metadata 307 may already be a mixing matrix or other rendering information, so in such examples, it is not necessary to use the rendering information determiner block 309.

[0058] A mixing matrix or other rendering information may be used to render the decoded time-frequency transport audio signal 327. As an example, the decoded time-frequency transport audio signal 327 may be denoted as a column vector s(b,n), where b is a frequency bin index and the rows of the vector represent the transport audio signal channels. The number of rows may be between 1 and 4, depending on the applied bit rate and any other suitable factors. If the number of channels is less than 4, the number of rows of the vector s(b,n) is also less than 4. In such an example, a new channel may be appended to the column vector to form a vector s'(b,n) having four rows. The new channel may be a decorrelated version of the first channel signal of s(b,n).

[0059] The FOA signal is then y(b,n)=A(k,n)s'(b,n) where k is the frequency band that frequency bin b resides in. The spatial metadata for a frequency band can correspond to one or more frequency bins of the filterbank used to transform the audio signal.

[0060] In the above equation, the mixing matrix A can be expressed as follows:

number

[0061] This notation means that the time resolution (i.e., metadata time resolution) of the signal s(b,n) and the mixing matrix A(k,n) are the same. This may be the case for system 101 using a filter bank such as STFT configured to apply a coarse time resolution. The coarse time resolution may use a time step of about 20 milliseconds for the filter bank. Other filter banks may have a finer time resolution. In such cases, the time resolution of the spatial metadata would be sparser than the resolution of the audio signal. In these examples, the same mixing matrix may be applied to multiple different time indexes of the audio signal, or the mixing matrix may be interpolated in time.

[0062] In an example of the present disclosure, the decoded spatial metadata 307 and / or rendering information obtained from the spatial metadata may be used to determine directional distribution information 331 indicative of a direction associated with at least some of the indirect audio.

[0063] In the example shown in Figure 3, the rendering information 311 is provided as an input to a rendering metadata determiner 313. The rendering metadata determiner 313 also receives a time-frequency transport audio signal 327 as an input.

[0064] The rendering metadata determiner 313 is configured to determine rendering metadata 315 suitable for rendering the spatial audio in any suitable format. The format of the rendering metadata 315 may be different from the format used for the decoded spatial metadata 307. For example, the decoded spatial metadata 307 may be suitable for a FOA format, and the rendering metadata 315 may be suitable for another format, such as a binaural format.

[0065] In the example of Figure 3, the rendering metadata 315 includes directional (azimuth, elevation) θ(k,n), φ(k,n) parameters and a direct-to-total energy ratio r(k,n) parameter. The parameters are provided in frequency bands. Other types of parameters may be used in other examples of the present disclosure. Other parameters may be used instead of or in combination with the directional and energy ratio parameters.

[0066] In some examples, the decoded spatial metadata 307 may already include the direction parameters and the energy ratio parameters and / or any other suitable parameters of the rendering metadata 315. In such examples, there is no need to use the rendering metadata determiner block 313.

[0067] The rendering information 311 is also provided as an input to a directivity distribution determiner block 329. The directivity distribution determiner block 329 is configured to use the rendering information 311 or the spatial metadata or other information obtained from the spatial metadata to determine the directivity distribution information. The directivity distribution information can be determined for both indirect audio and direct audio. In some examples, the directivity distribution information may be determined only for indirect audio, and the directivity distribution for the direct audio may be provided by a direction parameter of the spatial metadata.

[0068] In examples where the rendering information 311 includes a mixing matrix A(i,j,k,n), the mixing matrix A(i,j,k,n) is forwarded to the directivity distribution determiner block 329. In some examples, there may be only one transport audio signal and the rest of the audio signals to be processed by the mixing matrix A(i,j,k,n) may be decorrelated versions of the single transport audio signal. In other examples, there may be multiple transport audio signals. In such cases, the energy of the signals must be taken into account. In case of multiple transport audio signals, the directivity distribution determiner block 329 may also be configured to receive as input a time-frequency transport audio signal 327, or a covariance matrix of such a signal.

[0069] In the example of using FOA type parameters in the decoded spatial metadata 307, the first column of the mixing matrix A(i,j,k,n) (i.e., j=1) provided as the rendering information 311 corresponds to the direct audio, or mainly the direct audio. This part of the audio is coherently rendered from the same input, e.g., the omni-directional signal W.

[0070] Conversely, the other columns of the mixing matrix A(i,j,k,n) (i.e., 2≦j≦4) correspond to indirect audio, or primarily indirect audio. This portion of the audio is to be decorrelated or rendered incoherent.

[0071] In the example of the present disclosure, the directional distribution information 331 may be estimated from the columns of the mixing matrix A(i,j,k,n) corresponding to the indirect audio, for example, from the last three columns of the mixing matrix A(i,j,k,n).

[0072] Any suitable process may be used to determine the directivity distribution information 331. In some examples, the directivity distribution information 331 may be estimated by determining a gain for indirect audio. The indirect audio may be diffuse audio. The gain for the indirect audio may be determined in the X, Y, and Z directions. In this example, it is assumed that the columns of the mixing matrix A(i,j,k,n) are in the order X, Y, and Z. In other examples, they may be in other orders.

[0073] The gain is:

number

number

number

[0074] These gains are then added together to form the indirect sound gain g sum,diff These can be added together to determine the sum (k,n):

number

[0075] The proportion of indirect audio is then determined for each of the X, Y, and Z directions.

number

number

number

[0076] The directional distribution information 331 is the ratio r x,diff (k,n),r y,diff (k,n), and r z,diff(k,n) The indirect audio ratio r x,diff (k,n),r y,diff (k,n), and r z,diff (k,n) may be provided as an output of the directional distribution determiner 329.

[0077] Note that this is just one example of a parameterization of the directional distribution information 331. Other parameterizations for the directional distribution information 331 may be used in other examples.

[0078] The rendering metadata 315, the time-frequency transport audio signal 327, the rendering information 311 and the directivity distribution information 331 are provided as inputs to a spatial synthesizer 317. The spatial synthesizer 317 is configured to render the spatial audio output 111 using the rendering metadata 315, the time-frequency transport audio signal 327, the rendering information 311 and the directivity distribution information 331. The spatial audio output may comprise a binaural output, a multi-channel speaker signal, a Higher Order Ambisonic (HOA) signal, or any other suitable audio format.

[0079] 4 illustrates generally an exemplary spatial combiner 317. The exemplary spatial combiner 317 may be provided within a decoder 109, such as the decoder 109 illustrated in FIG.

[0080] The spatial synthesizer 317 receives the rendering metadata 315, the time-frequency transport audio signal 327, the directional distribution information 331 and the rendering information 311 as inputs.

[0081] 4, the time-frequency transport audio signal 327 and the rendering information 311 are provided as inputs to a synthesis input generator 401. The synthesis input generator 401 is configured to convert the time-frequency transport audio signal 327 into a format suitable for processing by the remaining blocks in the spatial synthesizer 317.

[0082] The process performed by the synthesis input generator 401 may depend on the number of transport channels used. In examples where the time-frequency transport audio signal 327 includes a single channel (mono transport), the synthesis input generator 401 may allow the time-frequency transport audio signal 327 to pass through without performing any processing on the time-frequency transport audio signal 327. In some examples, the single-channel signal may be replicated to generate a signal including two or more channels. This may provide a dual mono or pseudo-stereo signal, so that all subsequent blocks within the spatial synthesizer 317 may assume a stereo track, even if originally there was only one transport audio signal.

[0083] In an example where the time-frequency transport audio signal 327 comprises multiple channels, the synthesis input generator 401 may be configured to generate a stereo track, which may represent a cardioid pattern pointing in different directions, such as towards the left and towards the right.

[0084] The cardioid pattern can be obtained by using any suitable process, for example by applying a matrix A'(k,n) to the time-frequency transport audio signal 327. The matrix A'(k,n) comprises the first two rows of the matrix A(k,n). This therefore provides the W and Y spherical harmonic signals.

[0085] After applying the matrix A′(k,n), a left-right cardioid beamforming matrix may be applied to provide a preprocessed transport audio signal 403 .

[0086] The pre-processed transport audio signal x(b,n) 403 may be expressed as:

number

[0087] The pre-processed transport audio signal x(b,n) 403 is provided as an output of the synthesis input generator 401. The covariance matrix determiner may be configured to determine an input covariance matrix and a target covariance matrix.

[0088] The preprocessed transport audio signal 403, the rendering metadata 315 and the directivity distribution information 331 may be provided as inputs to a covariance matrix determiner 411. The covariance matrix determiner 411 is configured to determine an input covariance matrix and a target covariance matrix. The input covariance matrix represents the preprocessed transport audio signal 403 and the target covariance matrix represents the time-frequency space audio signal 407.

[0089] The input covariance matrix may be determined from the pre-processed transport audio signal 403 by:

number

[0090] As mentioned above, in this example the time resolution of the covariance matrix is ​​the same as the time resolution of the audio signal, in other examples the time resolution may be different, for example where a filter bank with high time selectivity is used.

[0091] If there are two or more transport audio signals in of s(b,n), the covariance matrix C x (k,n) can be expressed as follows:

number

number

[0092] Note that s'(b,n) was s(b,n) with the decorrelated version of the first channel of s(b,n) added. Therefore, C s’ (k,n) can also be estimated by estimating the covariance matrix of s(b,n) and then zero-padding it to a size of 4 × 4. The zero-padded diagonal entries can then be placed in the first diagonal entries of the estimated covariance matrix.

[0093] The target covariance matrix can be determined based on the rendering metadata 315 and the overall signal energy and directional distribution information.

[0094] The overall signal energy O(k,n) can be obtained as the average of the diagonal values ​​of C_x(k,n) or can be determined based on the all-directional components of the signal A'(k,n)s'(b,n). Then, in some examples, the rendering metadata 315 includes the directions θ(k,n), φ(k,n) and the direct-to-total ratio parameter r(k,n). If the output is assumed to be a binaural signal, the target covariance matrix is:

number

[0095] Here, h(k,θ(k,n),φ(k,n)) is the head-related transfer function column vector for band k and directions θ(k,n),φ(k,n).

[0096] h(k,θ(k,n),φ(k,n)) is a column vector. The vector contains two values. The values ​​can be complex-valued. These values ​​correspond to the left-ear and right-ear Head Related Transfer Function (HRTF) amplitudes and phases. At high frequencies, no phase difference is needed for perceptual reasons, so the HRTF values ​​can contain real values.

[0097] Any suitable process can be used to derive the HRTFs. The HRTFs can be derived for a given direction and frequency.

[0098] In the above formula, C d (k) is the indirect audio binaural covariance matrix. In systems not using implementations of the present disclosure, indirect audio can be assumed to be diffuse. Indirect audio can be assumed to be completely ambient and originate from all directions.

[0099] The indirect audio binaural covariance matrix, corresponding to the full ambient and diffuse audio distribution, can be obtained by choosing a uniform spatial distribution in the direction DOA_d, where d=1..D, and is expressed by:

number

[0100] In the examples of the present disclosure, indirect audio is not assumed to be always uniform. Indirect audio is not assumed to be completely diffuse or to originate from all directions. The inhomogeneity of indirect audio needs to be taken into account by the indirect audio binaural covariance matrix. As an example, the inhomogeneity of indirect audio can be described as follows:

[0101] First, the indirect binaural covariance matrices are determined for X, Y, and Z. These matrices can be determined as follows:

number

number

number

[0102] Here, x(DOA d ), y(DOA d), and z(DOA d ) is D.O.A. d The Cartesian coordinates corresponding to C bin,diff,x (k) contains the indirect audio contribution mainly coming from the X direction, and C bin,diff,y (k) contains the indirect audio contributions mainly coming from the Y direction, and C (bin,diff,z (k) contains the indirect audio contribution coming mainly from the Z direction.

[0103] The indirect audio binaural covariance matrix with directional distribution is the diffuse audio ratio r x,diff (k,n),r y,diff (k,n), and r z,diff (k,n), as well as the diffuse binaural covariance matrices C in the X, Y, and Z directions. bin,diff,x (k), C bin,diff,y (k), C bin,diff,z For example, the indirect audio binaural covariance matrix with directional distribution can be determined by:

number

[0104] Indirect Audio Ratio x,diff (k,n),r y,diff (k,n), and r z,diff In the example where (k,n) are equal, each of the ratios is 1 / 3, and the resulting indirect audio binaural covariance matrix C d (k,n) is the uniform diffusion binaural covariance matrix C bin,diff,uniform (k). In some cases, C bin,diff,x (k), C bin,diff,y (k), and C bin,diff,z It may be possible to make slight adjustments to the values ​​of (k) so that their average is exactly C bin,diff,uniform (k). Therefore, rendering of the audio signal with an even diffusion distribution (i.e., r x,diff (k,n),r y,diff (k,n), and r z,diff(where k,n are all 1 / 3) will yield the same results as an implementation that does not use the examples of this disclosure.

[0105] The covariance matrix determiner 411 provides as output a covariance matrix 413. The covariance matrix 413 provided as output is a function of the input covariance matrix C x (k,n) and the target covariance matrix C y It may include (k,n).

[0106] In the above equations, it is implied that processing is performed uniformly within the bins of each band k. In some examples, processing is performed at a higher frequency resolution, which is for each frequency bin b. In such examples, the equations given above are adapted such that a covariance matrix is ​​determined for each bin b, but using the parameters of the rendering metadata 315 for the band k in which the bin resides.

[0107] In some examples, the input covariance matrix and the target covariance matrix may be time-averaged. The time averaging may be implemented using infinite impulse repose (IIR), finite impulse response (FIR) averaging, or any other suitable type of time averaging. The covariance matrix determiner 411 may be configured to perform the time averaging such that a time-averaged covariance matrix 413 is provided as an output.

[0108] In this example for obtaining the target covariance matrix, only parameters related to the direction and energy ratio are considered. In other examples, other parameters can be taken into account when obtaining the target covariance matrix. For example, in addition to the direction and energy ratio, spatial coherence parameters, or any other suitable parameters, can be considered. The use of other types of parameters can enable the spatial audio output to be provided in a format other than a binaural format and / or can improve the accuracy with which the spatial sound can be reproduced.

[0109] The processing matrix determiner 415 determines the covariance matrix 413C x (k,n) and C y(k,n) as an input. The processing matrix determiner 415 is configured to determine the covariance matrix C x (k,n) and C y (k,n) to obtain the processing matrices M(k,n) and M r The processing matrices M(k,n) and M(k,n) may be determined using any suitable process. r In some examples, the process used is to determine the target covariance matrix C(k,n) based on the determined audio signal. y To achieve (k,n), we use the measured covariance matrix C x The method may include determining a mixing matrix for processing an audio signal having (k,n). Such a method may be used to generate binaural audio signals or surround loudspeaker signals or other types of audio signals. To formulate the processing matrix, the method may include using a matrix such as a prototype matrix. A prototype matrix is ​​a matrix that indicates what kind of signal is intended for each of the outputs for the optimization procedure. This may be within the constraint that the outputs must achieve a target covariance matrix. In an example where the spatial audio output format is a binaural format, the prototype matrix is:

number

[0110] This prototype matrix indicates that the signal for the left ear is rendered primarily from the left pre-processing transport channel, and the signal for the right ear is rendered primarily from the right pre-processing transport channel. In some examples, the orientation of the user's head can be tracked. If it is determined that the user is now facing towards the rear hemisphere, the prototype matrix becomes:

number

[0111] The processing matrix determiner 415 may be configured to determine the processing matrices M(k,n) and M_r(k,n) based on the prototype matrices and the input and target covariance matrices using the means described in Vilkamo, J., Backstroom, T., & Kuntz, A. (2013) “An Optimized Covariance Domain Framework for Time-Frequency Processing of Spatial Audio” “Journal of the Audio Engineering Society”, 61(6), 403-411. The processing matrix determiner 415 determines the processing matrices M(k,n) and M r (k,n) 417 as an output.

[0112] Processing matrices M(k,n) and M r The decorrelation and mixing block 405 also receives as input the preprocessed transport audio signal x(b,n) 403. The decorrelation and mixing block 405 receives the processing matrices M(k,n) and M r The pre-processed transport audio signal x(b,n) 403 may comprise any means that may be configured to decorrelate and mix the pre-processed transport audio signal x(b,n) 403 based on the pre-processed transport audio signal x(b,n) 417 .

[0113] Any suitable process may be used to decorrelate and mix the preprocessed transport audio signal x(b,n) 403. In some examples, decorrelation and mixing of the preprocessed transport audio signal x(b,n) 403 may be performed by processing the preprocessed transport audio signal x(b,n) 403 with the same prototype matrix applied by the processing matrix determiner 415 and combining the decorrelated signal x(b,n) 403 with the same prototype matrix applied by the processing matrix determiner 415. D and decorrelating the result to generate the decorrelated signal x (b,t). D (b,t) (pre-processed transport audio signal x(b,n) 403 ) may then be mixed using any suitable mixing procedure to generate a time-frequency audio signal 407 .

[0114] In some examples, the following mixing procedure may be used to generate the time-frequency audio signal 407.

number

[0115] As mentioned before, the notation used here is that of the processing matrices M(k,n) and M r It means that the time resolution of the processing matrix 417 as well as the preprocessed transport audio signal x(b,n) 403 is the same. In other examples, they may have different time resolutions. For example, the time resolution of the processing matrix 417 may be sparser than the time resolution of the preprocessed transport audio signal 403. In such examples, an interpolation process such as linear interpolation may be applied to the processing matrix 417 to achieve the same time resolution of the preprocessed transport audio signal 403. The interpolation rate may depend on any suitable factor. For example, the interpolation rate may depend on whether a start has been detected or not. If a start has been detected, fast interpolation may be used, and if a start has not been detected, normal interpolation may be used.

[0116] The decorrelation and mixing block 405 provides a time-frequency spatial audio signal 407 as an output.

[0117] The time-frequency spatial audio signal 407 is provided as an input to an inverse filter bank 409. The inverse filter bank 409 is configured to apply an inverse transform to the time-frequency spatial audio signal 407. The inverse transform applied to the time-frequency spatial audio signal 407 may be a transform corresponding to the transform used to convert the decoded transport audio signal 323 into the time-frequency transport audio signal 327 of Figure 3.

[0118] The inverse filter bank 409 is configured to provide as an output the spatial audio output 111. The spatial audio output 111 may be provided in any suitable audio format.

[0119] Different examples may use different methods instead of the covariance matrix based rendering other than the example used in FIG. 4. For example, in another example, the audio signal may be split into a directional part and an omnidirectional part. A ratio parameter from the spatial metadata may be used to split the signal into a directional part and an omnidirectional part. The directional part may then be positioned to a virtual loudspeaker using amplitude panning or any other suitable means. The omnidirectional part may then be distributed to all speakers and de-correlated. The processed directional and omnidirectional parts may then be added together. Each of the virtual speakers may then be processed with an HRTF to obtain a binaural output.

[0120] Thus, a system implementing examples of the present disclosure may be implemented such that the indirect audio is not distributed evenly, but instead is distributed over a ratio of indirect audio r x,diff (k,n),r y,diff (k,n), and r z,diff It is possible to provide a distribution of indirect audio to the virtual speakers based on (k,n) as well as the direction of the virtual speakers.

[0121] For example, the omnidirectional gain of a virtual speaker is calculated by multiplying the squared x coordinate of the virtual speaker by the diffuse audio ratio r x,diff Multiply (k,n) to the squared y coordinate of the virtual speaker and apply the diffuse audio ratio r y,diff The squared z-coordinate of the virtual speaker can be obtained by multiplying (k,n) by the diffuse audio ratio r z,diff (k,n) and sum the results. The omnidirectional sound gains of all virtual speakers can then be normalized so that the sum of their squares is equal to 1.

[0122] A similar approach can be used for multi-channel speaker output, where virtual loudspeakers replace the real loudspeakers.

[0123] In the above example, the bitstream 107 contains only a single encoded transport audio signal 319, and the mixing is performed using the single transport audio signal and a decorrelated version of it. As a result, each input signal to the mixing has the same energy and the spreading audio ratio r x,diff (k,n),r y,diff (k,n), and r z,diff (k,n) can be calculated without considering the energy. There is no need to calculate the energy since the energies of the decorrelated channels should correspond to the energy of the transport audio signal from which they were generated (this is the first omnidirectional transport audio signal). The energy value of the transport audio signal can be used instead.

[0124] In some examples, the bitstream 107 may include multiple transport audio signals. In such examples, the spreading audio ratio r x,diff (k,n),r y,diff (k,n), and r z,diff The energy needs to be taken into account when (k,n) is calculated. In such an example, the energy E(k,n,j) of the transport audio signal is calculated in frequency bands.

number

[0125] The diffusion audio gain may then be calculated using the energy E(k,n,j) and the mixing matrix A(i,j,k,n).

number

number

number

[0126] It should be noted that these formulas may be used for scenarios for use with a single transport audio signal, even though they are computationally more complex.

[0127] The remaining processing can be done as described above, i.e. the diffuse audio ratio can be calculated using these diffuse audio gains.

[0128] Examples of the present disclosure may be implemented in a system that enables head tracking of a listener's head orientation. In such an example, the matrix A(i,j,k,n) is a function of the diffuse audio gain g x,diff (k,n),g y,diff (k,n), and g z,diff Before (k,n) is estimated, it may be rotated using a rotation matrix according to the listener's head orientation. For example, when the FOA signal is generated from the transport audio signal by y(b,n)=A(k,n)s'(b,n), the rotated FOA signal is:

number

[0129] where R(n) is the FOA rotation matrix according to the listener's head orientation. The rotation matrix can mix the X, Y, Z channels into new X, Y, Z channels and align them according to the current listener's head position. Therefore, the head orientation can be taken into account by using the following rotation mixing matrix instead of A(k, n) in the formula:

number

[0130] In the above example, the spatial metadata in the spatial audio signal was provided in FOA format. In other examples, other formats may be used for the spatial metadata. In such examples, the diffuse audio ratio r x,diff (k,n),r y,diff (k, n), and r z,diff(k,n) can be estimated in different ways. For example, in some examples, the spatial metadata may include the direction (azimuth, elevation) θ(k,n), φ(k,n) and the direct-to-total energy ratio r(k,n) parameters. This type of spatial metadata can be obtained from a mobile device equipped with a microphone array, or from any other suitable type of device, or by using any suitable process.

[0131] In such an example, the diffuse audio ratio may be estimated based on the average direction. This may be implemented as follows: The direction is converted to Cartesian coordinates as follows: d x (k,n) = cosθ(k,n) cosφ(k,n) d y (k,n) = sinθ(k,n)cosφ(k,n) d z (k,n)=sinφ(k,n)

[0132] The diffuse audio gain can be determined by averaging their absolute values ​​over time weighted by how "diffuse" the sound is estimated to be (e.g., 1-r(k,n)).

number

number

number

[0133] In some examples, the spatial metadata may be multiple different types of parameters, or different types of parameters may be obtained from the spatial metadata. For example, the spatial metadata may include both Spatial Audio Rendering (SPAR) parameters and Directional Audio Coding (DirAC) parameters, or any other suitable type of parameters. In such a case, a first type of parameter may be used to calculate the diffuse audio ratio, and a second type of parameter may be used for rendering. For example, the diffuse audio ratio may be calculated using SPAR parameters, and DirAC parameters may be used for rendering.

[0134] In some examples, spatial metadata may be available in a first format for a first set of frequencies, and spatial metadata may be available in a second format for a second set of frequencies. For example, the spatial metadata may comprise SPAR parameters for the first set of frequencies and DirAC parameters for the second set of frequencies. In these cases, different processes may be used to estimate the diffuse audio ratio for the different frequencies. In some cases, different processes may also be used to determine the rendering metadata 315 at the different frequencies.

[0135] In the example shown in Fig. 3 and Fig. 4, the spread audio ratio r x,diff (k,n),ry,diff (k,n), and r z,diff The directional distribution information such as (k,n) is estimated in the decoder 109. In some examples, the diffuse audio ratio r x,diff (k,n),r y,diff (k,n), and r z,diff The directivity distribution information, such as (k,n), may be estimated within the encoder 105 and then transmitted to the decoder 109. The directivity distribution information may be transmitted together with the spatial metadata and any other suitable parameters.

[0136] In the above example, the directional distribution information was obtained separately for the X-axis, Y-axis, and Z-axis. In other examples, the directional distribution information can be obtained for other coordinate systems. For example, the diffuse component can be determined in a rotating coordinate system such that the rotation adaptively maximizes or substantially maximizes the energy of the first axis diffuse component. As an example, there may be one sound source at 45 degrees azimuth and 45 degrees elevation, and another sound source at -135 degrees azimuth and -45 degrees elevation (i.e., in the opposite direction). In this case, an embodiment measures and reproduces the diffuse component primarily on that axis, instead of focusing on fixed X, Y, and Z coordinates.

[0137] In some examples, the configuration of the diffuse components need not follow any set of rotating or non-rotating axes. For example, if the FOA covariance matrix is ​​determined, it is possible to determine the spatial energy spectrum in the spatial distribution of the surrounding directions using a minimum variance distortion-free response (MVDR) beamforming method based on the measurement of the signal covariance matrix y(b,n)=A(k,n)s'(b,n). To do this, the energy from the dth direction (i.e., direction of arrival d (DOA d)) is calculated as d ), corresponding to the sound coming from d, k, n, can be expressed as E'(d, k, n). These energy values ​​can be used to spatially weight the diffuse binaural covariance matrix:

number

[0138] Even if the method for determining the diffuse audio distribution follows a coordinate system such as X, Y, Z or a rotated one, we first calculate the ratio value g (x,y,z),diff By mapping the spatial energy distribution value E'(d,k,n) (or a rotated similar value) to E'(d,k,n) and then applying the above formula, C d The above formula for determining (k,n) can be used.

[0139] In the above example, the decorrelated sound was generated based on decorrelating the left and right preprocessed transport audio signals 403 and mixing them to obtain the residual component. In the mono case where there is a single transport audio signal, this means that the mono sound is decorrelated into left and right decorrelated sounds. The left and right decorrelated sounds are mixed using a covariance matrix based rendering scheme, which may assume the input covariance matrix (of the decorrelated part) to be a diagonal matrix.

[0140] In some examples, a mono transport audio signal may be decorrelated exactly once, and a two-channel signal may be generated by providing a decorrelated signal to the first channel and an inverted (multiplied by -1) decorrelated signal to the second channel. In such cases, the input covariance matrix of the decorrelation part is not diagonal, but the cross terms of the covariance matrix are the same as the diagonal values, but with negative signs. This procedure can be used for situations where the decorrelator itself falls short in generating a signal that can be assumed to be completely incoherent.

[0141] In some examples, the diffuse binaural covariance matrix may be generated based on the estimated FOA covariance matrix. In such examples, the transport signal covariance matrix becomes:

number

[0142] Next, C s (k,n) is zero-padded to size 4×4, and the values ​​of the first diagonal are placed on the zero-padded diagonal to create a padded matrix C s’ (k,n) is obtained. The FOA covariance matrix is ​​as follows:

number

[0143] The FOA matrix can be rotated by:

number

[0144] The diffuse binaural covariance matrix can then be determined as:

number

[0145] Here, H FOA (k) is the FOA to binaural processing matrix. In this method, if the spatial metadata indicates that the audio is indirect audio, then the target covariance matrix is ​​based primarily on the FOA-binaural rendering scheme. However, if the spatial metadata indicates that the sound is direct audio, then the target covariance matrix is ​​based primarily on the rendering metadata, which consists of directional parameters. This covariance matrix C d (k,n) can also be estimated by first generating the following signals:

number

[0146] 5 illustrates a schematic diagram of an exemplary apparatus 501 that may be used in some examples of the present disclosure. The apparatus 501 may comprise a controller apparatus and may be provided within an electronic device such as a phone, a camera, a computing device, a videoconferencing device, or any other suitable type of device.

[0147] 5, device 501 includes at least one processor 503 and at least one memory 505. It should be understood that device 501 may include additional components not shown in FIG.

[0148] 5, the implementation of device 501 may be implemented as a processing circuit. In some examples, device 501 may be implemented solely in hardware, may have some aspects in software including solely firmware, or may be a combination of hardware and software (including firmware).

[0149] As shown in FIG. 5, the apparatus 501 may be implemented with instructions enabling hardware functionality, for example by using executable instructions of a computer program 507 in a general-purpose or special-purpose processor 503, which may be stored in a computer-readable storage medium (disk, memory, etc.) for execution by such processor 503.

[0150] The processor 503 is configured to read from and write to the memory 505. The processor 503 may also include an output interface through which data and / or commands are output by the processor 503, and an input interface through which data and / or commands are input to the processor 503.

[0151] The memory 505 is configured to store a computer program 507 including computer program instructions (computer program code 509) that, when loaded into the processor 503, controls the operation of the device 501. The computer program instructions of the computer program 507 provide logic and routines that enable the device 501 to perform the methods illustrated in Figures 2-4. The processor 503 is able to load and execute the computer program 507 by reading the memory 505.

[0152] Thus, the apparatus 501 includes at least one processor 503, at least one memory 505 including computer program code 509, and computer program code 509 configured to cause the apparatus 501 using the at least one processor 503 to perform at least the steps of obtaining a spatial audio signal including one or more audio signals and associated spatial metadata, the associated spatial metadata being configured to enable rendering of spatial audio from the one or more audio signals, the spatial audio including direct audio and indirect audio, determining directional distribution information for the indirect audio using at least the associated spatial metadata, determining rendering information corresponding to the determined directional distribution information, and enabling rendering of the spatial audio using the determined rendering information, the one or more audio signals, and the associated spatial metadata.

[0153] As shown in FIG. 5, the computer program 507 may be delivered to the device 501 via any suitable delivery mechanism 511. The delivery mechanism 511 may be, for example, a machine-readable medium, a computer-readable medium, a non-transitory computer-readable storage medium, a computer program product, a memory device, a recording medium such as a compact disc read-only memory (CD-ROM) or a digital versatile disk (DVD), or a solid-state memory, an article of manufacture that includes or tangibly embodies the computer program 507. The delivery mechanism may be a signal configured to reliably transfer the computer program 507. The device 501 may propagate or transmit the computer program 507 as a computer data signal. In some examples, the computer program 507 may be transmitted to the device 501 using a wireless protocol such as Bluetooth®, Bluetooth® Low Energy, Bluetooth® Smart, 6LoWPan (IPv6 over low power personal area network) ZigBee, ANT+, Near Field Communication (NFC), Radio Frequency Identification, Wireless Local Area Network (WLAN), or any other suitable protocol.

[0154] The computer program 507 includes computer program instructions for causing the device 501 to perform at least the following steps: obtaining a spatial audio signal comprising one or more audio signals and associated spatial metadata, the associated spatial metadata configured to enable rendering of spatial audio from the one or more audio signals, the spatial audio comprising direct audio and indirect audio; determining directional distribution information for the indirect audio using at least the associated spatial metadata; determining rendering information corresponding to the determined directional distribution information; and enabling rendering of the spatial audio using the determined rendering information, the one or more audio signals, and the associated spatial metadata.

[0155] The computer program instructions may be included in a computer program 507, a non-transitory computer readable medium, a computer program product, a machine readable medium, etc. In some, but not necessarily all, examples, the computer program instructions may be distributed across multiple computer programs 507.

[0156] Although memory 505 is shown as a single component / circuit, it may be implemented as one or more separate components / circuits, some or all of which may be integrated / removable and / or provide permanent / semi-permanent / dynamic / cache storage.

[0157] Although the processor 503 is illustrated as a single component / circuit, it may be implemented as one or more separate components / circuits, some or all of which may be integrated / removable. The processor 503 may be a single-core or multi-core processor.

[0158] References to a "computer-readable storage medium," "computer program product," "tangibly embodied computer program," etc., or to a "controller," "computer," "processor," etc., should be understood to encompass computers having different architectures, such as single / multi-processor architectures and sequential (Von Neumann) / parallel architectures, as well as special purpose circuits, such as field programmable gate arrays (FPGAs), application specific circuits (ASICs), signal processing devices, and other processing circuits. References to computer programs, instructions, code, etc., should be understood to encompass software or firmware for a programmable processor, such as the programmable contents of a hardware device, whether for example, instructions for a processor, or configuration settings for a fixed function device, gate array, or programmable logic device, etc.

[0159] As used in this application, the term "circuitry" may refer to one or more or all of the following: (a) Hardware-only circuit implementation (e.g., implementation in analog and / or digital circuits only) (b) A combination of hardware circuitry and software, such as (where applicable) (i) A combination of analog and / or digital hardware circuitry and software / firmware. (ii) Any portion of software (including digital signal processors), hardware processors having software and memory(s) that work together to cause a device, such as a mobile phone or server, to perform various functions. (c) Hardware circuitry and / or processors, such as a microprocessor or part of a microprocessor that requires software (e.g., firmware) to operate, but the software may not be present when it is not needed for operation.

[0160] This definition of circuitry applies to all uses of the term in this application, including any claims. As a further example, the term circuitry, as used in this application, also encompasses merely hardware circuits or processors and their (or their) accompanying software and / or firmware implementations. The term circuitry also encompasses, for example, baseband integrated circuits for mobile devices, or similar integrated circuits in servers, cellular network devices, or other computing or network devices, if applicable to certain claim elements.

[0161] 2-4 may represent method steps and / or sections of code within the computer program 507. The illustration of a particular order to the blocks does not necessarily imply a required or preferred order to the blocks, and the order and arrangement of the blocks may be modified. Additionally, some blocks may be omitted.

[0162] The term "comprise" is used herein in an inclusive rather than exclusive sense, i.e. any reference to X containing Y indicates that X may contain only one Y or may contain multiple Y. If it is intended to use "comprise" in the exclusive sense, this will be made clear in the context by reference to "comprising only one..." or by using "consisting".

[0163] In this description, various examples are referenced. The description of features or functions with respect to an example indicates that those features or functions are present in that example. Use of the terms "example" or "for example" or "can" or "may" in the text, whether or not explicitly stated, indicates that such features or functions are present in at least the example described, whether or not described as an example, and that they may, but not necessarily, be present in some or all other examples. Thus, "example," "for example," "can" or "may" refers to a particular instance in a class of examples. A property of an instance may be a property of only that instance, or a property of the class, or a property of a subclass of the class that includes some but not all of the instances in the class. Thus, a feature described with respect to one example but not with respect to another example is implicitly disclosed to be usable as part of a working combination in other examples, where possible, but not necessarily in the other examples.

[0164] Although embodiments have been described in the preceding paragraphs with reference to various embodiments, it should be understood that modifications can be made to the given embodiments without departing from the scope of the claims.

[0165] Features set forth in the foregoing description may be used in combinations other than those explicitly set forth above.

[0166] Although functions are described with reference to particular features, these functions may be performed by other features, whether or not described.

[0167] Although features have been described with reference to particular examples, these features may be present in other examples, whether or not described.

[0168] The term "a" or "The" is used herein in an inclusive, not exclusive sense. That is, any reference to X with "a" / "the" Y indicates that X may include only one Y or may include more than one Y, unless the context clearly indicates the contrary. If it is intended to use "a" or "the" in an exclusive sense, it will be clear in the context. In some situations, the use of "at least one" or "one or more" may be used to emphasize an inclusive sense, but the absence of these terms should not be construed to infer any exclusive sense.

[0169] The presence of a feature (or combination of features) in a claim is a reference to that feature or (combination of features) per se, and to features that achieve substantially the same technical effect (equivalent features). Equivalent features include, for example, features that are modifications and achieve substantially the same result in substantially the same way. Equivalent features include, for example, features that perform substantially the same function in substantially the same way to achieve substantially the same result.

[0170] In this description, reference is made to various examples that use adjectives or adjective phrases to describe features of the examples. Such description of a property with respect to the examples indicates that the property is present in some examples exactly as described and in other examples substantially as described.

[0171] While the foregoing specification has attempted to draw attention to those features believed to be important, it should be understood that applicants may seek protection, through the claims, for any patentable feature or combination of features referred to hereinabove and / or shown in the drawings, whether emphasized or not.

Claims

1. 1. An apparatus comprising at least one processor and at least one memory containing computer program code, the at least one memory and the computer program code being configured by the at least one processor to: obtaining a spatial audio signal comprising one or more audio signals and associated spatial metadata, the associated spatial metadata being configured to enable rendering of spatial audio from the one or more audio signals, the spatial audio comprising direct audio and indirect audio; determining directional distribution information for the indirect audio using at least the associated spatial metadata; determining rendering information corresponding to the determined directional distribution information; and enabling rendering of the spatial audio using the determined rendering information, the one or more audio signals, and the associated spatial metadata.

2. The apparatus of claim 1 , wherein the indirect audio comprises omnidirectional audio.

3. The apparatus of claim 1 , wherein the indirect audio comprises diffuse audio.

4. The apparatus of claim 1 , wherein the determined directional distribution information indicates one or more directions associated with the indirect audio.

5. The apparatus of claim 1 , wherein the rendering information includes a target covariance matrix of the audio signal.

6. The apparatus of claim 1 , wherein the rendering information includes diffuse audio gains for channels of a multi-channel loudspeaker arrangement.

7. The apparatus of claim 1 , further comprising: a processor configured to generate a plurality of directional information for the direct audio;

8. The apparatus of claim 1 , wherein the associated spatial metadata includes information enabling mixing of audio signals to enable rendering of the spatial audio in a selected audio format.

9. The apparatus of claim 1 , wherein the associated spatial metadata includes information indicative of at least one of sound direction and sound directivity for one or more frequency subbands.

10. The apparatus of claim 1 , wherein the associated spatial metadata includes at least one of one or more prediction coefficients and one or more coherence parameters for one or more frequency subbands.

11. obtaining a spatial audio signal comprising one or more audio signals and associated spatial metadata, the associated spatial metadata being configured to enable rendering of spatial audio from the one or more audio signals, the spatial audio comprising direct audio and indirect audio; determining directional distribution information for the indirect audio using at least the associated spatial metadata; determining rendering information corresponding to the determined directional distribution information; and enabling rendering of the spatial audio using the estimated target spatial features, the one or more audio signals, and the associated spatial metadata.

12. The method of claim 11 , wherein the indirect audio comprises at least one of omnidirectional audio and diffuse audio.

13. The method of claim 11 , wherein the determined directional distribution information is indicative of the one or more directions associated with the indirect audio.

14. The method of claim 11 , wherein the rendering information includes a target covariance matrix of the audio signal.

15. The method of claim 11 , wherein the rendering information includes diffuse audio gains for channels of a multi-channel loudspeaker arrangement.

16. The method of claim 11 , wherein using at least the associated spatial metadata includes determining directional information for the direct audio.

17. The method of claim 11 , wherein the associated spatial metadata includes information enabling mixing of audio signals to enable rendering of the spatial audio in a selected audio format.

18. The associated spatial metadata includes: information indicative of at least one of a sound direction and a sound directivity for one or more frequency subbands; and For one or more frequency subbands, one or more prediction coefficients; The method of claim 11 , comprising at least one of:

19. The associated spatial metadata includes: one or more prediction coefficients for one or more frequency subbands; and one or more coherence parameters; The method of claim 11 , comprising at least one of:

20. obtaining a spatial audio signal comprising one or more audio signals and associated spatial metadata, the associated spatial metadata being configured to enable rendering of spatial audio from the one or more audio signals, the spatial audio comprising direct audio and indirect audio; determining directional distribution information for the indirect audio using at least the associated spatial metadata; determining rendering information corresponding to the determined directional distribution information; and enabling rendering of the spatial audio using the estimated target spatial features, the one or more audio signals, and the associated spatial metadata.

Images