Spatial Audio Representation and Rendering
The method dynamically applies room effects to spatial audio rendering based on metadata, addressing the unnatural reproduction of acoustic environments in existing technologies, ensuring accurate and optimized audio quality.
Patent Information
- Application Number
- JP2022521323
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2019-10-11
- Filing Date
- 2020-09-29
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2040-09-29
AI Technical Summary
Existing spatial audio rendering technologies fail to accurately reproduce the acoustic environment of the recording space, leading to unnatural listening experiences due to the lack of room effects in anechoic chambers and the addition of unwanted reverberation in certain audio types.
A method and apparatus for spatial audio rendering that dynamically applies room effects based on spatial metadata, using binaural audio signals with separate processing for early and late parts, and adjusts rendering based on indicators to maintain the natural acoustic environment.
Enhances the listening experience by accurately reproducing the intended acoustic environment, whether it includes or excludes room effects, optimizing audio quality for various input types.
Smart Images

Figure 0007590424000007 
Figure 0007590424000008 
Figure 0007590424000009
Abstract
Description
[Technical field]
[0001] This application relates to apparatus and methods for spatial audio representation and rendering, although not limited to audio representation for audio decoders. [Background technology]
[0002] Immersive audio codecs have been implemented that support a number of operating points ranging from low bitrate operation to transparency. One example of such a codec is the Immersive Voice and Audio Services (IVAS) codec, which is designed to be suitable for use over communication networks such as 3GPP® 4G / 5G networks, including for use in immersive services such as immersive voice and audio for virtual reality (VR). This voice codec is expected to handle the encoding, decoding, and rendering of speech, music, and general-purpose voice. In addition, it is expected to support channel-based audio and scene-based audio inputs that include spatial information about the sound field and sound sources. The codec is also expected to operate with low latency to enable conversational services under various transmission conditions and to support high error robustness.
[0003] The input signal may be presented to the IVAS encoder in one of several supported formats (and in several allowed combinations of formats). For example, a mono audio signal (without metadata) may be encoded using an Enhanced Voice Service (EVS) encoder. Other input formats may utilize new IVAS encoding tools. One input format proposed for IVAS is the Metadata-Assisted Spatial Audio (MASA) format, where the encoder may utilize, for example, a combination of mono and stereo encoding tools with metadata encoding tools for efficient transmission of the format. MASA is a parametric spatial audio format suitable for spatial audio processing. Parametric spatial audio processing is a branch of audio signal processing in which the spatial aspects of a sound (or sound scene) are described using a set of parameters. For example, in parametric spatial audio capture from a microphone array, it is a typical and valid choice to estimate a set of parameters such as the direction of the sound in a frequency band and the relative energy of the directional and non-directional parts of the captured sound in a frequency band from the microphone array signal, expressed for example as a direct-to-whole ratio or an ambient-to-whole energy ratio in a frequency band. These parameters are known to well describe the perceived spatial characteristics of the captured sound at the location of the microphone array, and can be used accordingly for spatial sound synthesis, either binaurally, through headphones, loudspeakers, or other formats such as Ambisonics.
[0004] For example, there may be two channels (stereo) of audio signals and spatial metadata. The spatial metadata may further define parameters such as a directional index describing the direction of arrival of the sound in the time-frequency parameter interval, a level / phase difference, a direct-to-total energy ratio representing the energy ratio of the directional index, coherence such as diffuseness, diffuse coherence representing the spread of energy representing the directional index, a diffuse-to-total energy ratio representing the energy ratio of an omnidirectional sound to the surrounding directions, a surround coherence representing the coherence of an omnidirectional sound to the surrounding directions, a reverberation-to-total energy ratio representing the energy ratio of a reverberant (such as microphone noise) sound whose energy ratios must sum to 1, a distance representing the distance of the sound originating from the direction of the index in meters on a logarithmic scale, a covariance matrix for multi-channel loudspeaker signals, or any data related to these covariance matrices, other parameters guiding a particular decoder, such as a central prediction coefficient or a 1:2 decoding coefficient (used, for example, in MPEG Surround). Any of these parameters may be determined in frequency bands.
[0005] Listening to a natural audio scene in an everyday environment is not just about sounds from a particular direction. Even without background ambience, it is typical that most of the sound energy reaching the ears is not from direct sounds, but from indirect sounds (i.e., reflections and reverberation) from the acoustic environment. Based on room effects, including discrete reflections and reverberation, the listener auditorily perceives sound source distance and room characteristics (small, big, wet, reverberant), among other features, and the room adds to the perceived sensation of the audio content. In other words, the acoustic environment is an intrinsic and perceptually relevant feature of spatial sound.
[0006] A listener listens to music in a normal room (as opposed to, for example, an anechoic chamber), and the music (e.g., stereo or 5.1 content) is typically produced in a way that it would be expected to be heard in a room with normal reverberation, which creates an envelope and spaciousness to the sound. Listening to normal music in an anechoic chamber is known to be unpleasant due to the lack of room effects. Thus, normal music will be heard with reverberation in a normal room (as it is essentially always heard). Summary of the Invention
[0007] According to a first aspect, an apparatus is provided comprising means configured to receive a spatial audio signal, the spatial audio signal including at least one audio signal and spatial metadata associated with the at least one audio signal; obtain a room effect control indicator; and determine based on the room effect control indicator whether a room effect should be applied to the at least one audio signal, wherein said means are configured to generate a first partial binaural audio signal based on the at least one audio signal and the spatial metadata when a room effect should be applied to the spatial audio signal, and to generate a second partial binaural audio signal based on the at least one audio signal, wherein the at least second partial binaural audio signal is generated at least in part with the room effect such that it has a response different from a response of the first partial binaural audio signal; and to combine the first partial binaural audio signal and the second partial binaural audio signal to generate a combined binaural audio signal.
[0008] The spatial metadata may comprise at least one directional parameter, and means configured to generate a first partial binaural audio signal based on the at least one audio signal, and the spatial metadata may be configured to generate the first partial binaural audio signal based on the at least one audio signal and the at least one directional parameter.
[0009] The spatial metadata may include at least one ratio parameter, and the means configured to generate the second partial binaural audio signal based on the at least one audio signal may be further configured to generate the second partial binaural audio signal based on the at least one audio signal and the at least one ratio parameter.
[0010] The at least one directional parameter may be a direction associated with a frequency band.
[0011] The means configured to generate a first partial binaural audio signal based on at least one audio signal and the spatial metadata may be configured to analyze the at least one audio signal to determine at least one stochastic characteristic associated with the at least one audio signal, and to generate the first partial binaural audio signal further based on the at least one stochastic characteristic associated with the at least one audio signal.
[0012] The at least one audio signal may include at least two audio signals, and the means configured to analyze the at least one audio signal to determine at least one stochastic characteristic associated with the at least one audio signal may be configured to estimate a covariance between the at least two audio signals, and the means configured to generate a first partial binaural audio signal may be configured to generate mixing coefficients based on the estimated covariance between the at least two audio signals further based on the at least one stochastic characteristic associated with the at least one audio signal, and to mix the at least two audio signals based on the mixing coefficients to generate the first partial binaural audio signal.
[0013] The means configured to generate the mixing coefficients based on an estimated covariance between the at least two carrier audio signals may be further configured to generate the mixing coefficients based on a target covariance.
[0014] The means may be further configured to generate an overall energy estimate based on the estimated covariance, determine head-related transfer function data based on the directional parameters, and determine a target covariance based on the head-related transfer function data, the spatial metadata, and the overall energy estimate.
[0015] The means configured to generate a second partial binaural audio signal based on the at least one audio signal may be configured to apply a reverberator to the at least one audio signal.
[0016] The means configured to obtain the room effect control index may be configured to perform at least one of receiving the room effect control index as a flag set by an encoder of the spatial audio signal, receiving the room effect control index as a user input, determining the room effect control index based on obtaining an indicator indicating a type of the spatial audio signal, and determining the room effect control index based on an analysis of the spatial audio signal to determine the type of the spatial audio signal.
[0017] The at least one audio signal may be at least one carrier audio signal generated by an encoder.
[0018] The second partial binaural signal may have a longer time response than the time response of the first partial binaural audio signal.
[0019] According to a second aspect, a method is provided, comprising the steps of receiving a spatial audio signal, the spatial audio signal including at least one audio signal and spatial metadata associated with the at least one audio signal, obtaining a room effect control indicator, and determining whether a room effect should be applied to the at least one audio signal based on the room effect control indicator, the method comprising the steps of generating a first partial binaural audio signal based on the at least one audio signal and the spatial metadata, generating a second partial binaural audio signal based on the at least one audio signal, the at least second partial binaural audio signal being generated at least in part with the room effect to have a response different from a response of the first partial binaural audio signal, and combining the first partial binaural audio signal and the second partial binaural audio signal to generate a combined binaural audio signal.
[0020] The spatial metadata may comprise at least one directional parameter, and generating the first partial binaural audio signal based on the at least one audio signal and the spatial metadata may include generating the first partial binaural audio signal based on the at least one audio signal and the at least one directional parameter.
[0021] The spatial metadata may include at least one ratio parameter, and generating the second portion of the binaural audio signal based on the at least one audio signal may further include generating the second portion of the binaural audio signal based on the at least one audio signal and the at least one ratio parameter.
[0022] The at least one directional parameter may be a direction associated with a frequency band.
[0023] Generating the first portion of the binaural audio signal based on the at least one audio signal and the spatial metadata may include analyzing the at least one audio signal to determine at least one stochastic characteristic associated with the at least one audio signal, and generating the first portion of the binaural audio signal further based on the at least one stochastic characteristic associated with the at least one audio signal.
[0024] The at least one audio signal may include at least two audio signals, and analyzing the at least one audio signal to determine at least one stochastic characteristic associated with the at least one audio signal may include estimating a covariance between the at least two audio signals, and generating the first portion of the binaural audio signal further based on the at least one stochastic characteristic associated with the at least one audio signal may include generating mixing coefficients based on the estimated covariance between the at least two audio signals, and mixing the at least two audio signals based on the mixing coefficients to generate the first portion of the binaural audio signal.
[0025] Generating the mixing coefficients based on an estimated covariance between the at least two carrier audio signals may further include generating the mixing coefficients based on a target covariance.
[0026] The method may further include generating an overall energy estimate based on the estimated covariance, determining head-related transfer function data based on the directional parameters, and determining a target covariance based on the head-related transfer function data, the spatial metadata, and the overall energy estimate.
[0027] Generating the second portion of the binaural audio signal based on the at least one audio signal may include applying reverb to the at least one audio signal.
[0028] The step of obtaining an in-room presentation control indication may include at least one of the steps of receiving the in-room presentation control indication as a flag set by an encoder of the spatial audio signal, receiving the in-room presentation control indication as a user input, determining the in-room presentation control indication based on obtaining an indicator indicating a type of the spatial audio signal, and determining the in-room presentation control indication based on an analysis of the spatial audio signal to determine the type of the spatial audio signal.
[0029] The at least one audio signal may be at least one carrier audio signal generated by an encoder.
[0030] The second partial binaural signal may have a longer time response than the time response of the first partial binaural audio signal.
[0031] According to a third aspect, an apparatus is provided comprising at least one processor and at least one memory comprising computer program code, the at least one processor comprising a processor for receiving a spatial audio signal, the at least one memory comprising computer program code, the at least one memory comprising computer program code, the at least one processor comprising a processor for receiving a spatial audio signal, the at least one memory comprising computer program code, the at least one processor comprising a processor for receiving a spatial audio signal, the at least one audio signal comprising spatial metadata associated with the spatial audio signal, the at least one audio signal comprising a processor for receiving a ... An apparatus is provided that is configured to determine whether a room effect should be applied to the spatial audio signal, where means are configured to generate a first portion binaural audio signal based on the at least one audio signal and the spatial metadata when a room effect should be applied to the spatial audio signal, and generate a second portion binaural audio signal based on the at least one audio signal, where the at least second portion binaural audio signal is generated, at least in part, with the room effect to have a response that differs from a response of the first portion binaural audio signal, and combine the first portion binaural audio signal and the second portion binaural audio signal to generate a combined binaural audio signal.
[0032] The spatial metadata may include at least one directional parameter, and the apparatus may cause a first partial binaural audio signal to be generated based on the at least one audio signal, and the spatial metadata may cause the first partial binaural audio signal to be generated based on the at least one audio signal and the at least one directional parameter.
[0033] The spatial metadata may include at least one ratio parameter, and the device for generating a second partial binaural audio signal based on the at least one audio signal may generate the second partial binaural audio signal based on the at least one audio signal and the at least one ratio parameter.
[0034] The at least one directional parameter may be a direction associated with a frequency band.
[0035] An apparatus configured to generate a first partial binaural audio signal based on at least one audio signal and spatial metadata may be configured to analyze the at least one audio signal to determine at least one stochastic characteristic associated with the at least one audio signal, and to generate the first partial binaural audio signal further based on the at least one stochastic characteristic associated with the at least one audio signal.
[0036] The at least one audio signal may include at least two audio signals, and the apparatus configured to analyze the at least one audio signal to determine at least one stochastic characteristic associated with the at least one audio signal may be configured to estimate a covariance between the at least two audio signals, and the apparatus configured to generate a first portion of a binaural audio signal further based on the at least one stochastic characteristic associated with the at least one audio signal may be configured to generate mixing coefficients based on the estimated covariance between the at least two audio signals and mix the at least two audio signals based on the mixing coefficients to generate the first portion of the binaural audio signal.
[0037] The device adapted to generate mixing coefficients based on an estimated covariance between at least two carrier audio signals may be further adapted to generate the mixing coefficients based on a target covariance.
[0038] The apparatus may be further configured to generate an overall energy estimate based on the estimated covariance, determine head-related transfer function data based on the directional parameters, and determine a target covariance based on the head-related transfer function data, the spatial metadata, and the overall energy estimate.
[0039] The device adapted to generate the second portion of the binaural audio signal based on the at least one audio signal may be adapted to apply a reverberator to the at least one audio signal.
[0040] An apparatus configured to obtain a room effect control index may be configured to perform at least one of receiving the room effect control index as a flag set by an encoder of the spatial audio signal, receiving the room effect control index as a user input, determining the room effect control index based on obtaining an indicator indicating a type of the spatial audio signal, and determining the room effect control index based on an analysis of the spatial audio signal to determine the type of the spatial audio signal.
[0041] The at least one audio signal may be at least one carrier audio signal generated by an encoder.
[0042] According to a fourth aspect, there is provided an apparatus comprising: a receiving circuit configured to receive a spatial audio signal, the spatial audio signal including at least one audio signal and spatial metadata associated with the at least one audio signal; a circuit configured to obtain a room effect control indication; and a determining circuit configured to determine whether a room effect should be added to the spatial audio signal based on the room effect control indication. Here, the device comprises: a determination circuit configured to determine whether a room effect should be added to the spatial audio signal, the circuit being configured to generate a first partial binaural audio signal based on the at least one audio signal and spatial metadata; a circuit configured to generate a second partial binaural audio signal based on the at least one audio signal when the added room effect should be added to the spatial audio signal, the second partial binaural audio signal being generated at least partially with the room effect such that it has a response different from a response of the first partial binaural audio signal; and a combining circuit configured to combine the first partial binaural audio signal and the second partial binaural audio signal to generate a combined binaural audio signal.
[0043] According to a fifth aspect, there is provided a computer program comprising instructions [or a computer readable medium comprising program instructions] for causing an apparatus to perform a method comprising: receiving a spatial audio signal, the spatial audio signal comprising at least one audio signal and spatial metadata associated with the at least one audio signal; obtaining a room effect control index; and determining whether room effects should be applied to the spatial audio signal based on the room effect control index. The method comprises generating a first partial binaural audio signal based on the at least one audio signal and the spatial metadata, generating a second partial binaural audio signal based on the at least one audio signal, the at least second partial binaural audio signal being generated at least in part with the room effects having a response different from that of the first partial binaural audio signal, and combining the first partial binaural audio signal and the second partial binaural audio signal to generate a combined binaural audio signal.
[0044] According to a sixth aspect, there is provided a non-transitory computer-readable medium comprising program instructions to cause an apparatus to perform the steps of receiving a spatial audio signal, the spatial audio signal comprising at least one audio signal and spatial metadata associated with the at least one audio signal; obtaining a room effect control indicator; determining whether a room effect should be applied to the spatial audio signal based on the room effect control indicator; generating a first partial binaural audio signal based on the at least one audio signal and the spatial metadata when the room effect is to be applied to the spatial audio signal; generating a second partial binaural audio signal based on the at least one audio signal when the room effect is to be applied to the spatial audio signal; generating at least the second partial binaural audio signal at least partially using the room effect to have a response different from a response of the first partial binaural audio signal; and combining the first partial binaural audio signal and the second partial binaural audio signal to generate a combined binaural audio signal when the room effect is to be applied to the spatial audio signal.
[0045] According to a seventh aspect, there is provided an apparatus comprising: means for receiving a spatial audio signal, the spatial audio signal comprising at least one audio signal and spatial metadata associated with the at least one audio signal; means for obtaining a room effect control indicator; means for determining whether a room effect control indicator should be applied to the spatial audio signal based on the room effect control indicator; means for generating a first partial binaural audio signal based on the at least one audio signal and the spatial metadata when a room effect is to be applied to the spatial audio signal; means for generating a second partial binaural audio signal based on the at least one audio signal when a room effect is to be applied to the spatial audio signal, the second partial binaural audio signal being generated at least in part with the room effect such that when the room effect is to be applied to the spatial audio signal, the at least second partial binaural audio signal has a response that differs from a response of the first partial binaural audio signal; and means for combining the first partial binaural audio signal and the second partial binaural audio signal when applying a room effect to the spatial audio signal to generate a combined binaural audio signal.
[0046] According to an eighth aspect, there is provided a computer-readable medium comprising program instructions to cause an apparatus to at least: receive a spatial audio signal, the spatial audio signal comprising at least one audio signal and spatial metadata associated with the at least one audio signal; obtain a room effect control indicator; determine based on the room effect control indicator whether the room effect control indicator should be applied to the spatial audio signal; generate a first partial binaural audio signal based on the at least one audio signal and the spatial metadata when the room effect is to be applied to the spatial audio signal; generate a second partial binaural audio signal based on the at least one audio signal when the room effect is to be applied to the spatial audio signal; the at least second partial binaural audio signal is generated at least in part with the room effect to have a response that differs from a response of the first partial binaural audio signal; and combine the first partial binaural audio signal and the second partial binaural audio signal to generate a combined binaural audio signal when the room effect is to be applied to the spatial audio signal.
[0047] An apparatus comprising means for performing the operations set forth above.
[0048] An apparatus configured to perform the operations of the methods described above.
[0049] A computer program comprising program instructions for causing a computer to carry out the method described above.
[0050] A computer program product stored on the medium can cause an apparatus to perform the methods described herein.
[0051] The electronic device may comprise an apparatus as described herein.
[0052] The chipset may comprise an apparatus as described herein.
[0053] SUMMARY OF THE PRESENT APPLICATION The embodiments of the present application aim to address problems associated with the state of the art. [Brief description of the drawings]
[0054] For a better understanding of the present application, reference is made, by way of example, to the accompanying drawings, in which: [Figure 1] FIG. 1 illustrates a schematic of a system of apparatus suitable for implementing some embodiments. [Diagram 2] FIG. 2 illustrates a flow diagram of an exemplary apparatus operation according to some embodiments. [Diagram 3] FIG. 3 illustrates a schematic of a synthesis processor as shown in FIG. 1, according to some embodiments. [Figure 4] FIG. 4 illustrates a flow diagram of the operation of an exemplary apparatus such as that shown in FIG. 3, according to some embodiments. [Diagram 5] FIG. 5 shows an exemplary apparatus suitable for implementing the apparatus shown in the previous figures. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0055] In the following, suitable apparatus and possible mechanisms for the control and addition of room effects to a rendered spatial metadata-aided audio signal are described in further detail.
[0056] It should be noted that although the following examples focus on MASA encoding and decoding, the presented method is applicable to any system that utilizes a carrier audio signal and spatial metadata. The spatial metadata can include some of the following parameters, e.g., in any type of combination of direction, level / phase difference, direct-to-total energy ratio, diffuseness, coherence (such as diffuse and / or ambient coherence), and distance. Typically, the parameters are given in the time-frequency domain. Therefore, in the following, when the terms IVAS and / or MASA are used, it should be understood that they can be replaced with any other suitable codec and / or metadata format and / or system.
[0057] In the example below, the IVAS stream can be decoded and rendered into a variety of output formats, including binaural, multi-channel, and ambisonic (FOA / HOA) output. Additionally, there may be an interface for external rendering, where the output format can correspond to the input format, etc.
[0058] Since the spatial (e.g. MASA) metadata describes the desired spatial audio perception in an output format independent manner, any stream with spatial metadata can be flexibly depicted in any of the output formats mentioned above. However, since the MASA stream can originate from various inputs, the characteristics of the carrying audio signal received by the decoder may be different. The decoder is therefore configured to take these aspects into account so that it can produce optimal audio quality.
[0059] A method for rendering parametric audio signals includes in MPEG Surround, where 5.1 sounds are conveyed in the form of a stereo downmix and spatial metadata that includes information for recomposing the 5.1 sound. Essentially, the spatial metadata consists of coefficients that generate a decoding matrix for steering and decorrelation of the stereo sound to the 5.1 sound. In an MPEG Surround binaural decoder, these parameters are utilized to select and mix the appropriate HRTFs to generate an efficient stereo inaural (2x2) mixing procedure without the need to generate intermediate 5.1 loudspeaker sounds.
[0060] Furthermore, the rendering of parametric audio signals can be performed using directional audio coding (DirAC), which in its first formulation estimates spatial metadata based on B-format microphone signals (consisting of four different beam patterns). In the rendering stage, the microphone signals are split into direct and ambient signals in frequency bands as a function of a diffuseness parameter (ambience / total energy ratio parameter). In the context of binaural reproduction, in one configuration, the reproduction is performed such that the direct part is amplitude panned to a virtual surround loudspeaker setup, the ambience is decorrelated to all or a subset of the virtual loudspeaker setup, and then the virtual 3D loudspeaker signals are processed with head-related transfer functions (HRTFs) to generate the binaural output.
[0061] Furthermore, rendering parametric audio signals can include a parametric spatial audio synthesis framework that avoids any intermediate signal generation (e.g., direct and ambient parts) and allows a least-squares optimized mixing solution to generate the spatial sound of interest directly from the available audio signals. This approach efficiently utilizes independent signals in the audio signal and reduces the need to use correlation reduction procedures that are known to degrade the perceived quality of the reproduced sound. This method is not a specific parametric renderer, but a method that can be applied by any parametric renderer, and can be utilized, for example, in multichannel rendering and binaural rendering, for example, in conjunction with DirAC. For example, this approach is described in more detail in "An Optimized Covariance Domain Framework for Time-Frequency Processing of Spatial Audio," J Vilkamo, TB Henkastrom, A Kuntz, Journal of Audio Engineering Society 61, no. 6 (2013): 403-411.
[0062] The concept discussed in the following embodiments relates to the addition of room effects to rendered spatial audio signals. Listening to a natural audio scene in an everyday environment is not just about sounds from a particular direction. Even without background ambience, most of the sound energy reaching the ears is not direct sound, but typically indirect sound (i.e., reflections and reverberation) from the acoustic environment. Based on room effects, including discrete reflections and reverberation, we auditorily perceive sound source distance and room characteristics (small, big, wet, reverberant), among other features, and the room adds to the perceived sensation of the audio content. In other words, the acoustic environment is a perceptually relevant feature of spatial sound.
[0063] Since listeners typically listen to music in a normal room (as opposed to, for example, an anechoic chamber), the music (e.g., stereo or 5.1 channel content) is produced in a way that is expected to be heard in a room with normal reverberation, which creates an envelope and spaciousness to the sound. Listening to normal music in an anechoic chamber is known to be unpleasant due to the lack of room effects. Therefore, normal music should (and is always) listened to in a normal room with reverberation.
[0064] For example, binaural spatial sound rendering of multichannel content (e.g., 5.1) using head-related transfer function (HRTF)-based rendering corresponds to listening in an anechoic chamber. Thus, the lack of room effect is perceived as unnatural and unpleasant. Binaural room impulse response (BRIR)-based techniques for adding room effect are typically used for binaural rendering of multichannel content (e.g., 5.1).
[0065] However, there are some signal types for which adding room effects is undesirable, for example binaural rendering of spatial sound captured on a mobile device. The goal of mobile device captured audio is typically to "transport" the listener to the location where the spatial sound was captured, faithfully rendering the sound scene as if the listener were present. The captured audio contains the natural reverberation of the recording space, and therefore HRTF-based binaural rendering methods that do not add additional reverberation are preferred. Because the rendered audio contains both the room effects of the capture space and the room effects of the rendering, adding room effects to the rendering would cause an unnatural listening experience.
[0066] Thus, there are situations in which it is necessary to add room effects in binaural rendering, and there are situations in which it is not necessary to add room effects.
[0067] A concept discussed in further detail below is the provision of apparatus and methods, for example in some embodiments a binaural renderer and / or method for binaural rendering is provided that operates on a spatial audio stream, which may include a carrier audio signal (from various sources, with any characteristics) and spatial metadata, typically including at least the direction of frequency bands (the direction may have any value). Further, in some embodiments the binaural renderer and / or method for binaural rendering is configured to render the binaural signal with and without added room effects (based on an indication of whether to render it).
[0068] Thus, the embodiments relate to binaural rendering of a spatial audio stream comprising a carrier audio signal(s) and spatial metadata (consisting at least of directions in frequency bands). In such an embodiment, a method is proposed that can render a binaural audio output with and without room effects based on the spatial audio stream (which can be from various sources such as mobile and 5.1). Furthermore, in some embodiments, the renderer is configured to perform the rendering by rendering the binaural signals based on the spatial metadata, the binaural rendering data for early part rendering and a probabilistic analysis of the carrier audio signal, and by rendering the binaural signals for added room effects (combined with the "early part" signal) "additional room effects" if the indication indicates these signals based on the binaural rendering data for added room effects.
[0069] In some embodiments, there may be a renderer and / or decoder that obtains a parametric audio stream, consisting of one or more carrier audio signals and spatial metadata. The spatial audio stream may be obtained, for example, by retrieving it from a storage device or by receiving it over a network.
[0070] The spatial metadata can include at least the directions of the frequency bands. These directions can point in any direction (instead of a predefined direction like a loudspeaker setup direction). Therefore, the rendering method needs to be set up to support rendering in any direction. Other parameters can include a ratio parameter that indicates how directional or ambient the sound is in the frequency band. Further parameters can include whether directional sounds should be played point-like or wide, or any other parameter.
[0071] In some embodiments, the carrier audio signal may be, for example, one of spatial microphone signals (with any potential pre-processing performed), co-incident microphone signals, a downmix of surround loudspeaker signals, a downmix of audio objects, an Ambisonic signal in any order, or a subset of an Ambisonic signal in any order, any mixture of the above, or any other type.
[0072] The renderer is further configured in some embodiments to receive an indication as to whether to render the added room response. The indication can be obtained in various ways. For example, it may be obtained from a user or received together with the spatial audio stream. It may also be determined based on the spatial audio stream. For example, if a 5.1 downmix is detected as the transport signal, the indication may be set to "add room effect". On the other hand, if a microphone signal is detected as the transport signal, the indication may be set to "no room effect".
[0073] In some embodiments, the "early part" and "room effect" binaural signals are rendered separately. The early part binaural signals are rendered in frequency bands, and therefore the carrier signal in some embodiments may be transformed into the time-frequency domain.
[0074] In some embodiments, the initial renderer is configured to perform the rendering by estimating the carrier signal probability characteristics (covariance matrix) in the frequency band. The covariance matrix contains the energy, correlation and cross-phase information of the transport channels. This information is then used to configure the rendering to adapt to the different signal characteristics due to the many carrier audio signal types. For example, "spaced", "matched" and "downmix" types can have very different probabilistic characteristics even in situations where the spatial metadata is similar.
[0075] Furthermore, in some embodiments, a target covariance matrix is determined in frequency bands using spatial metadata. For example, if a sound comes from a certain angle, it needs to have a certain spectrum in the left and right ear (each frequency) and have a certain phase dependency. These are determined using binaural rendering data for early part rendering, for example, using HRTF pairs at that particular angle. Similarly, ratio parameters affect how the binaural output channels should be correlated, and so on. Thus, a target covariance matrix is constructed that reflects all of these binaural characteristics that correspond to the received spatial metadata.
[0076] Then, if the carrier signal covariance matrix and the target covariance matrix are known, a mixing solution can be formulated, which (frequency band wise) is such that when applied to the transport audio signal, it produces an output signal with a covariance matrix according to the determined target covariance matrix. The resulting signal is converted back to the time domain and the result is the rendered early part binaural signal.
[0077] In this way, due to the procedure of measuring the signal probability characteristics and configuring the processing accordingly, the processing can be optimized for various carrier signal types. Such processing does not add room effects.
[0078] In some embodiments, when the indicator is set to render a room effect, the room effect is rendered. The rendering of the room effect can be performed by convolving the carrier audio signal with the binaural rendering data for additional room rendering, which can include, for example, the late part of the measured BRIR. For example, it is possible to attenuate / remove the early / directional part from the paired BRIR and use only the late (binaural) response as the convolution reverberator. The convolution can be efficiently implemented using FFT-based convolution techniques.
[0079] Furthermore, in some embodiments, the early portion binaural signal and the added room effect binaural signal are combined (e.g., by summing them) to result in a binaural signal in a suitable output format that can be played through headphones.
[0080] In some embodiments, an additional "alignment" delay may be introduced into one of the signal paths where one of the other paths is known to have a longer delay before combining.
[0081] Thus, as a result of rendering parametric spatial audio signals from various sources into a binaural output, embodiments as described in further detail herein can generate a binaural signal with (or without) room effects (based on appropriate metrics), and further, the binaural signal output can be optimized for the changing, non-predefined characteristics of the carrying audio signal. Embodiments can be configured to render audio signals in any direction.
[0082] Referring to FIG. 1, an exemplary apparatus and system for performing audio capture and rendering is shown, according to some embodiments.
[0083] System 199 is shown with an encoder / analyzer 101 portion and a decoder / synthesizer 105 portion.
[0084] The encoder / analyzer 101 portion in some embodiments includes an audio signal input configured to receive an input audio signal 110. The input audio signal may come from any suitable source, such as, for example, two or more microphones on a mobile phone, e.g., B-format microphones or other microphone arrays such as Eigenmike, Ambisonic signals, e.g., First Order Ambisonic (FOA), Higher Order Ambisonic (HOA), loudspeaker surround mix and / or object. The input audio signal 110 may be provided to an analysis processor 111 and a transport signal generator 113.
[0085] The encoder / analyzer 101 portion may include an analysis processor 111. The analysis processor 111 is configured to perform spatial analysis on the input audio signal that generates appropriate metadata 112. Thus, the purpose of the analysis processor 111 is to estimate spatial metadata in frequency bands. For all of the aforementioned input types, there are known methods for generating appropriate spatial metadata, such as direction and direct-to-total energy ratios in frequency bands (or similar parameters such as diffuseness, i.e., ambient-to-total ratios). These methods are detailed herein, but some examples may include performing an appropriate time-frequency transformation on the input signal, then estimating delay values of microphone pairs that maximize inter-microphone correlation in the frequency band when the input is a mobile phone microphone array, formulating a direction value corresponding to that delay (as described in GB Patent Application No. 1619573.7 and PCT Patent Application No. PCT / FI2017 / 050778), and formulating a ratio parameter based on the correlation value.
[0086] The metadata can take various forms and can include spatial and other metadata. A typical parameterization of spatial metadata is one directional parameter in each frequency band θ(k,n) and an associated direct-to-total energy ratio in each frequency band r(k,n), where k is the frequency band index and n is the time frame index. Determining or estimating the direction and ratio depends on the device or implementation in which the audio signal is obtained. For example, the metadata can be obtained or estimated using spatial audio capture (SPAC), using the methods described in GB Patent Application No. 1619573.7 and PCT Patent Application No. PCT / FI2017 / 050778. In other words, in this particular situation, the spatial audio parameters include parameters that aim to characterize the sound field. In some embodiments, the parameters generated may differ for each frequency band. Thus, for example, in band X, all parameters are generated and transmitted, but in band Y, only one of the parameters is generated and transmitted, and further, in band Z, no parameters are generated or transmitted. A practical example of this might be that for some frequency bands, such as the highest band, some of the parameters are not needed for perceptual reasons.
[0087] If the input is an FOA signal or a B-format microphone, the analysis processor 111 can be configured to determine parameters such as intensity vectors from which directional parameters are derived and to compare the intensity vector lengths with an overall sound field energy estimate to determine a ratio parameter. This method is known in the literature as Directional Audio Coding (DirAC).
[0088] If the input is an HOA signal, the analysis processor 111 can either take a FOA subset of the signal and use the above method, or split the HOA signal into multiple sectors, in each of which the above method is utilized. This sector-based method is known in the literature as High Order DirAC (HO-DirAC). In this case, there are two or more simultaneous directional parameters per frequency band.
[0089] If the input is a loudspeaker surround mix and / or objects, the analysis processor 111 may be configured to convert the signal into a FOA signal (through the use of spherical harmonic encoding gain) and analyze the directional and ratio parameters as described above.
[0090] The output of the analysis processor 111 is therefore spatial metadata determined in frequency bands. The spatial metadata can include direction and proportion in frequency bands, but can also have any of the metadata types listed above. The spatial metadata can vary in time and frequency.
[0091] In some embodiments, the spatial analysis may be performed outside of the system 199. For example, in some embodiments, spatial metadata associated with the audio signal may be provided to the encoder as a separate bitstream. In some embodiments, the spatial metadata may be provided as a set of spatial (directional) index values.
[0092] The encoder / analyzer 101 portion may comprise a carrier signal generator 113. The carrier signal generator 113 is configured to receive an input signal and generate a suitable carrier audio signal 114. The carrier audio signal may be a stereo or mono audio signal. The generation of the carrier audio signal 114 may be performed using known methods as summarized below.
[0093] If the input is a mobile phone microphone array audio signal, the carrier signal generator 113 may be configured to select a left and right microphone pair and apply appropriate processing to the signal pair, such as automatic gain control, microphone noise cancellation, wind noise cancellation, and equalization.
[0094] If the input is a FOA / HOA signal or a B-format microphone, the transport signal generator 113 may be configured to formulate a left-right oriented directional beam signal, such as two opposing cardioid signals.
[0095] If the input is a surround mix of loudspeakers and / or objects, the carrier signal generator 113 can be configured to combine the left side channels into a left downmix channel and generate the same downmix signal for the right side, and add a center channel to both carrier channels with an appropriate gain.
[0096] In some embodiments, the transport signal generator 113 is configured to bypass the input. For example, there are situations in which analysis and synthesis are performed in the same device in a single processing step, without intermediate encoding. The number of transport channels can also be any suitable number (preferably one or two channels, as discussed in the examples).
[0097] In some embodiments, the encoder / analyzer unit 101 may comprise an encoder / multiplexer 115. The encoder / multiplexer 115 may be configured to receive the carrier audio signal 114 and the metadata 112. The encoder / multiplexer 115 may be further configured to generate the metadata information and the carrier audio signal in an encoded or compressed form. In some embodiments, the encoder / multiplexer 115 may further interleave, multiplex, or embed the metadata within the encoded audio signal into a single data stream 116 prior to transmission or storage. The multiplexing may be performed using any suitable scheme.
[0098] The encoder / multiplexer 115 may be implemented as, for example, an IVAS encoder, or any other suitable encoder, and is thus configured to encode the audio signal and the metadata to form a bitstream 116 (e.g., an IVAS bitstream).
[0099] This bitstream 116 may then be transmitted / stored 103, as indicated by the dashed line. In some embodiments, the encoder / multiplexer 115 is not present (and therefore the decoder / demultiplexer 121 is not present, as described below).
[0100] System 199 may further include a decoder / synthesizer unit 105 configured to receive, extract, or otherwise obtain bitstream 116 and generate from the bitstream an appropriate audio signal that is presented to a listener / listener playback device.
[0101] The decoder / synthesizer unit 105 may include a decoder / demultiplexer 121 configured to receive the bitstream, demultiplex the encoded stream, and then decode the audio signal to obtain a transport signal 124 and metadata 122.
[0102] Further, in some embodiments, as described above, the demultiplexer / decoder 121 may not be present (e.g., there is no associated encoder / multiplexer 115 because both the encoder / analyzer section 101 and the decoder / synthesizer 105 are located within the same device).
[0103] The decoder / synthesizer unit 105 may comprise a synthesis processor 123 configured to obtain the carrier audio signal 124, the spatial metadata 122 and any added room effect control signals or indicators, and generate a binaural output signal 128 that can be played via headphones.
[0104] The operation of the system may be summarized with respect to a flow diagram as shown in FIG. 2, which illustrates an example of receiving an input audio signal as shown in step 201.
[0105] Next, the flow diagram shows the analysis (spatial) of the input audio signal to generate spatial metadata as shown in FIG.
[0106] A carrier audio signal is then generated from the input audio signal, as shown in FIG. 2, via step 204 .
[0107] The generated carrier audio signal and metadata may then be multiplexed as shown in Figure 2 by step 205. This is shown in Figure 2 as an optional dashed box.
[0108] The encoded signal may be further demultiplexed and decoded to generate the carrier audio signal and spatial metadata, as shown in Figure 2 by step 207. This is also shown as an optional dashed box.
[0109] Next, as shown in FIG. 2 by step 209, a binaural audio signal may be synthesized based on the carrier audio signal, the spatial metadata, and the added room effect control signals or indicators.
[0110] The synthesized binaural audio signal can then be output to an appropriate output device, such as a set of headphones, as shown in FIG. 2 by step 211 .
[0111] Referring now to FIG. 3, the compositing processor 123 is shown in further detail.
[0112] In some embodiments, the synthesis processor 123 comprises a time-frequency transformer 301. The time-frequency transformer 301 is configured to receive a carrier audio signal 122 that is transformed into the time-frequency domain (time domain). Suitable transforms include, for example, a short-time Fourier transform (STFT) and a complex-modulated quadrature mirror filter bank (QMF). The resulting signal is i (b,n), where i is the channel index, b is the frequency bin index of the time-frequency transform, and n is the time index. The time-frequency signal is represented here in vector form (e.g., in the case of two channels).
number
[0113] The following processing operations can then be performed in the time-frequency domain and across the frequency bands. A frequency band can be one or more frequency bins (individual frequency components) of an applied time-frequency transformer (filter bank). In some embodiments, the frequency bands can approximate a perceptually relevant resolution such as Bark frequency bands, which are spectrally more selective at low frequencies than at high frequencies. Alternatively, in some implementations the frequency bands can correspond to frequency bins. The frequency bands are typically frequency bands (or approximate frequency bands) for which spatial metadata has been determined by the analysis processor. Each frequency band k is a frequency band that is divided into a lowest frequency bin b low (k) and the highest frequency bin b high (k) can be defined.
[0114] The time-frequency carrier signal 302 in some embodiments may be provided to a covariance matrix estimator 307 and a mixer 311 .
[0115] The synthesis processor 123 comprises, in some embodiments, a covariance matrix estimator 307 configured to receive the time-frequency domain carrier signals 302 and estimate the covariance matrix of the time-frequency carrier signals and their overall energy estimate (within the frequency band). The covariance matrix may, for example, in some embodiments, be
number
[0116] The estimated covariance matrix 310 may be output to the mixing rule determiner 309 .
[0117] The covariance matrix estimator 307 also calculates the overall energy estimate E(k,n) 308, i.e., C xThe (k,n) diagonal sum may be generated and this overall energy estimate may be provided to the target covariance matrix determiner 305.
[0118] In some embodiments, the synthesis processor 123 comprises an HRTF determiner 303. The HRTF determiner 303 may include a set of suitably dense HRTFs or an HRTF interpolator. The HRTF determiner is configured to determine a 2x1 complex-valued head-related transfer function (HRTF) h(θ(k,n),k) for the angle θ(k,n) and frequency band k. In some embodiments, the HRTF determiner 303 is configured to receive the spatial metadata 124 and determine an output HRTF from the angle (which is a direction parameter in the spatial metadata).
[0119] For example, the HRTF at the mid-frequency of band k can be determined. If tracking of the listener's head orientation is included, the directional parameters θ(k,n) can be modified before obtaining the HRTF to take into account the current head orientation.
[0120] The HRTF data set of the HRTF determiner 303 may in some embodiments be pre-formulated and fixed for the synthesis processor 123 and may be selected from multiple HRTF data sets. In some embodiments, the HRTF data set of the HRTF determiner 303 also comprises a diffuse field covariance matrix for each band k, e.g., d=1..D, in the direction θ d The covariance matrix of the diffuse field can be formulated as
number
[0121] The HRTF data can be rendered and interpolated by using any suitable method. For example, in some embodiments, a set of HRTFs is decomposed into left and right interaural time differences and energies as a function of frequency. Then, when an HRTF at a given angle is required, the closest existing data point in the HRTF set is found and the delay and energy at the given angle are interpolated. These energies and delays can then be transformed as complex multipliers are used.
[0122] In some embodiments, the HRTFs are interpolated to convert the HRTF dataset into a set of spherical harmonic binaural decoding matrices in the frequency band. The HRTF for any angle can then be determined by formulating a spherical harmonic weighting vector for that angle and multiplying it with the matrix. The result is again a 2×1 HRTF vector.
[0123] In some embodiments, interpolation of HRTFs may be performed by treating them as virtual loudspeakers, for example, to obtain the interpolated HRTFs via amplitude panning.
[0124] By definition, HRTF refers to the response from a particular direction to the ears in an anechoic space. However, instead of an HRTF dataset, it is entirely possible to use another dataset that also contains (in addition to the HRTF portion) the early part of the binaural room impulse response. Such a dataset would also contain, for example, the spectrum and other features due to reflections from the ground floor or walls.
[0125] HRTF data 304 (which is h(θ(k,n),k) and C D (k) may be output by the HRTF determiner 303 and passed to the target covariance matrix determiner 305.
[0126] In some embodiments, the synthesis processor 123 comprises a target covariance matrix determiner 305. The target covariance matrix determiner 305 is configured to receive spatial metadata 124, which in this example may comprise at least one directional parameter θ(k,n), at least one direct-to-total energy ratio parameter r(k,n), HRTF data 304, and a total energy estimate E(k,n) 308. The covariance matrix determiner 305 is then configured to determine a target covariance matrix 306 based on the spatial metadata 124, the HRTF data 304, and the total energy estimate 308. For example, the target covariance matrix determiner 305 may
number
[0127] Next, the target covariance matrix C y The (k,n) 306 may be provided to a mixing rule determiner 309 .
[0128] The synthesis processor 123, in some embodiments, comprises a mixing rule determiner 309. The mixing rule determiner 309 is configured to receive the target covariance matrix 306 and the estimated covariance matrix 310. The mixing rule determiner 309 determines the target covariance matrix C y (k,n) 306 and the measured covariance matrix C x (k,n) 310.
[0129] In some embodiments, the mixing matrix is generated based on the method described in "Optimized covariance domain framework for time-frequency processing of spatial audio", J Vilkamo, T Backstrom, A Kuntz, Journal of Audio Engineering Society 61, no. 6 (2013): 403-411.
[0130] In some embodiments, the mixing rule determiner 309 determines the prototype matrices that guide the generation of the mixing matrix.
number
[0131] In summary, the covariance matrix C x When applied to a signal with (k,n), the covariance matrix C y A least-squares optimized method can be used to provide a mixing matrix M(k,n) that produces signals with (k,n). The matrix Q guides the signal content in such mixing. In this example, the matrix is simply the identity matrix, since the left and right processed signals should be as similar as possible to the original left and right signals. In other words, the design requires a C y The goal is to minimally modify the signal while obtaining (k,n). A mixing matrix M(k,n) is formulated for each frequency band k and provided to the mixer 311.
[0132] In this example, the mixing matrix is defined based on the input being a two-channel carrier audio signal. However, these methods can be adapted to embodiments for any number of carrier audio channels. The synthesis processor 123 comprises in some embodiments a mixer 311. The mixer 311 receives the time-frequency audio signal 302 and a mixing matrix 312. The mixer 311 is configured to process the time-frequency audio signal (input signal) in each frequency bin b to generate two processed (first or initial part) time-frequency signals 314. This can be done, for example, by:
number
[0133] The above procedure assumes that the input signals x(b,n) have a suitable incoherence between them to render the output signal y(b,n) with the desired target covariance matrix characteristics. In some situations, the input signals do not have a suitable inter-channel incoherence, for example, when only a single channel carrier signal is present, or when the signals are otherwise highly correlated. Therefore, in some embodiments, a decorrelation operation is implemented to generate a decorrelated signal based on the decorrelated signal x(b,n) and mix the decorrelated signal into a certain residual signal that is added to the signal y(b,n) in the above equation. Procedures for obtaining such residual signals are known and are described, for example, in the above standards.
[0134] The processed binaural (early part) time-frequency signal y(b,n) 314 is fed to an inverse T / F transformer 313 .
[0135] In some embodiments, the synthesis processor 123 includes an inverse T / F transformer 313 configured to receive the binaural (early part) time-frequency signal y(b,n) 314 and apply an inverse time-frequency transformation corresponding to the applied time-frequency transformation applied by the T / F transformer 301. The output of the inverse T / F transformer 313 is a binaural (early part) signal 316 corresponding to the early / dry part of the binaural processing (i.e., not including late reverberation). Thus, the above procedure describes the (first) early / dry part of the binaural processing, and the following process describes the (second) late / wet part of the binaural processing. In one embodiment, the synthesis processor 123 comprises a reverberator 351 configured to receive the carrier audio signal 122 and apply a time-domain reverberation operation to the carrier audio signal 122 to generate a late reverberant binaural room effect signal 318 based on the added room effect control (indicator) 126. However, the reverberator in some embodiments includes a time-frequency domain reverberator, which, when implemented, is configured to receive a time-frequency carrier signal (e.g., as generated by the T / F transformer 301), the output of which is mixed or combined with the output of the mixer 311 (or combined with the binaural T / F early part signal 314 in the mixer) before the inverse T / F transformer 313.
[0136] The reverberator 351 is also configured to receive the added room effect control signal or information 126, including an indication of whether room effects (i.e., binaural reverberation) should be output. If room effects should not be output, the reverberator 351 is configured to provide no output. If room effects should be output, the reverberator may be configured to add room effects, as described in more detail below. The determination or acquisition of the added room effect control 126 may be based on any suitable method. For example, in some embodiments, the added room effect control 126 may be acquired from a user. In some further embodiments, the added room effect control 126 may be received together with the spatial audio stream (e.g., a flag set by the encoder in the spatial metadata). The added room effect control 126 may be determined based on the spatial audio stream. For example, the added room effect control 126 may be determined based on the type of spatial audio signal (e.g., the bitstream includes an indication that the spatial audio signal originates from a 5.1 surround mix, and then the decoder knows to render the room effects). In some embodiments, the added room effect control may be determined based on an analysis of the spatial audio signal. For example, the spatial audio signal and metadata may be monitored to determine whether the spatial audio signal originates from a 5.1 channel signal or some other type for which additional room effects are desired, as opposed to from a spatial audio capture system, such as a cell phone capturing the spatial audio, where additional room effects are not desired because, for example, the ambience and / or reverberation required for the spatial audio signal is already present.
[0137] For example, if a 5.1 sound downmix is detected as the transport signal, the indication may be set to "add room effects", whereas if a microphone signal is detected as the transport signal, the indication may be set to "no room effects". In some embodiments, the added room effects control may also have other information that controls the reverberation, such as the overall level as a function of reverberation time and frequency.
[0138] The reverberator 351 may implement any suitable reverberation method to generate the reverberation. For example, in some embodiments, the reverberator 351 is configured to perform a convolution with a predefined reverberation response. The convolution may be efficiently applied using Fast Fourier Transform (FFT) convolution or partial FFT convolution, as described, for example, in Gardner, William G. "Efficient convolution without input / output delay" In Audio Engineering Society Convention 97. Audio Engineering Society, 1994.
[0139] The reverberation response may be obtained from the binaural room impulse response (BRIR) by appropriate windowing, for example where the first or early part of the BRIR (corresponding to the HRTF / dry rendering) is completely attenuated, leaving only the second or late part. Such a response can be applied in an efficient convolution operation to generate the binaural room effect signal.
[0140] In some embodiments, the carrier audio signal is summed into a single channel that is processed with a pair of reverberation responses. As in a typical setup of a BRIR, there will be responses from several directions, and the reverberation response could be windowed from one of the responses in the setup, such as a center-front BRIR. The reverberation response could also be a combined (e.g. averaged) response based on the BRIR from multiple directions.
[0141] In some embodiments, the transport audio channels are processed with different pairs of reverberation responses, and the results are summed together to obtain a two-channel output. In this case, the reverberation response for the left transport signal can be windowed, for example, from a 90 degree BRIR on the left and correspondingly on the right. In these embodiments, the reverberation responses can also be combined (e.g., averaged) based on the BRIR from multiple directions.
[0142] In some embodiments, the reverberator includes a feedback delay network (FDN), which is a time-domain reverberator or a sparse frequency-domain reverberator, as described in Vilkamo, J., Neugebauer, Band Plogsties, J., "Sparse frequency-domain reverberators", Journal of Audio Engineering Society, 59(12), pp. 936-943. In such embodiments, it may be possible to perceptually approximate an existing late reverberation response by any reverberation structure that allows configuring the reverberation time (T60, i.e. the time it takes for a sound to decay by 60 dB) and the energy of the frequency band. These reverberation parameters of the reverberator algorithm can be set to match the corresponding characteristics of the existing response being approximated. The reverberation parameters may also be manually configured if the goal is not to mimic the existing late part response.
[0143] The late reverberation for the binaural output should be generated such that the diffuse field correlations match as a function of frequency, a feature that is described in various known methods. The diffuse field correlations over the frequency bands are given by the diffuse field covariance matrix C D It can be found from (k).
[0144] The binaural room effect signal 318 (the reverberant processed time domain signal) may then be fed to a combiner 315 .
[0145] The combiner 315 is configured to receive the early (catamaran early part signal 316 from the inverse T / F transformer 313) and the late (catamaran room effect signal 318 from the reverberator 351) and combine or sum them (separately for left and right channels). This combiner thus combines the binaural time domain signal corresponding to the early / dry part of the binaural processing with the binaural room effect signal to generate a resulting spatialized binaural time domain signal with additional room effect when needed / required. This signal can be played on headphones.
[0146] Referring to FIG. 4, a flow diagram illustrating the operation of the synthesis processor is shown.
[0147] The flow diagram illustrates the operations of receiving a carrier audio signal, spatial metadata, and added room effect control indicators, etc., as illustrated in FIG. 4 by step 401 .
[0148] Further, the HRTF data is determined as shown in FIG.
[0149] The generation of a room effect binaural audio signal based on the carrier audio signal and additional room effect controls is illustrated in FIG.
[0150] The generation of the time-frequency domain carrier audio signal is illustrated in FIG.
[0151] The estimation of the covariance matrix based on the total energy based on the T / F carrier audio signal and the covariance matrix is illustrated in FIG.
[0152] The determination of the target covariance matrix based on the HRTF data, spatial metadata, and energy estimates is illustrated in FIG.
[0153] Once the target covariance matrix and the estimated covariance matrix have been determined, a mixing rule is determined based on the estimated covariance matrix and the target covariance matrix, as shown in FIG. 4, in step 411 .
[0154] The time-frequency carrier signals may then be mixed based on the mixing rules as shown in FIG.
[0155] These mixed audio signals are then converted back to the time domain or a time domain equivalent audio signal is generated by step 415 as shown in FIG.
[0156] The room effects binaural audio signal (if required) and the early mix audio signal may then be combined (or summed) by step 417 as shown in FIG.
[0157] The combined binaural audio signal may then be output, per step 419, as shown in FIG.
[0158] In some embodiments, it is possible to utilize spatial metadata to control the late reverberation process. An important purpose of binaural reverberation in the context of binaural playback is to enable externalization / distance perception of sounds. Therefore, in some situations, it may be useful to render the reverberation to be more directing than the ambient parts. Therefore, a direct-to-total energy ratio parameter (or an equivalent parameter) can be applied to control the signal fed to the reverberator. This can be achieved by multiplying the transport signal in the frequency band by √(r(k,n)) before applying the reverberator and using a frequency-domain binaural reverberator algorithm. Furthermore, any type of control over the amount of signal fed to the reverberator based on the spatial metadata can be implemented.
[0159] An exemplary electronic device that may be used as any of the device components of the system, as described above with respect to FIG. 5. The device may be any suitable electronic device or equipment. For example, in some embodiments, the device 1700 is a mobile device, a user device, a tablet computer, a computer, an audio playback device, etc. The device may be configured to implement, for example, the encoder / analyzer unit 101 or the decoder / synthesizer unit 105 as shown in FIG. 1, or any of the functional blocks as described above.
[0160] In some embodiments, the device 1700 comprises at least one processor or central processing unit 1707 .
[0161] The processor 1707 may be configured to execute various program codes, such as the methods described herein.
[0162] In some embodiments, the device 1700 includes a memory device 1711 .
[0163] In some embodiments, the at least one processor 1707 is coupled to a storage device 1711. The storage device 1711 may be any suitable storage means.
[0164] In one embodiment, the storage device 1711 includes program code sections for storing program code implementable on the processor 1707. Additionally, in some embodiments, the storage device 1711 may further comprise a stored data section for storing data, e.g., data that has been processed or is to be processed according to the embodiments described herein. The implemented program code stored in the program code sections and the data stored in the stored data sections may be retrieved by the processor 1707 whenever needed via the memory-processor coupling.
[0165] In some embodiments, the device 1700 comprises a user interface 1705. The user interface 1705 may be coupled to a processor 1707 in some embodiments. In some embodiments, the processor 1707 may control the operation of the user interface 1705 and receive input from the user interface 1705. In some embodiments, the user interface 1705 may allow a user to input commands to the device 1700, for example, via a keypad. In some embodiments, the user interface 1705 may allow a user to obtain information from the device 1700. For example, the user interface 1705 may comprise a display configured to display information from the device 1700 to the user. The user interface 1705 may comprise a touch screen or touch interface that in some embodiments can both allow information to be input into the device 1700 and further display information to a user of the device 1700. In some embodiments, the user interface 1705 may be a user interface for communication.
[0166] In some embodiments, the apparatus 1700 comprises an input / output port 1709. The input / output port 1709 comprises a transceiver in some embodiments. The transceiver in such embodiments may be coupled to the processor 1707 and configured to enable communication with other apparatuses or electronic devices, for example, via a wireless communication network. The transceiver or any suitable transceiver or transmitter and / or receiver means may be configured in some embodiments to communicate with other electronic devices or apparatuses via a wire or wired coupling.
[0167] The transceiver may communicate with the further device by any suitable known communication protocol, for example in some embodiments the transceiver may use a suitable Universal Mobile Telecommunications System (UMTS) protocol, a Wireless Local Area Network (WLAN) protocol such as IEEE 802.X, a suitable short range radio frequency communication protocol such as Bluetooth, or an Infrared Data Path (IRDA).
[0168] The transceiver input / output port 1709 may be configured to receive a signal.
[0169] In some embodiments, the device 1700 may be used as at least a part of a synthesis device. The input / output port 1709 may be coupled to headphones (which may be head-tracked or non-tracked headphones) or the like.
[0170] In general, various embodiments of the present invention can be implemented in hardware or dedicated circuits, software, logic, or any combination thereof. For example, some aspects may be implemented in hardware, and other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device, but the present invention is not limited thereto. Although various aspects of the present invention may be illustrated and illustrated as block diagrams, flow charts, or using some other pictorial representations, it should be well understood that these blocks, devices, systems, techniques, or methods contemplated herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general-purpose hardware or controller, or other computing device, or some combination thereof, as non-limiting examples.
[0171] The embodiments of the present invention may be realized in computer software executable by a data processor of a mobile device. For example, it may be implemented by a processor entity, or by hardware, or by a combination of software and hardware. Furthermore, in this regard, it is noted that any block of the logic flow as in the diagram may represent program steps, or interconnected logic circuits, blocks and functions, or a combination of program steps and logic circuits, blocks and functions. The software may be stored on a physical medium such as a memory chip, or a memory block implemented in a processor, a magnetic medium such as a hard disk or a floppy disk, and an optical medium such as a DVD or a CD, for example, a variant of the data.
[0172] The memory may be of any type suitable for the local technology environment and may be implemented using any suitable data storage technology, such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed and removable memories, etc. The data processor may be of any type suitable for the local technology environment and may include, by way of non-limiting examples, one or more of general purpose computers, special purpose computers, microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), gate level circuits, and processors based on multi-core processor architectures.
[0173] Embodiments of the present invention can be implemented in a variety of components, such as integrated circuit modules. The design of integrated circuits is a highly automated process and is large scale. Complex and powerful software tools are available to convert logic level designs into complete semiconductor circuit designs that are ready to be etched and formed on semiconductor substrates.
[0174] Programs such as those offered by Synopsys, Inc. of Mountain View, California and Cadence Design, San Jose, California, use well-established rules of design and a library of prestored design modules to automatically route conductors and identify the locations of components on a semiconductor chip.
[0175] Once the design of a semiconductor circuit is completed, the resulting design in a standardized electronic format (e.g., Opus, GDSII, etc.) may be sent to a semiconductor manufacturing facility or "fab" for production.
[0176] The preceding description provides a complete and informative description of exemplary embodiments of the present invention, by way of illustrative and non-limiting examples.
[0177] However, various modifications and adaptations will become apparent to those skilled in the art in view of the foregoing description, upon perusal of the accompanying drawings and the appended claims.
[0178] However, all such similar modifications of the teachings of this invention will still fall within the scope of the present invention as defined in the appended claims.
Claims
1. An apparatus comprising at least one processor and at least one memory containing computer program code, the apparatus comprising at least receiving a spatial audio signal, the spatial audio signal including at least one audio signal and spatial metadata associated with the at least one audio signal; obtaining room effect control indicia; determining whether a room effect should be applied to the at least one audio signal based on the room effect control indication; configured to run The apparatus further comprises, when the room effect is to be applied to a spatial audio signal: generating a first partial binaural audio signal based on the at least one audio signal and spatial metadata; generating a second partial binaural audio signal based on the at least one audio signal, the at least second partial binaural audio signal being generated at least in part with room effects such that the at least second partial binaural audio signal has a response different from a response of the first partial binaural audio signal; combining the first partial binaural audio signal and the second partial binaural audio signal to generate a combined binaural audio signal; configured to run The device that generated the first partial binaural audio signal further comprises: analyzing the at least one audio signal to determine at least one stochastic characteristic associated with the at least one audio signal; generating the first partial binaural audio signal further based on the at least one stochastic characteristic associated with the at least one audio signal; configured to run the at least one audio signal includes at least two audio signals; the at least one audio signal is analyzed to determine the at least one probabilistic characteristic that enables the apparatus to estimate a covariance between the at least two audio signals; The apparatus for generating the first partial binaural audio signal further based on the at least one stochastic characteristic may further comprise: generating mixing coefficients based on an estimated covariance between the at least two audio signals; mixing at least two audio signals based on a mixing coefficient to generate a binaural audio signal of a first part; configured to run The apparatus further comprises: generating an overall energy estimate based on the estimated covariance; determining head-related transfer function data based on at least one directional parameter, said spatial metadata including the at least one directional parameter; determining a target covariance based on the head-related transfer function data, the spatial metadata and a global energy estimate; configured to execute Device.
2. the spatial metadata includes at least one orientation parameter; the apparatus generating the first partial binaural audio signal based on the at least one audio signal and the at least one directional parameter.
2. The apparatus of claim 1.
3. The spatial metadata includes at least one ratio parameter; the apparatus generating the second partial binaural audio signal based on the at least one audio signal and the at least one ratio parameter.
2. The apparatus of claim 1.
4. The apparatus of claim 2 , wherein the at least one directional parameter is a direction associated with a frequency band.
5. The apparatus of claim 1 , wherein the mixing coefficients are generated based on a target covariance.
6. The apparatus of claim 1 , wherein the apparatus that generated the second partial binaural audio signal is further configured to perform the step of applying a reverberator to the at least one audio signal.
7. The device that obtains the indoor effect control sign further includes: receiving the room effect control indicia as a flag set by an encoder of the spatial audio signal; receiving the room effects control indicia as a user input; determining the room effect control indicia based on obtaining an indication of a type of spatial audio signal; determining the room effects control indicia based on an analysis of the spatial audio signal to determine a type of the spatial audio signal; Executing at least one of the following:
2. The apparatus of claim 1.
8. The apparatus of claim 1 , wherein the at least one audio signal is at least one carrier audio signal generated by an encoder.
9. receiving a spatial audio signal, the spatial audio signal comprising: providing at least one audio signal and spatial metadata associated with said at least one audio signal; obtaining room effect control indicia; determining whether a room effect should be applied to the at least one audio signal based on a room effect control indication; generating a first partial binaural audio signal based on the at least one audio signal and spatial metadata when the room effect is applied to a spatial audio signal; generating a second partial binaural audio signal based on the at least one audio signal when the room effect is applied to the spatial audio signal, the at least second partial binaural audio signal being generated at least in part with the room effect such that it has a response that differs from a response of the first partial binaural audio signal; combining the first partial binaural audio signal and the second partial binaural audio signal to generate a combined binaural audio signal when a room effect is applied to the spatial audio signal; Including, The step of generating the first partial binaural audio signal based on the at least one audio signal and spatial metadata comprises: analyzing the at least one audio signal to determine at least one stochastic characteristic associated with the at least one audio signal; generating the first partial binaural audio signal further based on the at least one stochastic characteristic associated with the at least one audio signal; Including, the at least one audio signal includes at least two audio signals; Analysing the at least one audio signal to determine at least one stochastic characteristic associated with the at least one audio signal comprises estimating a covariance between the at least two audio signals; generating the first partial binaural audio signal further based on the at least one stochastic characteristic associated with the at least one audio signal, generating mix coefficients based on an estimated covariance between the at least two audio signals; mixing at least two audio signals based on a mixing coefficient to generate a binaural audio signal of a first part; Including, generating an overall energy estimate based on the estimated covariance; determining head-related transfer function data based on at least one directional parameter, said spatial metadata including said at least one directional parameter; determining a target covariance based on the head-related transfer function data, the spatial metadata, and the global energy estimate; The method further comprises:
10. the spatial metadata includes at least one orientation parameter; generating a first partial binaural audio signal based on the at least one audio signal and the spatial metadata comprises generating the first partial binaural audio signal based on the at least one audio signal and the at least one directional parameter.
10. The method of claim 9.
11. The spatial metadata includes at least one ratio parameter; The step of generating a second partial binaural audio signal based on the at least one audio signal comprises: generating a second partial binaural audio signal based on the at least one audio signal and the at least one ratio parameter.
10. The method of claim 9.
12. The method of claim 9 , wherein generating mixing coefficients based on the estimated covariance further comprises generating the mixing coefficients based on a target covariance.
13. generating a second partial binaural audio signal based on the at least one audio signal comprises applying a reverberator to the at least one audio signal.
10. The method of claim 9.
14. obtaining a room effect control indication includes receiving a room effect control indicator as a flag set by an encoder of the spatial audio signal; receiving the room effects control index as a user input; determining the room effect control indicia based on obtaining an indication of a type of spatial audio signal; determining the room effects control indicia based on an analysis of the spatial audio signal to determine a type of the spatial audio signal; The method of claim 9 , comprising at least one of:
Citation Information
Patent Citations
Encoding and Reproduction of Spatial Audio of Diffuse Sound
JP2013541275A
Binaural rendering apparatus and method for playing back of multiple audio sources
WO2018079254A1
Spatial audio parameters and associated spatial audio playback
WO2019193248A1
