Spatial Audio Representation and Rendering
By converting and matching the types of transmitted audio signals in the audio stream, the problem of rendering 6DoF content in MPEG-I audio technology is solved, and low-latency communication and multi-content mix audio rendering is achieved, improving audio quality and system compatibility.
Patent Information
- Application Number
- CN202010584221.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2019-06-25
- Filing Date
- 2020-06-24
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2040-06-24
AI Technical Summary
The existing MPEG-I audio technology has not yet defined how to render 6DoF content to users, especially in format support and low-latency communication scenarios, especially in social VR applications, it is difficult to effectively mix different metadata to control the mixing of multiple contents.
Through a device and method, the transmitted audio signal in the audio stream is obtained and converted to match the target type, and indicators and metadata associated with the target transmitted audio signal are generated for rendering by the renderer to realize the 6DoF content Flexible rendering.
It realizes effective rendering of 6DoF content, supports low-latency communication and multi-content mixing, and improves audio quality and system compatibility.
Smart Images

Figure CN112133316B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to apparatus and methods for spatial audio representation and rendering, but is not limited to audio representation for audio decoders. Background Art
[0002] Immersive audio codecs are being implemented to support a large number of operating points ranging from low bitrate operation to transparency. An example of such a codec is the Immersive Voice and Audio Service (IVAS) codec, which is designed to be suitable for use on communication networks such as 3GPP 4G / 5G networks, including use in immersive services such as immersive voice and audio for virtual reality (VR). The audio codec is expected to handle the encoding, decoding, and rendering of speech, music, and general audio. It is also expected to support channel-based audio and scene-based audio input, including spatial information about sound fields and sound sources. The codec is also expected to operate with low latency to enable conversational services and support high error robustness under various transmission conditions.
[0003] The input signal can be presented to the IVAS encoder in one of a variety of supported formats (and in some allowed format combinations). For example, a single-channel audio signal (without metadata) can be encoded using an enhanced voice service (EVS) encoder. Other input formats can utilize new IVAS encoding tools. One input format proposed for IVAS is the metadata-assisted spatial audio (MASA) format, in which the encoder can utilize, for example, a combination of single-channel and stereo encoding tools and metadata encoding tools for efficient transmission of the format. MASA is a parametric spatial audio format suitable for spatial audio processing. Parametric spatial audio processing is an audio signal processing field that uses a set of parameters to describe the spatial aspects of a sound (or sound scene). For example, in parametric spatial audio capture from a microphone array, estimating a set of parameters from a microphone array signal (e.g., the direction of the sound in a frequency band, the relative energy of the directional part and the non-directional part of the captured sound in the frequency band, such as being expressed as a direct-to-total energy ratio or an ambient-to-total energy ratio in a frequency band) is a typical and effective choice. It is known that these parameters describe well the perceived spatial characteristics of the captured sound at the location of the microphone array. Therefore, these parameters can be used in the synthesis of spatial sound for binaural headphones, for loudspeakers, or other formats such as Ambisonic.
[0004] For example, there may be two channels (stereo) of audio signals and spatial metadata. In addition, the spatial metadata may define the following parameters: a directional index, describing the direction of arrival of the sound at intervals of a time-frequency parameter; a direct-to-total energy ratio, describing the energy ratio for the directional index (i.e., time-frequency subframe); extended coherence, describing the energy spread for the directional index (i.e., time-frequency subframe); a diffuse-to-total energy ratio, describing the energy ratio of non-directional sounds in the surrounding directions; surround coherence, describing the coherence of non-directional sounds in the surrounding directions; a remainder-to-total energy ratio, describing the energy ratio of the remaining part (such as microphone noise) of the sound energy to meet the requirement that the sum of the energy ratios is 1; and a distance, describing the distance of the sound originating from the directional index (i.e., time-frequency subframe) in meters on a logarithmic scale.
[0005] IVAS streams can be decoded and rendered into various output formats, including binaural output, multi-channel output, and Ambisonic (FOA / HOA) output. In addition, there may be an interface for external rendering, where the output format may correspond to, for example, the input format.
[0006] Since spatial (e.g. MASA) metadata depicts the desired spatial audio perception in an output format-independent manner, any stream with spatial metadata can be flexibly rendered into any of the above output formats. However, since MASA streams can originate from various inputs, the transmitted audio signal received by the decoder may have different characteristics. Therefore, the decoder must take these aspects into account in order to be able to produce the best audio quality.
[0007] Immersive media technologies are currently being standardized by MPEG, under the name MPEG-I. These technologies include methods for various virtual reality (VR), augmented reality (AR) or mixed reality (MR) use cases. MPEG-I is divided into three stages: Stage 1a, Stage 1b and Stage 2. These stages are characterized by how the so-called degrees of freedom in 3D space are considered. Stages 1a and 1b consider 3DoF and 3DoF+ use cases, and then Stage 2 will allow at least significantly unrestricted 6DoF.
[0008] An example of an augmented reality (AR) / virtual reality (VR) / mixed reality (MR) application is audio (or audio-visual) environment immersion in which 6 degrees of freedom (6DoF) content rendering is achieved.
[0009] It is currently expected that MPEG-I Audio will be based on MPEG-H 3D Audio. However, additional 6DoF technology will be required on top of MPEG-H 3D Audio, including at least: additional metadata to support 6DoF and an interactive 6DoF renderer that also supports linear conversion. Note that MPEG-H 3D Audio includes an Ambisonic signal, and MPEG-I Audio is expected to support this. MPEG-I will also include support for low-latency communication audio, for example to support use cases such as social VR. The audio may be spatial. It is not yet defined how this audio will be rendered to the user (e.g. format support, mixing with native MPEG-I content). It is at least expected that there will be some metadata support to control the mixing of at least two contents. Summary of the invention
[0010] According to a first aspect, an apparatus is provided, comprising components configured to: obtain at least one audio stream, wherein the at least one audio stream comprises one or more transmission audio signals, wherein the one or more transmission audio signals are transmission audio signals of a defined type; and convert the one or more transmission audio signals into at least one or more other transmission audio signals, wherein the one or more other transmission audio signals are transmission audio signals of another defined type.
[0011] The defined type of transmission audio signal and / or the further defined type of transmission audio signal may be associated with an origin of the transmission audio signal or with a simulated origin of the transmission audio signal.
[0012] The component may also be configured to obtain an indicator representing another defined type of transmission audio signal, and wherein the component configured to convert the one or more transmission audio signals into at least one or more other transmission audio signals, the one or more other transmission audio signals may be transmission audio signals of another defined type, is configured to convert the one or more transmission audio signals into at least one or more other transmission audio signals based on the indicator.
[0013] The indicator may be obtained from a renderer configured to receive the one or more other transport audio signals and to render the one or more other transport audio signals.
[0014] The component may also be configured to provide at least one other transmission audio signal for rendering.
[0015] The component may also be configured to: generate an indicator associated with another defined type of transmission audio signal; and provide the indicator associated with at least one other transmission audio signal as additional metadata together with the at least one other transmission audio signal for rendering.
[0016] The component may also be configured to determine a transmitted audio signal of a defined type.
[0017] At least one audio stream may further include an indicator identifying a transmission audio signal of a defined type associated with the one or more transmission audio signals, wherein the component configured to determine the transmission audio signal of the defined type may be configured to determine the transmission audio signal of the defined type associated with the one or more transmission audio signals based on the indicator.
[0018] The means configured to determine a transmission audio signal of a defined type may be configured to determine a transmission audio signal of a defined type based on an analysis of one or more transmission audio signals.
[0019] A component configured to convert one or more transmission audio signals into at least one or more other transmission audio signals, the one or more other transmission audio signals being transmission audio signals of another defined type, may be configured to: generate at least one prototype signal based on at least one transmission audio signal, a defined type of the transmission audio signal, and another defined type of the transmission audio signal; determine at least one expected characteristic of the one or more other transmission audio signals; and, based on the determined at least one expected characteristic of the one or more other transmission audio signals, mix at least one prototype signal with a decorrelated version of at least one prototype signal to generate at least one other transmission audio signal.
[0020] The defined type of at least one audio signal may be at least one of: capture microphone arrangement; capture microphone separation distance; capture microphone parameters; transmission channel identifier; cardioid audio signal type; interval audio signal type; downmix audio signal type; coincident audio signal type; and transmission channel arrangement.
[0021] The component may also be configured to render one or more other transmitted audio signals.
[0022] The component configured to render at least one other audio signal may be configured to perform one of: converting one or more other transmission audio signals into an Ambisonic audio signal representation; converting one or more other transmission audio signals into a two-channel audio signal representation; and converting one or more other transmission audio signals into a multi-channel audio signal representation.
[0023] At least one of the audio streams may include spatial metadata associated with the one or more transmitted audio signals.
[0024] The component may also be configured to provide at least one other transmitted audio signal and spatial metadata associated with the one or more transmitted audio signals for rendering.
[0025] According to a second aspect, a method is provided, comprising: obtaining at least one audio stream, wherein the at least one audio stream comprises one or more transmission audio signals, wherein the one or more transmission audio signals are transmission audio signals of a defined type; and converting the one or more transmission audio signals into at least one or more other transmission audio signals, wherein the one or more other transmission audio signals are transmission audio signals of another defined type.
[0026] The defined type of transmission audio signal and / or the further defined type of transmission audio signal may be associated with an origin of the transmission audio signal or with a simulated origin of the transmission audio signal.
[0027] The method may further include: obtaining an indicator representing another defined type of transmission audio signal, and wherein converting one or more transmission audio signals into at least one or more other transmission audio signals, the one or more other transmission audio signals being another defined type of transmission audio signal, may include: converting one or more transmission audio signals into at least one or more other transmission audio signals based on the indicator.
[0028] The indicator may be obtained from a renderer configured to receive the one or more other transport audio signals and to render the one or more other transport audio signals.
[0029] The method may further comprise providing at least one other transmission audio signal for rendering.
[0030] The method may further comprise: generating an indicator associated with another defined type of transmission audio signal; and providing the indicator associated with at least one other transmission audio signal as additional metadata together with the at least one other transmission audio signal for rendering.
[0031] The method may further comprise determining a transmission audio signal of a defined type.
[0032] The at least one audio stream may further include an indicator identifying a defined type of transmission audio signal associated with the one or more transmission audio signals, wherein determining the defined type of transmission audio signal includes determining the defined type of transmission audio signal associated with the one or more transmission audio signals based on the indicator.
[0033] Determining the defined type of transmission audio signal may include determining the defined type of transmission audio signal based on an analysis of one or more transmission audio signals.
[0034] Converting one or more transmission audio signals into at least one or more other transmission audio signals, the one or more other transmission audio signals being transmission audio signals of another defined type, may include: generating at least one prototype signal based on at least one transmission audio signal, a defined type of the transmission audio signal, and another defined type of the transmission audio signal; determining at least one expected characteristic of the one or more other transmission audio signals; and mixing at least one prototype signal with a decorrelated version of at least one prototype signal based on the determined at least one expected characteristic of the one or more other transmission audio signals to generate at least one other transmission audio signal.
[0035] The defined type of at least one audio signal may be at least one of: capture microphone arrangement; capture microphone separation distance; capture microphone parameters; transmission channel identifier; cardioid audio signal type; interval audio signal type; downmix audio signal type; coincident audio signal type; and transmission channel arrangement.
[0036] The method may further include rendering one or more other transmission audio signals.
[0037] Rendering at least one other audio signal may include one of: converting one or more other transmitted audio signals into an Ambisonic audio signal representation; converting one or more other transmitted audio signals into a two-channel audio signal representation; and converting one or more other transmitted audio signals into a multi-channel audio signal representation.
[0038] At least one of the audio streams may include spatial metadata associated with the one or more transmitted audio signals.
[0039] The method may further comprise providing at least one other transmitted audio signal and spatial metadata associated with the one or more transmitted audio signals for rendering.
[0040] According to a third aspect, an apparatus is provided, the apparatus comprising at least one processor and at least one memory comprising computer program code, the at least one memory and the computer program code being configured to, together with the at least one processor, cause the apparatus to at least: obtain at least one audio stream, wherein the at least one audio stream comprises one or more transmission audio signals, wherein the one or more transmission audio signals are transmission audio signals of a defined type; and convert the one or more transmission audio signals into at least one or more other transmission audio signals, the one or more other transmission audio signals being transmission audio signals of another defined type.
[0041] The defined type of transmission audio signal and / or the further defined type of transmission audio signal may be associated with an origin of the transmission audio signal or with a simulated origin of the transmission audio signal.
[0042] The apparatus may also be caused to obtain an indicator representing another defined type of transmission audio signal, and wherein the apparatus may be caused to convert the one or more transmission audio signals into at least one or more other transmission audio signals, the one or more other transmission audio signals being transmission audio signals of another defined type, and the apparatus may be caused to convert the one or more transmission audio signals into at least one or more other transmission audio signals based on the indicator.
[0043] The indicator may be obtained from a renderer configured to receive one or more other transport audio signals and to render the one or more other transport audio signals.
[0044] The apparatus may also be caused to provide at least one other transmission audio signal for rendering.
[0045] The apparatus may also be caused to: generate an indicator associated with another defined type of transmission audio signal; and provide the indicator associated with at least one other transmission audio signal as additional metadata together with the at least one other transmission audio signal for rendering.
[0046] The apparatus may also be caused to determine a transmission audio signal of a defined type.
[0047] At least one audio stream may further comprise an indicator identifying a transmission audio signal of a defined type associated with the one or more transmission audio signals, wherein the means which may be caused to determine the transmission audio signal of the defined type may be caused to determine the transmission audio signal of the defined type associated with the one or more transmission audio signals based on the indicator.
[0048] The means which may be caused to determine a transmission audio signal of a defined type may be caused to determine a transmission audio signal of a defined type based on an analysis of one or more transmission audio signals.
[0049] An apparatus that can be caused to convert one or more transmission audio signals into at least one or more other transmission audio signals, the one or more other transmission audio signals being transmission audio signals of another defined type, can be caused to: generate at least one prototype signal based on at least one transmission audio signal, a defined type of the transmission audio signal, and another defined type of the transmission audio signal; determine at least one expected characteristic of the one or more other transmission audio signals; and mix at least one prototype signal with a decorrelated version of at least one prototype signal based on the determined at least one expected characteristic of the one or more other transmission audio signals to generate at least one other transmission audio signal.
[0050] The defined type of at least one audio signal may be at least one of: capture microphone arrangement; capture microphone separation distance; capture microphone parameters; transmission channel identifier; cardioid audio signal type; interval audio signal type; downmix audio signal type; coincident audio signal type; and transmission channel arrangement.
[0051] The apparatus may also be caused to render one or more other transmitted audio signals.
[0052] The device caused to render at least one other audio signal may be caused to perform one of: converting one or more other transmitted audio signals into an Ambisonic audio signal representation; converting one or more other transmitted audio signals into a two-channel audio signal representation; and converting one or more other transmitted audio signals into a multi-channel audio signal representation.
[0053] At least one of the audio streams may include spatial metadata associated with the one or more transmitted audio signals.
[0054] The apparatus may also be caused to provide at least one other transmission audio signal and spatial metadata associated with the one or more transmission audio signals for rendering.
[0055] According to a fourth aspect, an apparatus is provided, comprising: a component for obtaining at least one audio stream, wherein the at least one audio stream comprises one or more transmission audio signals, wherein the one or more transmission audio signals are transmission audio signals of a defined type; a component for converting the one or more transmission audio signals into at least one or more other transmission audio signals, wherein the one or more other transmission audio signals are transmission audio signals of another defined type.
[0056] According to a fifth aspect, a computer program comprising instructions (or a computer-readable medium comprising program instructions) is provided, which instructions (or program instructions) cause an apparatus to perform at least the following operations: obtain at least one audio stream, wherein the at least one audio stream comprises one or more transmission audio signals, wherein the one or more transmission audio signals are transmission audio signals of a defined type; and convert the one or more transmission audio signals into at least one or more other transmission audio signals, wherein the one or more other transmission audio signals are transmission audio signals of another defined type.
[0057] According to a sixth aspect, a non-transitory computer-readable medium is provided, comprising program instructions for causing an apparatus to perform at least the following operations: obtaining at least one audio stream, wherein the at least one audio stream comprises one or more transmission audio signals, wherein the one or more transmission audio signals are transmission audio signals of a defined type; and converting the one or more transmission audio signals into at least one or more other transmission audio signals, wherein the one or more other transmission audio signals are transmission audio signals of another defined type.
[0058] According to the seventh aspect, a device is provided, comprising: an acquisition circuit configured to obtain at least one audio stream, wherein the at least one audio stream includes one or more transmission audio signals, wherein the one or more transmission audio signals are transmission audio signals of a defined type; and a conversion circuit configured to convert the one or more transmission audio signals into at least one or more other transmission audio signals, wherein the one or more other transmission audio signals are transmission audio signals of another defined type.
[0059] According to an eighth aspect, a computer-readable medium is provided, comprising program instructions for causing an apparatus to perform at least the following operations: obtain at least one audio stream, wherein the at least one audio stream comprises one or more transmission audio signals, wherein the one or more transmission audio signals are transmission audio signals of a defined type; and convert the one or more transmission audio signals into at least one or more other transmission audio signals, wherein the one or more other transmission audio signals are transmission audio signals of another defined type.
[0060] An apparatus comprises means for performing the actions of the method as described above.
[0061] A device is configured to perform the actions of the method described above.
[0062] A computer program includes program instructions for causing a computer to execute the method described above.
[0063] A computer program product stored on a medium may cause an apparatus to perform the method described herein.
[0064] An electronic device may include an apparatus as described herein.
[0065] A chipset may include the apparatus as described herein.
[0066] The embodiments of the present application are intended to solve the problems associated with the prior art. BRIEF DESCRIPTION OF THE DRAWINGS
[0067] For a better understanding of the present application, reference will now be made by way of example to the accompanying drawings, in which:
[0068] Figure 1 schematically illustrates a system suitable for implementing the apparatus of some embodiments;
[0069] Figure 2 a flow chart illustrating the operation of an example apparatus according to some embodiments;
[0070] Figure 3 Schematically illustrating how Figure 1 The transmission audio signal type converter shown in ;
[0071] Figure 4 a flow chart illustrating the operation of an example apparatus according to some embodiments;
[0072] Figure 5 shows a linearly generated cardioid pattern according to a first example implementation as shown in some embodiments;
[0073] Figure 6 and Figure 7 A system schematically illustrates another apparatus suitable for implementing some embodiments;
[0074] Figure 8 An example device suitable for implementing the means as shown in the previous figures is shown. DETAILED DESCRIPTION
[0075] Suitable apparatus and possible mechanisms for providing efficient rendering of spatial metadata aided audio signals are described in further detail below.
[0076] Although the following examples focus on MASA encoding and decoding, it should be noted that the proposed method is applicable to any system that utilizes a transmitted audio signal and spatial metadata. The spatial metadata may include, for example, some of the following parameters in any kind of combination: direction; level / phase difference; direct to total energy ratio; diffuseness; coherence (such as extended coherence and / or ambient coherence); and distance. Typically, these parameters are given in the time-frequency domain. Therefore, when the terms IVAS and / or MASA are used below, it should be understood that they can be replaced by any other suitable codec and / or metadata format and / or system.
[0077] As discussed above, the IVAS codec is expected to be able to process MASA streams with different kinds of transport audio signals. However, IVAS is also expected to support external renderers. In this case, it is not guaranteed that all external renderers support MASA streams with all possible types of transport audio signals and therefore cannot be optimally utilized with external renderers.
[0078] For example, an external renderer may utilize an Ambisonic based binaural rendering, where the transmitted signal type is assumed to be cardioid, and summation and difference operations may be used to directly generate the W and Y components of the Ambisonic signal from the cardioid. Therefore, if the transmitted signal type is not cardioid, such a spatial audio stream cannot be used directly with that external renderer.
[0079] Furthermore, MASA streams (or any other spatial audio stream consisting of a transport audio signal and spatial metadata) can be used outside the IVAS codec.
[0080] The concepts discussed in the following embodiments are apparatus and methods that can modify transmission audio signals so that they match the target type and can therefore be used more flexibly.
[0081] Thus, embodiments as discussed in further detail herein relate to processing of spatial audio streams (comprising a transport audio signal and metadata). Furthermore, these embodiments discuss apparatus and methods for changing the transport audio signal type of a spatial audio stream to achieve compatibility with systems requiring a specific transport audio signal type. Furthermore, in these embodiments, the transport audio signal type may be changed by: obtaining a spatial audio stream; determining the transport audio signal type of the spatial audio stream; obtaining a target transport audio signal type; modifying the transport audio signal to match the target transport audio signal type; changing the transport audio signal type field of the spatial audio stream to the target transport audio signal type (if the field exists); and allowing the modified spatial audio stream to be processed by systems requiring a specific transport audio signal type.
[0082] In the following embodiments, the apparatus and method enable changing the type of a spatial audio stream transmission audio signal. Thus, the spatial audio stream can be converted to be compatible with a system that allows the use of spatial audio streams with certain types of transmission audio signal types. The apparatus and method can, for example, use the spatial audio stream to render two-channel (or multi-channel speaker) audio.
[0083] In some embodiments, the method and apparatus may be implemented, for example, in the context of IVAS (e.g., in a mobile device supporting IVAS). The embodiment may be utilized between an IVAS decoder and an external renderer (e.g., a binaural renderer). In some embodiments where the external renderer only supports a certain transport audio signal type, the embodiments may be configured to modify a spatial audio stream having a different transport audio signal type to match the supported transport audio signal type.
[0084] The type of transmitted audio signal may be a type such as that described in UK patent application number GB 1904261.3. These types may include types such as "spaced", "cardioid", "coincident" and the like.
[0085] about Figure 1 , shows an example apparatus and system for implementing audio capture and rendering according to some embodiments (and converting a spatial audio stream of type “interval” into a transmission audio signal of type “cardioid”).
[0086] The system 199 is shown with a microphone array audio signal 100 input. In the following examples, a microphone array audio signal 100 input is described, but in other embodiments any suitable multi-channel input (or synthesized multi-channel) format may be implemented.
[0087] The system 199 may include a spatial analyzer 101. The spatial analyzer 101 is configured to perform spatial analysis on microphone signals, thereby generating a transmission audio signal 102 and metadata 104.
[0088] In some embodiments, the spatial analyzer and spatial analysis may be implemented external to the system 199. For example, in some embodiments, spatial metadata associated with the audio signal may be provided to the encoder as a separate bitstream. In some embodiments, the spatial metadata may be provided as a set of spatial (directional) index values.
[0089] The spatial analyzer 101 can be configured to create the transmission audio signal 102 in any suitable manner. For example, in some embodiments, the spatial analyzer is configured to select two microphone signals for use as the transmission audio signal. For example, the two selected microphone audio signals can be one microphone audio signal on the left side of the mobile device and another microphone audio signal on the right side of the mobile device. Therefore, the transmission audio signal can be considered as an interval microphone signal. In addition, typically, some pre-processing (such as equalization, noise reduction, and automatic gain control) is applied to the microphone signals.
[0090] Metadata can have various forms and can include spatial metadata and other metadata. Typical parameterizations for spatial metadata are a directional parameter θ(k,n) in each frequency band and an associated direct to total energy ratio r(k,n) in each frequency band, where k is the frequency band index and n is the time frame index. Determining or estimating the direction and ratio depends on the device or implementation from which the audio signal is obtained. For example, the metadata can be obtained or estimated using spatial audio capture (SPAC) using the methods described in UK patent application No. 1619573.7 and PCT patent application No. PCT / FI2017 / 050778. In other words, in this particular context, spatial audio parameters include parameters intended to characterize the characteristics of the sound field. In some embodiments, the generated parameters may be different in different frequency bands. Thus, for example, in frequency band X, all parameters are generated and sent, while in frequency band Y, only one parameter is generated and sent, and in frequency band Z, no parameters are generated or sent. A practical example of this may be that for some frequency bands (such as the highest frequency band), certain parameters are not required for perceptual reasons.
[0091] In some embodiments, the obtained metadata may include metadata in addition to spatial metadata. For example, in some embodiments, the obtained metadata may be a "channel audio format" parameter that describes the type of the transmitted audio signal. In this example, the "channel audio format" parameter may have a value of "interval". In addition, in some embodiments, the metadata also includes a parameter that defines or represents the distance between microphones. In some embodiments, the distance parameter may be sent with a signal. The transmitted audio signal and metadata may be arranged or configured in a MASA or in any other suitable form.
[0092] The transport audio signal 102 (of type “interval”) and metadata 104 may be output from the spatial analyzer 101 to an encoder 105 .
[0093] In some embodiments, the system 199 includes an encoder 105. The encoder 105 can be configured to receive a transmission audio signal 102 (of type "interval") and metadata 104 from the spatial analyzer 101. In some embodiments, the encoder 105 can be a mobile device, a user device, a tablet computer, a computer (running suitable software stored on a memory and at least one processor), or (alternatively) a specific device utilizing, for example, an FPGA or ASIC. The encoder can be configured to implement any suitable encoding scheme. In addition, the encoder 105 can also be configured to receive metadata and generate information in an encoded or compressed form. In some embodiments, the encoder 105 can further interleave the metadata, multiplex it into a single data stream 106, or embed the metadata in the encoded audio signal before sending or storing. Any suitable scheme can be used to implement multiplexing.
[0094] The encoder may be an IVAS encoder or any other suitable encoder. Thus, the encoder 105 is configured to encode the audio signal and the metadata and form a bitstream 106 (eg, an IVAS bitstream).
[0095] In addition, the system 199 may also include a decoder 107. The decoder 107 is configured to receive, retrieve or otherwise obtain the bitstream 106, demultiplex the encoded stream from the bitstream, and decode the audio signal to obtain the transmission signal 108. Similarly, the decoder 107 can be configured to receive and decode the encoded metadata 110. In some embodiments, the decoder 107 can be a mobile device, a user device, a tablet computer, a computer (running suitable software stored on a memory and at least one processor), or (alternatively) a specific device using, for example, an FPGA or an ASIC.
[0096] The system 199 may further include a transmission signal type converter 111. The transmission signal type converter 111 may be configured to obtain a transmission audio signal 108 (in this example, of type "interval") and metadata 110, and also receive a "target" transmission audio signal type input 118 from a spatial synthesizer 115. The transmission signal type converter 111 may be configured to convert the input transmission signal type to a "target" transmission signal type based on the transmission audio signal type 118 indicator received from the spatial synthesizer 115. In some embodiments, the transmission signal type converter 111 is configured to convert the input or original transmission audio signal based on the (spatial) metadata, the input transmission audio signal type, and the target transmission audio signal type so that the new transmission audio signal matches the target transmission audio signal type. In some embodiments, the (spatial) metadata is not used in the conversion. For example, the conversion of the FOA transmission audio signal to the cardioid transmission audio signal can be implemented with a linear operation without any (spatial) metadata. In some embodiments, the transmission signal type converter is configured to convert the input or original transmission audio signal without explicitly receiving the target transmission audio signal type.
[0097] In this example, the goal is to use the spatial synthesizer 115 to render spatial audio (e.g., two-channel audio) with these signals. However, in this example, the spatial synthesizer 115 only accepts spatial audio streams whose transmission audio signals are of the "cardioid" type. In other words, the spatial synthesizer expects two coincident cardioids pointing at ±90 degrees, for example, and is configured to process any two signal inputs accordingly. Therefore, the spatial audio stream from the decoder cannot be used directly to achieve correct rendering, but a transmission audio signal type converter 111 is used between the decoder 107 and the spatial synthesizer 115.
[0098] In this example, the "target" type is a coincident cardioid pointing at ±90 degrees (this is just an example, it can be any type.) Additionally, if the metadata has a field describing the transmitted audio signal type (e.g., a channel audio format metadata parameter), it can be configured to change that indicator or parameter to indicate the new transmitted audio signal type (e.g., "cardioid").
[0099] The modified transmitted audio signal (eg “cardioid” type) 112 and (possibly) modified metadata 114 are forwarded to a spatial synthesizer 115 .
[0100] In some embodiments, the system 199 includes a spatial synthesizer 115, which is configured to receive a (modified) transmission audio signal 112 (of type "cardioid" in this example) and (possibly) modified metadata 114. Thus, since the transmission audio signal is of a supported type, the spatial synthesizer 115 can be configured to render spatial audio (e.g., binaural audio) using the spatial audio stream it receives.
[0101] In some embodiments, the spatial synthesizer 115 is configured to create a first-order Ambisonic (FOA) signal. W and Y are obtained linearly from the transmitted audio signal (whose type is "cardioid") by the following formula:
[0102] W(b,n)=S card,left (b,n)+S card,right (b,n)
[0103] Y(b,n)=S card,left (b,n)-S card,right (b,n)
[0104] In some embodiments, the spatial synthesizer 115 may be configured to generate the X and Z dipoles from the omnidirectional signal W using a suitable parametric processing procedure such as discussed in UK patent application 1616478.2 and PCT patent application PCT / FI2017 / 050664. The index b indicates the frequency bin index of the applied time-frequency transform and n indicates the time index.
[0105] Then, in some embodiments, the spatial synthesizer 115 can be configured to generate or synthesize a binaural signal from the FOA signal (W, Y, Z, X). This can be achieved by applying a static matrix operation to the FOA signal in the frequency domain, which has been designed (for each frequency point) to approximate the head-related transformation function (HRTF) data set for the FOA input. In some embodiments, the transformation from FOA to HRTF can take the form of a filter matrix. In some embodiments, the FOA signal rotation matrix can be applied according to the user's head orientation before the matrix operation (or filtering).
[0106] The system operates on Figure 2 The flowchart shown is summarized. Figure 2 For example, the receiving of an audio signal by a microphone array is shown, as shown in step 201 .
[0107] The flow chart then shows the analysis (spatial) of the microphone array audio signal, as Figure 2 As shown in step 203 in .
[0108] The generated transmission audio signal (in this example, an interval type transmission audio signal) and metadata can then be encoded, such as Figure 2 As shown in step 205 in .
[0109] Then, the transmission audio signal (in this example, a transmission audio signal of the interval type) and the metadata can be decoded, such as Figure 2 As shown in step 207 in .
[0110] The transmitted audio signal can then be converted to a "target" type, in this example a cardioid type transmitted audio signal, such as Figure 2 As shown in step 209 in .
[0111] The spatial audio signals can then be synthesized to output a suitable output format, such as Figure 2 As shown in step 211 in .
[0112] about Figure 3 , shows a transmission signal type converter 111 which is adapted to convert a "spaced" transmission audio signal type into a "cardioid" transmission audio signal type.
[0113] In some embodiments, the transmission signal type converter 111 includes a time-frequency converter 301. The time-frequency converter 301 is configured to receive the transmission audio signal 108 and convert it to the time-frequency domain, in other words, output a suitable T / F domain transmission audio signal 302. Suitable transformations include, for example, short-time Fourier transform (STFT) and complex modulated quadrature mirror filter bank (QMF). The resulting signal is denoted as S i(b, n), where i is the channel index, b is the frequency point index, and n is the time index. This can be omitted when the transmitted audio signal (output from the extractor and / or decoder) is already in the time-frequency domain, or alternatively, it can include a transformation from one time-frequency domain representation to another. The T / F domain transmitted audio signal 302 can be forwarded to the prototype signal creator 303.
[0114] In some embodiments, the transmission signal type converter 111 includes the prototype signal creator 303. The prototype signal creator 303 is configured to receive the T / F domain transmitted audio signal 302. The prototype signal creator 303 is also configured to receive an indicator of the target transmitted audio signal type 118 and, in some embodiments, an indicator of the original transmitted audio signal type 304. The prototype signal creator 303 is then configured to output the time-frequency domain prototype signal 308 to the decorrelator 305 and the mixer 307. The creation of the prototype signal depends on the original transmitted audio signal type and the target transmitted audio signal type. In this example, the original transmitted signal type is "omnidirectional" and the target transmitted signal type is "cardioid".
[0115] Spatial metadata is determined in frequency band k, where each frequency band relates to one or more frequency points b. In some embodiments, the resolution is such that the high frequency bands k contain more frequency points b than the low frequency bands, thus approximating the frequency selectivity characteristics of human hearing. However, in some embodiments, the resolution can be any suitable frequency band arrangement for any appropriate number of frequency points. In some embodiments, the prototype signal creator 303 operates over three frequency ranges.
[0116] In this example, the three frequency ranges are as follows:
[0117] - The low frequency range (k ≤ K1) includes frequency points b where the audio wavelength is considered long relative to the microphone spacing of the transmitted audio signal
[0118] - The high frequency range (K2 < k) includes frequency points b where the audio wavelength is considered short relative to the microphone spacing of the transmitted audio signal
[0119] - The mid frequency range (K1 < k ≤ K2)
[0120] The audio wavelength being long means that the signals in the transmitted audio signal are highly similar, and thus the difference operation (e.g., S1(b, n) - S2(b, n)) provides a signal with a very small magnitude. This may result in a signal with a poor SNR because the microphone noise is not attenuated at the differential signal.
[0121] The fact that the audio wavelength is short means that the beamforming process does not work well and spatial aliasing occurs. For example, a linear combination of the transmitted signals can generate a beam pattern with a cardioid shape for the mid-frequency range. However, in the high-frequency range, it is not possible to generate such a pattern by linear operations. As is well known in the art of microphone array processing, the resulting pattern will have several side lobes and the generated pattern is not useful in this example. For example, Figure 5 shows what happens if a linear operation is applied at high frequencies. For example, Figure 5 It is shown that for frequencies above about 1 kHz, the output pattern is not that good.
[0122] In some embodiments, the frequency ranges K1 and K2 may be determined based on the distance d (in meters) between the microphones transmitting the signal. For example, the following formula may be used to determine the frequency limits in Hz:
[0123]
[0124]
[0125] Where c is the speed of sound. K1 is the highest frequency band index, where the frequency corresponding to the lowest frequency point index is lower than f1. K2 is the lowest frequency band index, where the frequency corresponding to the highest frequency point index is higher than f2.
[0126] In some cases, the distance d can be obtained from a transmitted audio signal type parameter or other suitable parameter or indicator. In other cases, the distance can be estimated. For example, the inter-microphone delay values can be monitored to determine the highest highly coherent delay between the microphones, and the microphone distance can be estimated based on the highest delay value. In some embodiments, the normalized cross-correlation of the microphone signals as a function of frequency can be measured over a suitable time interval, and the resulting cross-correlation pattern can be compared with an ideal diffuse field cross-correlation pattern for different distances d, and the best fit d is then selected.
[0127] In some embodiments, the prototype signal creator 303 is configured to implement the following processing operations on the low frequency range and the high frequency range.
[0128] Since the low frequency range has a highly coherent microphone audio signal, the prototype signal creator 303 is configured to generate a prototype signal by adding or combining the T / F transmission audio signals together.
[0129] The prototype signal creator 303 is configured not to combine or add together the T / F transmission audio signals for the high frequency range, as this would produce an undesirable comb filtering effect. Therefore, in some embodiments, the prototype signal creator 303 is configured to generate the prototype signal by selecting one channel (e.g., the first channel) of the T / F transmission audio signal.
[0130] The prototype signals generated for the high frequency range and the low frequency range are single-channel signals.
[0131] The prototype signal creator 303 (for the low frequency range and the high frequency range) can then be configured to equalize the generated prototype signals using appropriate time smoothing. This equalization is performed so that the output audio signal has a signal S i The average energy of (b,n).
[0132] The prototype signal creator 303 is configured to output the intermediate frequency range of the T / F transmission audio signal 302 as a T / F prototype signal 308 (in the intermediate frequency range) without any processing.
[0133] The low frequency range and high frequency range are marked as S p,mono The equalized prototype signal of (b, n) and the unprocessed intermediate frequency range transmission audio signal are output as the prototype audio signal 308 to the decorrelator 305 and the mixer 307 .
[0134] In some embodiments, the transmission signal type converter 111 includes a decorrelator 305. The decorrelator 305 is configured to generate a non-coherent decorrelated signal in the low frequency range and the high frequency range based on the prototype signal. In the medium frequency range, no decorrelated signal is required. The output is provided to the mixer 307. The decorrelated signal is denoted as S d,mono (b,n). The decorrelated signal ideally has the same p,mono (b,n) have the same energy, but these signals are ideally incoherent with each other.
[0135] In some embodiments, the transmission signal type converter 111 includes a target signal characteristic determiner 309. The target signal characteristic determiner 309 is configured to receive the spatial metadata 110 and the target transmission audio signal type 118. The target signal characteristic determiner 309 is configured to formulate a target covariance matrix using the metadata azimuth angle azi(k,n), elevation angle ele(k,n) and direct-to-total energy ratio r(k,n). For example, the target signal characteristic determiner 309 is configured to formulate left and right cardioid gains:
[0136] g l (k,n)=0.5+0.5 sin(azi(k,n))cos(ele(k,n))
[0137] g r (k,n)=0.5-0.5sin(azi(k,n))cos(ele(k,n))
[0138] Then, the target covariance matrix is
[0139]
[0140] The rightmost matrix definition is related to the energy and correlation of the two cardioid signals in the diffuse field. The target covariance matrix as the target signal characteristic 320 is provided to the mixer 307 .
[0141] In some embodiments, the transmission signal type converter 111 includes a mixer 307. The mixer 307 is configured to receive outputs from the decorrelator 305 and the prototype signal creator 303. Furthermore, the mixer 307 is configured to receive a target covariance matrix as the target signal characteristic 320.
[0142] The mixer can be configured for the low frequency range and the high frequency range to define the input signal of the mixing operation as a combination of the prototype signal (first channel) and the decorrelated signal (second channel):
[0143]
[0144] The mixing process may use any suitable process, for example, based on "An optimized covariance domain framework for time-frequency processing of spatial audio" (J Vilkamo, A Kuntz - Journal of the Audio Engineering Society, 2013) method for generating mixing matrices.
[0145] The formulated mixing matrix M (time index and frequency index are temporarily omitted) can be based on the following matrix.
[0146] In the above, the target covariance matrix is determined in normalized form (i.e., without absolute energy), so the covariance matrix of the signal x can also be determined in normalized form: the signals contained in x are incoherent but have the same energy, and therefore, its covariance matrix can be fixed as The prototype matrix can be determined as This matrix guides the generation of the mixing matrix. The principles of these matrices and the formulas for obtaining the mixing matrix M based on them have been fully explained in the references cited above and will not be repeated here. In short, the method provides a mixing matrix M that, when applied to a matrix with a covariance matrix C x The signal with covariance matrix C is generated by least squares optimization. yThe matrix Q guides the signal content in this mixture: in this example, mainly non-decorrelated sounds are utilized, and when necessary, decorrelated sounds with positive signs are directed to the first output channel and decorrelated sounds with negative signs are directed to the second output channel.
[0147] The mixing matrix M(k,n) can be formulated for each frequency band k and applied to each frequency point b within the frequency band k to generate the output signal
[0148] y(b,n)=M(k,n)x(b,n).
[0149] For the mid-frequency range, the mixer 307 has the information that a “cardioid” transmission audio signal type is to be rendered and therefore formulates a mixing matrix M for each frequency point (within the frequency band of the mid-frequency range): mid And applies it to the input signal (which is a T / F transmission audio signal in the intermediate frequency range) to generate a new transmission audio signal.
[0150]
[0151] In some embodiments, the mixing matrix M mid can be formulated as a function of d as follows. In this example, each frequency bin b has a center frequency f b First, the mixer is configured to determine the normalized gain:
[0152]
[0153]
[0154] The mixing matrix is then determined by the following matrix multiplication
[0155]
[0156] The matrix on the right performs the conversion of the microphone frequency point signal to W and Y signals (approximation), and the matrix on the left converts the result to a cardioid signal. The normalization formulated above achieves unity gain in the 90 and -90 degree directions for the cardioid pattern, but not in the opposite direction. Figure 5 The pattern generated according to the above function is illustrated in . The figure also illustrates that the linear method only works for a limited frequency range, and for high frequency ranges, the other methods mentioned above are required.
[0157] The signal y(b,n) formulated for the middle frequency range may then be combined with the y(b,n) previously formulated for the low frequency range and the high frequency range and then provided to the inverse T / F converter 311 .
[0158] In some embodiments, the transmission signal type converter 111 includes an inverse T / F converter 311 . The inverse T / F converter 311 converts y(b,n) 310 to the time domain and outputs it as a modified transmission audio signal 312 .
[0159] about Figure 4 , showing the outline operation of the transmission signal type converter 111.
[0160] like Figure 4 As shown in step 401, a transmission audio signal and metadata are received.
[0161] Then, if Figure 4 As shown in step 403, the transmission audio signal is subjected to time-frequency transformation.
[0162] like Figure 4 As shown in step 402, original and target transmission audio signal types are received.
[0163] Then, if Figure 4 As shown in step 405, a prototype transmission audio signal is created.
[0164] In addition, if Figure 4 As shown in step 409, the prototype transmission audio signal is decorrelated.
[0165] like Figure 4 As shown in step 407, the target signal characteristic is determined.
[0166] Then, if Figure 4 As shown in step 411, the prototype (and decorrelated prototype) signals are mixed based on the determined target signal characteristics.
[0167] Then, if Figure 4 As shown in step 413, the mixed audio signal is subjected to inverse time-frequency transformation.
[0168] Then, if Figure 4 As shown in step 415, the mixed time domain audio signal is output.
[0169] In addition, if Figure 4 As shown in step 417, metadata is output.
[0170] like Figure 4 As shown in step 419 of , the target audio type is output as a new "transmission audio signal type" (because the transmission audio signal has been modified to match this type). In some embodiments, the output transmission audio signal type can be optional (for example, the output stream does not have this field or indicator that identifies the signal type).
[0171] about Figure 6, shows an example apparatus and system for implementing audio capture and rendering (and converting a spatial audio stream of “mono-channel” type into a transmission audio signal of “cardioid” type) according to some embodiments.
[0172] The system 699 is shown with a microphone array audio signal 100 input. In the following example, a microphone array audio signal 100 input is described, but in other embodiments any suitable multi-channel input (or synthesized multi-channel) format may be implemented.
[0173] The system 699 may include a spatial analyzer 101. The spatial analyzer 101 is configured to perform spatial analysis on the microphone signals, thereby generating a transmission audio signal 602 and metadata 104.
[0174] In some embodiments, the spatial analyzer and spatial analysis may be implemented external to the system 699. For example, in some embodiments, spatial metadata associated with the audio signal may be provided to the encoder as a separate bitstream. In some embodiments, the spatial metadata may be provided as a set of spatial (directional) index values.
[0175] The spatial analyzer 101 may be configured to create the transmission audio signal 602 in any suitable manner. For example, in some embodiments, the spatial analyzer is configured to create a single transmission audio signal. This may be useful, for example, when the device has only one high-quality microphone and the other microphones are intended or used only for spatial analysis. In this case, the signal from the high-quality microphone is used as the transmission audio signal (usually after some pre-processing (such as equalization)).
[0176] Metadata can take various forms and can be used in conjunction with Figure 1 The spatial metadata and other metadata are included in the same manner discussed for the illustrated example.
[0177] In some embodiments, the obtained metadata may include metadata in addition to spatial metadata. For example, in some embodiments, the obtained metadata may be a "channel audio format" parameter describing the type of the transmitted audio signal. In this example, the "channel audio format" parameter may have a value of "single channel".
[0178] The transmission audio signal 602 (of type “mono channel”) and the metadata 104 may be output from the spatial analyzer 101 to the encoder 105 .
[0179] In some embodiments, the system 699 includes an encoder 105. The encoder 105 can be configured to receive a transmission audio signal 602 (of type "single channel") and metadata 104 from the spatial analyzer 101. In some embodiments, the encoder 105 can be a mobile device, a user device, a tablet computer, a computer (running suitable software stored on a memory and at least one processor), or (alternatively) a specific device utilizing, for example, an FPGA or an ASIC. The encoder can be configured to implement any suitable encoding scheme. In addition, the encoder 105 can also be configured to receive metadata and generate information in an encoded or compressed form. In some embodiments, the encoder 105 can further interleave, multiplex, or embed metadata into a single data stream 106 or within the encoded audio signal before sending or storing. Multiplexing can be implemented using any suitable scheme.
[0180] The encoder may be an IVAS encoder or any other suitable encoder. Thus, the encoder 105 is configured to encode the audio signal and the metadata and form a bitstream 106 (eg, an IVAS bitstream).
[0181] In addition, the system 699 may also include a decoder 107. The decoder 107 is configured to receive, retrieve or otherwise obtain the bitstream 106, demultiplex the encoded stream from the bitstream, and decode the audio signal to obtain the transmission signal 608 (type "single channel"). Similarly, the decoder 107 can be configured to receive and decode the encoded metadata 110. In some embodiments, the decoder 107 can be a mobile device, a user device, a tablet computer, a computer (running suitable software stored on a memory and at least one processor), or (alternatively) a specific device using, for example, an FPGA or an ASIC.
[0182] The system 699 may further include a transmission signal type converter 111. The transmission signal type converter 111 may be configured to obtain the transmission audio signal 608 (in this example, of type "mono channel") and the metadata 110, and also receive a transmission audio signal type input 118 from the spatial synthesizer 115. The transmission signal type converter 111 may be configured to convert the input transmission signal type to a "target" transmission signal type based on the transmission audio signal type 118 indicator received from the spatial synthesizer 115.
[0183] In this example, the goal is to use the spatial synthesizer 115 to render spatial audio (e.g., two-channel audio) with these signals. However, in this example, the spatial synthesizer 115 only accepts spatial audio streams whose transmission audio signals are of the "cardioid" type. In other words, the spatial synthesizer expects two coincident cardioids pointing at ±90 degrees, for example, and is configured to process any two signal inputs accordingly. Therefore, the spatial audio stream from the decoder cannot be used directly to achieve correct rendering, but a transmission audio signal type converter 111 is used between the decoder 107 and the spatial synthesizer 115.
[0184] In this example, the "target" type is a coincident cardioid pointing at ±90 degrees (this is just an example, it can be any type.) Additionally, if the metadata has a field describing the transmitted audio signal type (e.g., a channel audio format metadata parameter), it can be configured to change that indicator or parameter to indicate the new transmitted audio signal type (e.g., "cardioid").
[0185] The modified transmitted audio signal (eg “cardioid” type) 112 and the (possibly modified) metadata 114 are forwarded to a spatial synthesizer 115 .
[0186] The transmission signal type converter 111 can be used in the same Figure 3 The conversion is performed for all frequencies in the same manner as described for the low and high frequency ranges in the context of . In such an embodiment, the transmission signal type converter 111 is configured to generate a single channel prototype signal and then use the prototype signal to process the converted output. In the context of system 699, the transmission audio signal is already a single channel signal and can be used as a prototype signal, as in Figure 3 The conversion process is performed for all frequencies as described for the low and high frequency ranges in the context of the illustrated example.
[0187] The modified transmitted audio signal (now, of type "cardioid") and the (possibly modified) metadata may then be forwarded to a spatial synthesizer, which uses the spatial audio stream it receives to render spatial audio (eg, binaural audio).
[0188] about Figure 7 , showing example apparatus and systems for implementing audio capture and rendering (and converting a spatial audio stream of “downmix” type into a “cardioid” type transmission audio signal) according to some embodiments.
[0189] The system 799 is shown having a multi-channel audio signal 700 input.
[0190] The system 799 may include a spatial analyzer 101. The spatial analyzer 101 is configured to perform analysis on a multi-channel audio signal, thereby generating a transmission audio signal 702 and metadata 104.
[0191] In some embodiments, the spatial analyzer and spatial analysis may be implemented external to the system 799. For example, in some embodiments, spatial metadata associated with the audio signal may be provided to the encoder as a separate bitstream. In some embodiments, the spatial metadata may be provided as a set of spatial (directional) index values.
[0192] The spatial analyzer 101 may be configured to create the transmission audio signal 702 by downmixing. A simple way to create the transmission audio signal 702 is to use a static downmix matrix for a 5.1 multichannel signal (e.g. and ). In some embodiments, active or adaptive down-mixing may be implemented.
[0193] Metadata can take various forms and can be used in conjunction with Figure 1 The spatial metadata and other metadata are included in the same manner discussed for the illustrated example.
[0194] In some embodiments, the obtained metadata may include metadata in addition to spatial metadata. For example, in some embodiments, the obtained metadata may be a "channel audio format" parameter describing the type of the transmitted audio signal. In this example, the "channel audio format" parameter may have a value of "downmix".
[0195] The transmission audio signal 702 (of type “downmix”) and the metadata 104 may be output from the spatial analyzer 101 to the encoder 105 .
[0196] In some embodiments, the system 799 includes an encoder 105. The encoder 105 can be configured to receive a transmission audio signal 702 (type "downmix") and metadata 104 from the spatial analyzer 101. In some embodiments, the encoder 105 can be a mobile device, a user device, a tablet computer, a computer (running suitable software stored on a memory and at least one processor), or (alternatively) a specific device such as an FPGA or ASIC. The encoder can be configured to implement any suitable encoding scheme. In addition, the encoder 105 can also be configured to receive metadata and generate information in an encoded or compressed form. In some embodiments, the encoder 105 can further interleave, multiplex, or embed metadata into a single data stream 106 or within the encoded audio signal before sending or storing. Multiplexing can be implemented using any suitable scheme.
[0197] The encoder may be an IVAS encoder or any other suitable encoder. Thus, the encoder 105 is configured to encode the audio signal and the metadata and form a bitstream 106 (eg, an IVAS bitstream).
[0198] In addition, the system 799 may also include a decoder 107. The decoder 107 is configured to receive, retrieve or otherwise obtain the bitstream 106, demultiplex the encoded stream from the bitstream, and decode the audio signal to obtain the transmission signal 708 (type "downmix"). Similarly, the decoder 107 can be configured to receive and decode the encoded metadata 110. In some embodiments, the decoder 107 can be a mobile device, a user device, a tablet computer, a computer (running suitable software stored on a memory and at least one processor), or (alternatively) a specific device using, for example, an FPGA or an ASIC.
[0199] The system 799 may further include a transmission signal type converter 111. The transmission signal type converter 111 may be configured to obtain the transmission audio signal 708 (of type "downmix" in this example) and the metadata 110, and also receive a transmission audio signal type input 118 from the spatial synthesizer 115. The transmission signal type converter 111 may be configured to convert the input transmission signal type to a target transmission signal type based on the transmission audio signal type 118 indicator received from the spatial synthesizer 115.
[0200] In this example, the goal is to render spatial audio (eg, two-channel audio) with these signals using the spatial synthesizer 115. However, in this example, the spatial synthesizer 115 only accepts spatial audio streams whose transmitted audio signals are of the "cardioid" type.
[0201] The modified transmitted audio signal (eg “cardioid” type) 112 and the (possibly modified) metadata 114 are forwarded to a spatial synthesizer 115 .
[0202] The transmission signal type converter 111 may implement the conversion by first generating W and Y signals based on the down-mixed audio signal and then mixing them to generate a cardioid output.
[0203] For all frequency points, linear W and Y signal generation is performed. When S1(b,n) and S2(b,n) are the left and right down-mixed T / F signals, the temporary (non-energy normalized) W and Y signals are generated by
[0204] S W (b,n)=S1(b,n)+S2(b,n),
[0205] S Y(b,n)=S1(b,n)-S2(b,n).
[0206] Then, the energy estimates of these signals in the frequency bands are formulated as
[0207] E W (k,n)=∑ b∈k |S W (b,n)| 2 ,
[0208] E Y (k,n)=∑ b∈k |S y (b,n)| 2 .
[0209] Then, the overall energy estimate is also formulated as
[0210] E O (k,n)=∑ b∈k |S1(b,n)| 2 +|S2(b,n)| 2 .
[0211] The converter can then formulate target energies for the W and Y signals.
[0212] T W (k,n)=E O (k,n)
[0213]
[0214] T can then be averaged over appropriate time intervals, for example by using IIR averaging. Y , T W 、E Y and E W Then, the processing matrix for frequency band k is
[0215]
[0216] And the cardioid signal for frequency point b within each frequency band k is processed as
[0217]
[0218] The modified transmitted audio signal (now of type "cardioid") and the (possibly) modified metadata are then forwarded to the spatial synthesizer, which uses the spatial audio stream it receives to render spatial audio (e.g., binaural audio).
[0219] These examples are merely examples, and the converter may be configured to change the transmission audio signal type from a type different from the above to another different type.
[0220] When implementing these embodiments, the following advantages are achieved:
[0221] Using the present invention, by first transforming the transmission audio signal type, an effector (or any other system) that only accepts a specific transmission audio signal type can be used with an audio stream of any transmission audio signal type. In addition, since these embodiments allow flexible transformation of the transmission audio signal type, an original spatial audio stream with any transmission audio signal type can be created and / or stored without worrying about whether it can be used with certain systems later.
[0222] In some embodiments, the input transmission audio signal type may be detected (rather than signaled) in the manner discussed in UK patent application 19042361.3, for example. For example, in some embodiments, the transmission audio signal type converter 111 may be configured to receive or otherwise determine the transmission audio signal type.
[0223] In some embodiments, the transmission audio signal may be a first-order Ambisonic (FOA) signal (with or without spatial metadata). These FOA signals may be converted into another transmission audio signal of the "cardioid" type. The conversion may be performed, for example, according to the following process:
[0224] S1(b,n)=0.5S W (b,n)+0.5S Y (b,n),
[0225] S2(b,n)=0.5S W (b,n)-0.5S Y (b,n)
[0226] about Figure 8 , the example electronic device can be used as any device part of the system described above. The device can be any suitable electronic device or device. For example, in some embodiments, the device 1700 is a mobile device, a user device, a tablet computer, a computer, an audio playback device, etc.
[0227] In some embodiments, device 1700 includes at least one processor or central processing unit 1707. Processor 1707 may be configured to execute various program codes, such as the methods described herein.
[0228] In some embodiments, the device 1700 includes a memory 1711. In some embodiments, at least one processor 1707 is coupled to the memory 1711. The memory 1711 can be any suitable storage device. In some embodiments, the memory 1711 includes a program code portion for storing program codes that can be implemented on the processor 1707. In addition, in some embodiments, the memory 1711 can also include a storage data portion for storing data (e.g., data that has been processed or is to be processed according to the embodiments described herein). Whenever necessary, the implemented program code stored in the program code portion and the data stored in the storage data portion can be obtained by the processor 1707 via the memory-processor coupling.
[0229] In some embodiments, the device 1700 includes a user interface 1705. In some embodiments, the user interface 1705 may be coupled to a processor 1707. In some embodiments, the processor 1707 may control the operation of the user interface 1705 and receive input from the user interface 1705. In some embodiments, the user interface 1705 may enable a user to enter commands to the device 1700, for example, via a keypad. In some embodiments, the user interface 1705 may enable a user to obtain information from the device 1700. For example, the user interface 1705 may include a display configured to display information from the device 1700 to the user. In some embodiments, the user interface 1705 may include a touch screen or touch interface that enables information to be input into the device 1700 and to display information to a user of the device 1700. In some embodiments, the user interface 1705 may be a user interface for communication.
[0230] In some embodiments, the device 1700 includes an input / output port 1709. In some embodiments, the input / output port 1709 includes a transceiver. In such embodiments, the transceiver can be coupled to the processor 1707 and configured to communicate with other devices or electronic devices, for example, via a wireless communication network. In some embodiments, the transceiver or any suitable transceiver or transmitter and / or receiver components can be configured to communicate with other electronic devices or devices via wired or wired coupling.
[0231] The transceiver can communicate with other devices via any suitable known communication protocol. For example, in some embodiments, the transceiver can use an appropriate Universal Mobile Telecommunications System (UMTS) protocol, a wireless local area network (WLAN) protocol such as IEEE 802.X, a suitable short-range radio frequency communication protocol such as Bluetooth, or an infrared data communication path (IRDA).
[0232] The transceiver input / output port 1709 may be configured to receive signals.
[0233] In some embodiments, device 1700 can be used as at least part of a composite device. Input / output port 1709 can be coupled to any suitable audio output, such as a multi-channel speaker system and / or headphones (which can be head-mounted or non-head-mounted), etc.
[0234] In general, various embodiments of the present invention may be implemented in hardware or dedicated circuits, software, logic, or any combination thereof. For example, some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device, but the present invention is not limited thereto. Although various aspects of the present invention may be illustrated and described as block diagrams, flow charts, or using some other graphical representation, it is well known that the blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general hardware or controllers or other computing devices, or some combination thereof, as non-limiting examples.
[0235] Embodiments of the present invention can be implemented by computer software that can be executed by a data processor of a mobile device (such as in a processor entity), or by hardware, or by a combination of software and hardware. In addition, it should be noted that any block of the logic flow as in the accompanying drawings can represent program steps, or interconnected logic circuits, blocks and functions, or a combination of program steps and logic circuits, blocks and functions. The software can be stored on a physical medium such as a memory chip or a memory block implemented in a processor, on a magnetic medium such as a hard disk or a floppy disk, and on an optical medium such as a DVD and its data variant CD.
[0236] The memory may be of any type suitable for the local technical environment and may be implemented using any appropriate data storage technology, such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory, and removable memory. The data processor may be of any type suitable for the local technical environment and may include, as non-limiting examples, one or more of a general purpose computer, a special purpose computer, a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a gate level circuit based on a multi-core processor architecture, and a processor.
[0237] Embodiments of the present invention may be practiced in various components such as integrated circuit modules. The design of integrated circuits is generally a highly automated process. Complex and powerful software tools are available to convert a logic level design into a semiconductor circuit design ready to be etched and formed on a semiconductor substrate.
[0238] Programs, such as those offered by Synopsys, Inc. of Mountain View, Calif., and Cadence Design, of San Jose, Calif., can automatically route conductors and position components on a semiconductor chip using well-established design rules and a library of pre-stored design modules. Once the design of a semiconductor circuit is complete, the resulting design in a standardized electronic format (e.g., Opus, GDSII, etc.) can be transmitted to a semiconductor manufacturing facility or "fab" for fabrication.
[0239] The foregoing description has provided a complete and useful description of exemplary embodiments of the present invention by way of exemplary and non-limiting examples. However, various modifications and adaptations will become apparent to those skilled in the relevant art in view of the above description when read in conjunction with the accompanying drawings and the appended claims. However, all such and similar modifications of the teachings of the present invention will still fall within the scope of the present invention as defined by the appended claims.
Claims
1. An apparatus for spatial audio representation, comprising at least one processor and at least one memory including computer program code, the at least one memory and the computer program code being configured to, with the at least one processor, cause the apparatus to at least: obtaining at least one audio stream, wherein the at least one audio stream comprises one or more transport audio signals, wherein the one or more transport audio signals are transport audio signals of a first type in a format; obtaining an indicator representing a second type of transmitted audio signal in the format, wherein: The indicator is obtained from a synthesizer configured to perform rendering to a user, and wherein the second type comprises a target transport audio signal type and is different from the first type; and Based at least in part on the indicator, the one or more transmit audio signals are processed to generate one or more other transmit audio signals, the one or more other transmit audio signals being of the target transmit audio signal type.
2. The device according to claim 1, wherein: At least one of the first type or the second type is associated with at least one of: the origin of the one or more transmitted audio signals; or The one or more transmitted audio signals are of analog origin.
3. The device according to claim 1, wherein: The apparatus is further configured to process the one or more transmission audio signals to generate one or more other transmission audio signals: Based on the indicator, the one or more transmission audio signals are converted into the one or more other transmission audio signals.
4. The device according to claim 3, wherein: The indicator is obtained from a renderer configured to receive the one or more other transmission audio signals and to render the one or more other transmission audio signals.
5. The apparatus according to claim 1, further configured to: providing the one or more other transmitted audio signals for rendering; generating an indicator associated with the second type; and The indicator is provided as additional metadata together with the one or more other transmission audio signals for the rendering. The apparatus according to claim 1 , further configured to: determine the first type.
7. The device according to claim 6, wherein: The at least one audio stream further comprises an indicator identifying the first type, and wherein the apparatus is configured to determine the first type based on the indicator. 8 . The apparatus of claim 6 , further configured to determine the first type based on an analysis of the one or more transmission audio signals.
9. The device according to claim 1, wherein: Processing the one or more transmission audio signals of the at least one audio stream comprises the at least one memory and the computer program code being further configured to, with the at least one processor, cause the apparatus to: generating at least one prototype signal based on the one or more transmitted audio signals of the at least one audio stream, the first type, and the second type; determining at least one transmit audio signal characteristic of the one or more other transmit audio signals; as well as The at least one prototype signal and a decorrelated version of the at least one prototype signal are mixed based on the determined at least one transmit audio signal characteristic to generate the one or more other transmit audio signals.
10. The device according to claim 1, wherein: The first type is at least one of the following: capture microphone placement; Capture microphone separation distance; Capture microphone parameters; transmission channel identifier; Cardioid audio signal type; Interval audio signal type; Downmix audio signal type; Coincident audio signal type; Transmission channel layout.
11. The device according to claim 1, wherein: The at least one memory and the computer program code are further configured to, with the at least one processor, cause the apparatus to perform one of the following: converting the one or more other transmitted audio signals into a panoramic surround sound audio signal representation for rendering; converting the one or more other transmitted audio signals into a two-channel audio signal representation for rendering; and The one or more other transmitted audio signals are converted into a multi-channel audio signal representation for rendering.
12. The device according to claim 1, wherein: The at least one audio stream includes spatial metadata associated with the one or more transmitted audio signals.
13. The apparatus according to claim 12, further configured to provide the one or more other transmission audio signals and the spatial metadata associated with the one or more transmission audio signals for rendering, wherein The apparatus comprises the synthesizer.
14. A method for spatial audio representation, comprising: obtaining at least one audio stream, wherein the at least one audio stream comprises one or more transport audio signals, wherein the one or more transport audio signals are transport audio signals of a first type in a format; obtaining an indicator representing a second type of transmission audio signal in the format, wherein the indicator is obtained from a synthesizer configured to perform rendering to a user, and wherein the second type comprises a target transmission audio signal type and is different from the first type; and Based at least in part on the indicator, the one or more transmit audio signals are processed to generate one or more other transmit audio signals, the one or more other transmit audio signals being of the target transmit audio signal type.
15. The method according to claim 14, wherein: At least one of the first type or the second type is associated with at least one of: the origin of the one or more transmitted audio signals; or The one or more transmitted audio signals are of analog origin.
16. The method according to claim 14, wherein: Processing the one or more transmission audio signals to generate one or more other transmission audio signals further comprises: Based on the indicator, the one or more transmission audio signals are converted into the one or more other transmission audio signals.
17. The method according to claim 16, wherein: The indicator is obtained from a renderer configured to receive the one or more other transmission audio signals and to render the one or more other transmission audio signals.
18. The method according to claim 14, further comprising: The one or more other transmit audio signals are provided for rendering.
19. The method according to claim 14, wherein: Processing the one or more transmission audio signals further comprises: generating at least one prototype signal based on the one or more transmitted audio signals of the at least one audio stream, the first type, and the second type; determining at least one transmit audio signal characteristic of the one or more other transmit audio signals; and The at least one prototype signal and a decorrelated version of the at least one prototype signal are mixed based on the determined at least one transmit audio signal characteristic to generate the one or more other transmit audio signals.
20. The method according to claim 14, wherein: The at least one audio stream comprises spatial metadata associated with the one or more transport audio signals, and the method further comprises providing the one or more other transport audio signals and the spatial metadata associated with the one or more transport audio signals for rendering.
