Spatial audio parameter merging

Abstract

An apparatus performs the following: determining at least one metadata parameter for at least one first audio signal having an audio signal format; determining at least one other metadata parameter for at least one other audio signal having another audio signal format; a combination of the at least one metadata parameter and the at least one other metadata parameter is controlled to generate combined metadata, where the combined metadata is configured to be associated with a combined audio signal formed from the at least one first audio signal and the at least one other audio signal, such that the combined metadata includes at least one spatial audio parameter.

Description

[0001] This application is a divisional application of the Chinese invention patent application with the invention name “Spatial Audio Parameter Merger” (application number 201980050110.X, application date May 29, 2019). Technical Field

[0002] The present application relates to an apparatus and method for estimating sound field related parameters in a frequency band, but does not exclusively relate to an apparatus and method for estimating sound field related parameters in the time-frequency domain for an audio encoder and decoder. Background Art

[0003] Parametric spatial audio processing is a field of audio signal processing in which a set of parameters is used to describe the spatial aspects of sound. For example, in parametric spatial audio capture from a microphone array, estimating a set of parameters from the microphone array signal, such as the direction of the sound in the frequency band, and the ratio of the directional to non-directional parts of the captured sound in the frequency band, is a typical and effective choice. It is well known that these parameters describe the perceived spatial characteristics of the captured sound at the position of the microphone array. These parameters can be used accordingly in the synthesis of spatial sound for binaural headphones, speakers, or other formats such as Ambisonics.

[0004] Therefore, the direction and direct-to-total energy ratio in a frequency band is a particularly effective parameterization for describing spatial audio. Summary of the invention

[0005] According to a first aspect, an apparatus is provided, comprising means for determining, for at least one first audio signal having an audio signal format, at least one metadata parameter; for at least one other audio signal having another audio signal format, at least one other metadata parameter; controlling a combination of the at least one metadata parameter with the at least one other metadata parameter to generate combined metadata, wherein the combined metadata is configured to be associated with a combined audio signal formed from the at least one first audio signal and the at least one other audio signal, such that the combined metadata comprises at least one spatial audio parameter.

[0006] The means for determining at least one metadata parameter for at least one first / other audio signal may be configured to perform at least one of: receiving at least one metadata parameter associated with at least one first / other audio signal; analyzing at least one first / other audio signal to determine at least one metadata parameter; and decoding at least one first / other audio signal to determine at least one metadata parameter.

[0007] The component for controlling the combination of at least one metadata parameter associated with a first audio signal and at least one metadata parameter associated with other audio signals to generate combined metadata can be used to: extract at least one metadata parameter including at least one spatial audio parameter as a metadata block, which metadata parameter is associated with at least one first / other audio signal; add the extracted at least one metadata parameter including at least one spatial audio parameter as an auxiliary metadata block to the metadata associated with at least one other / first audio signal, respectively, which metadata parameter is associated with at least one first or other audio signal.

[0008] The means for extracting as a metadata block at least one metadata parameter including at least one spatial audio parameter, which metadata parameter is associated with at least one first / other audio signal, may be used to extract as a metadata block at least one of the following: at least one directional parameter; at least one energy ratio parameter; and at least one coherence parameter associated with at least one first / other audio signal, and wherein the means for adding as an auxiliary metadata block the extracted at least one metadata parameter including at least one spatial audio parameter to metadata respectively associated with at least one other / first audio signal, which metadata parameter is associated with the first / other audio signal, may be used to append as an auxiliary metadata block at least one of the following to metadata associated with at least one other / first audio signal: at least one directional parameter; at least one energy ratio parameter; and at least one coherence parameter associated with at least one first / other audio signal, the metadata associated with at least one other / first audio signal may also include: at least one master metadata block, which includes at least one spatial audio parameter associated with at least one other / first audio signal, and the at least one spatial audio parameter associated with at least one other / first audio signal includes at least one of the following: at least one directional parameter; at least one energy ratio parameter; and at least one coherence parameter associated with the other / first audio signal; and a common metadata block, which is related to at least one other / first audio signal and includes at least one non-spatial audio-related parameter, and the at least one non-spatial audio-related parameter includes at least one of the following: a version identifier; a time-frequency resolution identifier; multiple directional identifiers.

[0009] The component for controlling the combination of at least one metadata parameter and at least one other metadata parameter to generate combined metadata can be used to: determine at least one first signal ratio associated with at least one metadata parameter associated with at least one first audio signal; generate at least one first signal weight based on the at least one first signal ratio; determine at least one other signal ratio associated with at least one metadata parameter associated with at least one other audio signal; generate at least one other signal weight based on the at least one other signal ratio; compare the at least one first signal weight and the at least one other signal weight; and generate the combined metadata based on the comparison of the at least one first signal weight and the at least one other signal weight.

[0010] The component for generating combined metadata based on comparing at least one first signal weight and at least one other signal weight can be used to: when the comparison indicates that the at least one first signal weight is greater than the at least one other signal weight by a certain threshold, use at least one metadata parameter associated with the at least one first audio signal as the combined metadata; when the comparison indicates that the at least one other signal weight is greater than the at least one first signal weight by another certain threshold, use at least one metadata parameter associated with the at least one other audio signal as the combined metadata; when the comparison indicates that the at least one other signal weight is greater than the at least one first signal weight by another certain threshold, generate a weighted average of the at least one metadata parameter associated with the at least one first audio signal and the at least one metadata parameter associated with the at least one other audio signal when the comparison indicates otherwise.

[0011] The component for controlling the combination of at least one metadata parameter associated with at least one first audio signal and at least one metadata parameter associated with at least one other audio signal to generate combined metadata can also be used to: determine at least one first signal energy associated with at least one metadata parameter associated with at least one first audio signal; determine at least one other signal energy associated with at least one metadata parameter associated with at least one other audio signal, wherein the component for generating at least one first signal weight based on at least one first signal ratio is further based on the at least one first signal energy, wherein the component for generating at least one other signal weight based on at least one other signal ratio is further based on the at least one other signal energy.

[0012] The component for controlling the combination of at least one metadata parameter associated with at least one first audio signal and at least one metadata parameter associated with at least one other audio signal to generate combined metadata can also be used to: determine at least one first signal user input associated with at least one metadata parameter associated with at least one first audio signal; determine at least one other signal user input associated with at least one metadata parameter associated with at least one other audio signal, wherein the component for generating at least one first signal weight based on at least one first signal ratio can be further based on at least one first user input energy, wherein the component for generating at least one other signal weight based on at least one other signal ratio can be further based on at least one other signal user input.

[0013] The component for controlling the combination of at least one metadata parameter associated with at least one first audio signal and at least one metadata parameter associated with at least one other audio signal to generate combined metadata can also be used to: determine at least one first signal server input associated with at least one metadata parameter associated with at least one first audio signal; determine at least one other signal server input associated with at least one metadata parameter associated with at least one other audio signal, wherein the component for generating at least one first signal weight based on at least one first signal ratio is further based on at least one first server input energy, wherein the component for generating at least one other signal weight based on at least one other signal ratio can be further based on at least one other signal server input.

[0014] The means may also be configured to combine at least one metadata parameter associated with at least one first audio signal and at least one metadata parameter associated with at least one other audio signal to generate combined metadata.

[0015] At least one first audio signal having an audio signal format may be at least one of: 2-N channels of a spatial microphone array; 2-N channels of a multi-channel audio signal; a first-order panoramic surround sound signal; a higher-order panoramic surround sound signal; and a spatial audio signal.

[0016] At least one other audio signal having another audio signal format may be at least one of: 2-N channels of a spatial microphone array; 2-N channels of a multi-channel audio signal; a first-order panoramic surround sound signal; a higher-order panoramic surround sound signal; and a spatial audio signal.

[0017] The means for controlling the combination of at least one metadata parameter with at least one other metadata parameter to generate combined metadata may be such that the combined metadata comprises at least one spatial audio parameter and at least one non-spatial audio related parameter.

[0018] According to a second aspect, an apparatus is provided, comprising at least one processor and at least one memory including computer program code, the at least one memory and the computer program code being configured to, together with the at least one processor, cause the apparatus to at least: determine at least one metadata parameter for at least one first audio signal having an audio signal format; determine at least one other metadata parameter for at least one other audio signal having another audio signal format; control a combination of the at least one metadata parameter with the at least one other metadata parameter to generate combined metadata, wherein the combined metadata is configured to be associated with a combined audio signal formed from the at least one first audio signal and the at least one other audio signal so that the combined metadata includes at least one spatial audio parameter.

[0019] The apparatus caused to determine at least one metadata parameter for at least one first / other audio signal may be further caused to perform at least one of: receiving at least one metadata parameter associated with the at least one first / other audio signal; analyzing the at least one first / other audio signal to determine the at least one metadata parameter; and decoding the at least one first / other audio signal to determine the at least one metadata parameter.

[0020] The device, which is caused to control the combination of at least one metadata parameter associated with the first audio signal and at least one metadata parameter associated with the other audio signals to generate combined metadata, can be further caused to: extract at least one metadata parameter including at least one spatial audio parameter as a metadata block, which metadata parameter is associated with at least one first / other audio signal; add the extracted at least one metadata parameter including at least one spatial audio parameter as an auxiliary metadata block to the metadata associated with at least one other / first audio signal, respectively, which metadata parameter is associated with at least one first or other audio signal.

[0021] The device, which is caused to extract at least one metadata parameter including at least one spatial audio parameter as a metadata block, which metadata parameter is associated with at least one first / other audio signal, may be caused to extract at least one of the following as a metadata block: at least one directional parameter; at least one energy ratio parameter; and at least one coherence parameter associated with at least one first / other audio signal, and wherein the device, which is caused to add the extracted at least one metadata parameter including at least one spatial audio parameter as an auxiliary metadata block to metadata respectively associated with at least one other / first audio signal, which metadata parameter is associated with the first / other audio signal, may further be caused to append at least one of the following as an auxiliary metadata block to metadata associated with at least one other / first audio signal: at least one directional parameter; to at least one energy ratio parameter; and at least one coherence parameter associated with at least one first / other audio signal, the metadata associated with at least one other / first audio signal may also include: at least one master metadata block, which includes at least one spatial audio parameter associated with at least one other / first audio signal, and the at least one spatial audio parameter associated with at least one other / first audio signal may include at least one of the following: at least one directional parameter; at least one energy ratio parameter; and at least one coherence parameter associated with the other / first audio signal; and a common metadata block, which is related to at least one other / first audio signal and may include at least one non-spatial audio-related parameter, and the at least one non-spatial audio-related parameter includes at least one of the following: a version identifier; a time-frequency resolution identifier; multiple directional identifiers.

[0022] The device, which is caused to control the combination of at least one metadata parameter with at least one other metadata parameter to generate combined metadata, may be caused to: determine at least one first signal ratio associated with at least one metadata parameter associated with at least one first audio signal; generate at least one first signal weight based on the at least one first signal ratio; determine at least one other signal ratio associated with at least one metadata parameter associated with at least one other audio signal; generate at least one other signal weight based on the at least one other signal ratio; compare the at least one first signal weight and the at least one other signal weight; and generate the combined metadata based on the comparison of the at least one first signal weight and the at least one other signal weight.

[0023] The apparatus caused to generate combined metadata based on comparing at least one first signal weight and at least one other signal weight may be caused to: use at least one metadata parameter associated with at least one first audio signal as the combined metadata when the comparison indicates that the at least one first signal weight is greater than the at least one other signal weight by a certain threshold; use at least one metadata parameter associated with at least one other audio signal as the combined metadata when the comparison indicates that the at least one other signal weight is greater than the at least one first signal weight by another certain threshold; and generate a weighted average of at least one metadata parameter associated with the at least one first audio signal and at least one metadata parameter associated with the at least one other audio signal when the comparison indicates otherwise.

[0024] The device configured to control a combination of at least one metadata parameter associated with at least one first audio signal and at least one metadata parameter associated with at least one other audio signal to generate combined metadata may be further configured to: determine at least one first signal energy associated with at least one metadata parameter associated with at least one first audio signal; determine at least one other signal energy associated with at least one metadata parameter associated with at least one other audio signal, wherein the device configured to generate at least one first signal weight based on at least one first signal ratio may be further based on the at least one first signal energy, wherein the device configured to generate at least one other signal weight based on at least one other signal ratio may be further based on the at least one other signal energy.

[0025] The device configured to control a combination of at least one metadata parameter associated with at least one first audio signal and at least one metadata parameter associated with other audio signals to generate combined metadata may be further configured to: determine at least one first signal user input associated with at least one metadata parameter associated with at least one first audio signal; determine at least one other signal user input associated with at least one metadata parameter associated with at least one other audio signal, wherein the device configured to generate at least one first signal weight based on at least one first signal ratio may be further based on at least one first user input energy, wherein the device configured to generate at least one other signal weight based on at least one other signal ratio may be further based on at least one other signal user input.

[0026] The device configured to control a combination of at least one metadata parameter associated with at least one first audio signal and at least one metadata parameter associated with at least one other audio signal to generate combined metadata may be further configured to: determine at least one first signal server input associated with at least one metadata parameter associated with at least one first audio signal; determine at least one other signal server input associated with at least one metadata parameter associated with at least one other audio signal, wherein the device configured to generate at least one first signal weight based on at least one first signal ratio may be further based on at least one first server input energy, wherein the device configured to generate at least one other signal weight based on at least one other signal ratio may be further based on at least one other signal server input.

[0027] The apparatus may further be caused to combine at least one metadata parameter associated with the at least one first audio signal and at least one metadata parameter associated with at least one other audio signal to produce combined metadata.

[0028] At least one first audio signal having an audio signal format may be at least one of: 2-N channels of a spatial microphone array; 2-N channels of a multi-channel audio signal; a first-order panoramic surround sound signal; a higher-order panoramic surround sound signal; and a spatial audio signal.

[0029] At least one other audio signal having another frequency signal format may be at least one of: 2-N channels of a spatial microphone array; 2-N channels of a multi-channel audio signal; a first-order panoramic surround sound signal; a higher-order panoramic surround sound signal; and a spatial audio signal.

[0030] The apparatus being caused to control a combination of at least one metadata parameter with at least one other metadata parameter to generate combined metadata may cause the combined metadata to include at least one spatial audio parameter and at least one non-spatial audio related parameter.

[0031] According to a third aspect, a method is provided, comprising: determining at least one metadata parameter for at least one first audio signal having an audio signal format; determining at least one other metadata parameter for at least one other audio signal having another audio signal format; controlling a combination of the at least one metadata parameter and the at least one other metadata parameter to generate combined metadata, wherein the combined metadata is configured to be associated with a combined audio signal formed from the at least one first audio signal and the at least one other audio signal so that the combined metadata includes at least one spatial audio parameter.

[0032] Determining at least one metadata parameter for at least one first / other audio signal may further include at least one of: receiving at least one metadata parameter associated with at least one first / other audio signal; analyzing at least one first / other audio signal to determine the at least one metadata parameter; and decoding at least one first / other audio signal to determine the at least one metadata parameter.

[0033] Controlling the combination of at least one metadata parameter associated with the first audio signal and at least one metadata parameter associated with the other audio signals to generate combined metadata may also include: extracting at least one metadata parameter including at least one spatial audio parameter as a metadata block, which metadata parameter is associated with at least one first / other audio signal; adding the extracted at least one metadata parameter including at least one spatial audio parameter as an auxiliary metadata block to the metadata associated with at least one other / first audio signal, respectively, which metadata parameter is associated with at least one first or other audio signal.

[0034] Extracting at least one metadata parameter including at least one spatial audio parameter as a metadata block, which metadata parameter is associated with at least one first / other audio signal, may also include extracting at least one of the following as a metadata block: at least one directional parameter; at least one energy ratio parameter; and at least one coherence parameter associated with at least one first / other audio signal, and wherein the extracted at least one metadata parameter including at least one spatial audio parameter is added as an auxiliary metadata block to metadata associated with at least one other / first audio signal, respectively, which metadata parameter is associated with the first / other audio signal, may also include appending at least one of the following as an auxiliary metadata block to metadata associated with at least one other / first audio signal: at least one directional parameter; at least one energy ratio parameter number; and at least one coherence parameter associated with at least one first / other audio signal, the metadata associated with at least one other / first audio signal may also include: at least one master metadata block, which includes at least one spatial audio parameter associated with at least one other / first audio signal, and the at least one spatial audio parameter associated with at least one other / first audio signal includes at least one of the following: at least one direction parameter; at least one energy ratio parameter; and at least one coherence parameter associated with the other / first audio signal; and a common metadata block, which is related to at least one other / first audio signal and includes at least one non-spatial audio-related parameter, and the at least one non-spatial audio-related parameter includes at least one of the following: a version identifier; a time-frequency resolution identifier; multiple direction identifiers.

[0035] Controlling the combination of at least one metadata parameter and at least one other metadata parameter to generate combined metadata may also include: determining at least one first signal ratio associated with at least one metadata parameter associated with at least one first audio signal; generating at least one first signal weight based on the at least one first signal ratio; determining at least one other signal ratio associated with at least one metadata parameter associated with at least one other audio signal; generating at least one other signal weight based on the at least one other signal ratio; comparing the at least one first signal weight and the at least one other signal weight; and generating the combined metadata based on the comparison of the at least one first signal weight and the at least one other signal weight.

[0036] Generating combined metadata based on comparing at least one first signal weight and at least one other signal weight may also include: when the comparison indicates that the at least one first signal weight is greater than the at least one other signal weight by a certain threshold, using at least one metadata parameter associated with the at least one first audio signal as the combined metadata; when the comparison indicates that the at least one other signal weight is greater than the at least one first signal weight by another certain threshold, using at least one metadata parameter associated with the at least one other audio signal as the combined metadata; when the comparison indicates otherwise, generating a weighted average of at least one metadata parameter associated with the at least one first audio signal and at least one metadata parameter associated with the at least one other audio signal.

[0037] Controlling the combination of at least one metadata parameter associated with at least one first audio signal and at least one metadata parameter associated with at least one other audio signal to generate combined metadata may also include: determining at least one first signal energy associated with at least one metadata parameter associated with at least one first audio signal; determining at least one other signal energy associated with at least one metadata parameter associated with at least one other audio signal, wherein generating at least one first signal weight based on at least one first signal ratio may be further based on at least one first signal energy, wherein generating at least one other signal weight based on at least one other signal ratio may be further based on at least one other signal energy.

[0038] Controlling the combination of at least one metadata parameter associated with at least one first audio signal and at least one metadata parameter associated with other audio signals to generate combined metadata may also include: determining at least one first signal user input associated with at least one metadata parameter associated with at least one first audio signal; determining at least one other signal user input associated with at least one metadata parameter associated with at least one other audio signal, wherein generating at least one first signal weight based on at least one first signal ratio may be further based on at least one first user input energy, wherein generating at least one other signal weight based on at least one other signal ratio may be further based on at least one other signal user input.

[0039] Controlling the combination of at least one metadata parameter associated with at least one first audio signal and at least one metadata parameter associated with at least one other audio signal to generate combined metadata may also include: determining at least one first signal server input associated with at least one metadata parameter associated with at least one first audio signal; determining at least one other signal server input associated with at least one metadata parameter associated with at least one other audio signal, wherein generating at least one first signal weight based on at least one first signal ratio may be further based on at least one first server input energy, wherein generating at least one other signal weight based on at least one other signal ratio may be further based on at least one other signal server input.

[0040] The method may further comprise combining at least one metadata parameter associated with the at least one first audio signal and at least one metadata parameter associated with the at least one other audio signal to produce combined metadata.

[0041] At least one first audio signal having an audio signal format may be at least one of: 2-N channels of a spatial microphone array; 2-N channels of a multi-channel audio signal; a first-order panoramic surround sound signal; a higher-order panoramic surround sound signal; and a spatial audio signal.

[0042] At least one other audio signal having another audio signal format may be at least one of: 2-N channels of a spatial microphone array; 2-N channels of a multi-channel audio signal; a first-order panoramic surround sound signal; a higher-order panoramic surround sound signal; and a spatial audio signal.

[0043] Controlling the combination of the at least one metadata parameter with the at least one other metadata parameter to generate the combined metadata may cause the combined metadata to include at least one spatial audio parameter and at least one non-spatial audio related parameter.

[0044] According to a fourth aspect, there is provided a computer program (or a computer-readable medium) comprising instructions, the instructions being used to cause an apparatus to perform at least the following operations: determining at least one metadata parameter for at least one first audio signal having an audio signal format; determining at least one other metadata parameter for at least one other audio signal having another audio signal format; controlling a combination of at least one metadata parameter with at least one other metadata parameter to generate combined metadata, wherein the combined metadata is configured to be associated with a combined audio signal formed from at least one first audio signal and at least one other audio signal so that the combined metadata includes at least one spatial audio parameter.

[0045] According to a fifth aspect, a non-transitory computer-readable medium is provided, comprising program instructions for causing an apparatus to perform at least the following operations: determining at least one metadata parameter for at least one first audio signal having an audio signal format; determining at least one other metadata parameter for at least one other audio signal having another audio signal format; controlling a combination of at least one metadata parameter with at least one other metadata parameter to generate combined metadata, wherein the combined metadata is configured to be associated with a combined audio signal formed from at least one first audio signal and at least one other audio signal so that the combined metadata includes at least one spatial audio parameter.

[0046] According to a sixth aspect, a device is provided, comprising: a first metadata determination circuit, configured to determine at least one metadata parameter for at least one first audio signal having an audio signal format; another metadata determination circuit, configured to determine at least one other metadata parameter for at least one other audio signal having another audio signal format; and a control circuit, configured to control a combination of the at least one metadata parameter and at least one other metadata parameter to generate combined metadata, wherein the combined metadata is configured to be associated with a combined audio signal formed from the at least one first audio signal and the at least one other audio signal so that the combined metadata includes at least one spatial audio parameter.

[0047] According to a seventh aspect, a computer-readable medium comprising program instructions is provided, the program instructions being used to cause an apparatus to perform at least the following operations: determining at least one metadata parameter for at least one first audio signal having an audio signal format; determining at least one other metadata parameter for at least one other audio signal having another audio signal format; controlling a combination of at least one metadata parameter with at least one other metadata parameter to generate combined metadata, wherein the combined metadata is configured to be associated with a combined audio signal formed from at least one first audio signal and at least one other audio signal so that the combined metadata includes at least one spatial audio parameter.

[0048] An apparatus comprises means for performing the actions of the method as described above.

[0049] A device is configured to perform the actions of the method described above.

[0050] A computer program comprises program instructions for causing a computer to execute the method described above.

[0051] A computer program product stored on a medium can cause an apparatus to execute the method as described herein.

[0052] An electronic device may include the apparatus described herein.

[0053] A chipset may include the device described herein.

[0054] The embodiments of the present application are intended to solve the problems associated with the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] For a better understanding of the present application, reference will now be made by way of example to the accompanying drawings, in which:

[0056] Figure 1 schematically illustrates an exemplary combination scenario according to some embodiments;

[0057] Figure 2 schematically illustrates an exemplary pre-processor having a mixer according to some embodiments;

[0058] Figure 3 shows an exemplary processor following combination according to some embodiments;

[0059] Figure 4 schematically illustrates a first example of a mixer according to some embodiments;

[0060] Figure 5 a flow chart illustrating the operation of a first example of a mixer according to some embodiments;

[0061] Figure 6 a flow chart illustrating the operation of a second example of a mixer according to some embodiments;

[0062] Figure 7 a flow chart illustrating the operation of a second example of a mixer according to some embodiments;

[0063] Figure 8 schematically illustrates an implementation of a mixer within a conference bridge example according to some embodiments;

[0064] Fig. 9 illustrates an exemplary user input device control user interface according to some embodiments;

[0065] Fig.10 Exemplary devices suitable for implementing the means described herein are schematically shown. DETAILED DESCRIPTION

[0066] Suitable apparatus and possible mechanisms for providing efficient merging or combining of spatial metadata parameters are described in more detail below.

[0067] As discussed previously, parametric spatial audio can be used in various forms of telecommunications (e.g., this can include one-to-one immersive calls or telepresence, multi-user teleconferencing, etc.). In these contexts, it often happens that there are multiple parametric spatial audio streams. In other words, in the same client or server node, there can be multiple audio and metadata streams with multiple sources (e.g., spatial microphone grids, premixed 5.1, etc.). However, it is generally desirable to use a minimum bit rate in audio transmission. This is especially the case in mobile communications where the transmission channel can sometimes become very limited and therefore efficient transmission is very important.

[0068] A concept discussed in more detail below in embodiments is to combine the streams (by mixing or some other combining method) before transmission. Embodiments can achieve this without converting the signals into a common non-parametric audio format (e.g., B-format), and are therefore computationally simpler and avoid any quality degradation through repeated synthesis and analysis operations. Therefore, embodiments below attempt to produce a direct mix or combination of parametric audio formats.

[0069] Thus, embodiments describe apparatus and methods for combining multiple streams (e.g., multi-microphone capture of spatial audio and channel-based audio) in the parameter domain. In some embodiments, the combination may be implemented as part of a capture processing apparatus / operation and thereby create a combined audio representation that may be used, for example, as an input to an audio codec. In some embodiments, the combination may be performed internally to the audio codec.

[0070] In order to achieve efficient codec operation, in some embodiments, the apparatus and operations are implemented outside the codec, thereby generating a single input stream. This has the advantage of reducing the number of streams to be encoded, which can reduce the associated codec mode selection and compression complexity.

[0071] In these embodiments, the spatial parameters of the audio are first analyzed and a transmission signal is generated (in other words, the audio signal is converted into a suitable parameter format). The resulting parameterized signals are then combined. This can be based on associated bit rate requirements. An exemplary implementation that can use a higher bit rate is to combine the transmission signals and add new spatial metadata blocks to the spatial metadata or modify the spatial metadata by adding, for example, a second analysis direction. Alternatively, in some embodiments, a bit rate reduction option is to fully combine the spatial metadata from the two sources in addition to combining the transmission signals.

[0072] In some embodiments, a user and / or a hybrid (combining) service (eg, on a device, on a conference bridge, or in a network) may generate weights and apply them to the combination for a particular flow.

[0073] about Figure 1 , schematically illustrates an exemplary combination scenario according to some embodiments. In this example, combination is performed for an interactive call, where the call is initialized first and then the capture begins, but in some embodiments, the capture and / or audio background signal is started before the call is made, or in a pure recording scenario, the capture and audio selection and combination are performed independently of the call. For example, some embodiments can be implemented in a capture pre-processing system before encoding.

[0074] In this example, under the control of the user, the device is configured to initiate or negotiate a call, such as in Figure 1 In the example of negotiating a call (or implementing only video or audio capture at one end) is shown in step 101. Figure 1 , which is shown in the lower left image portion, which shows user 153 operating a mobile device and attempting to capture another user 151 using a camera and microphone.

[0075] The operation of capturing the first audio is Figure 1 This is shown by step 103.

[0076] Furthermore, in some embodiments, as in Figure 1 As shown in step 105 , the first audio is then processed to generate a suitable transmission audio signal and spatial metadata.

[0077] The processed transmit audio signal and spatial metadata may then be passed to a "direct" mixer, as will be discussed in further detail below.

[0078] When a call is being initialized / negotiated or indeed during a call (or more generally before or during a capture of a video-only or audio-only capture), the user 153 may be able to configure the device to initialize immersive audio enhancement. Figure 1 In some embodiments, this can be activated based on interaction with a user interface element (such as a displayed icon) to initiate a conventional mono call and an immersive call. In addition, in some embodiments, for example on a 3GPP device, an immersive audio streaming / sharing service can be provided. In such an embodiment, a user can initiate an immersive call. The user initiating immersive sharing can use a defined streaming service that has at least immersive audio (for example, the service can also have traditional video or VR video). Then, call negotiation 101 is performed.

[0079] In some embodiments, the audio capture 103 may be captured before the call is initiated and not sent to anyone until the call is established.

[0080] Adding background music 112 (or any other relevant audio enhancement) can then be implemented at any time during the call via the immersive audio call / enhancement application. Thus, the call can start with added background music, or the user can, for example, add 5 minutes of music to the call.

[0081] Thus, for example, a user may be able to configure the device to add a background music audio signal, such as a 5.0 channel audio signal 155, as in Figure 1 , as shown in step 112. In some embodiments, the selection may be communicated to a direct mixer configured to mix the captured audio (in the form of a suitable transport audio signal and spatial metadata) and the background music audio signal.

[0082] As in Figure 1 As shown in step 113, the background music audio signal (second audio signal) can be acquired or opened.

[0083] Furthermore, the background audio signal (second audio signal) may be processed to generate a suitable second audio signal transmission audio signal and spatial metadata and transmitted to the mixer.

[0084] In some embodiments, the apparatus may be configured to determine whether the mixer directly mixes the first audio signal and the second audio signal, such as in Figure 1 As shown in step 107.

[0085] If the device is configured not to mix the first and second audio signals (including examples of only the first audio), only the first audio signal is transmitted to the encoder for transmission, and if the device is configured to mix the first and second audio signals, the audio signals and the spatial metadata are directly mixed or combined for transmission, as in Figure 1 As shown in step 121.

[0086] Then, in some embodiments, the device may convert the mixed (or unmixed) transmission audio signal and spatial metadata into a suitable input format for encoding and transmission, such as in Figure 1 As shown in step 123.

[0087] The apparatus may then be configured to transmit (or store) an encoded bitstream comprising the first and second audio signals, such as in Figure 1 As shown in step 125.

[0088] about Figure 2 and 3 , showing an exemplary mixer implementation according to some embodiments.

[0089] like Figure 2 The exemplary implementation shown in shows the implementation of audio preprocessing and input format generation. In the preprocessing example, all mixing is performed before encoding (before the encoder "sees" the stream). This is therefore the most straightforward implementation in terms of encoding perspective and implementation.

[0090] therefore, Figure 2 An exemplary first audio input terminal 201 is shown, which is configured to receive a first audio signal. The first audio input signal can be, for example, a captured microphone or microphone array audio signal, in other words, an audio signal including one or more channels. The input signal can be one of any suitable formats, wherein the following example describes an input signal of 2-N channels of a spatial microphone array analyzed using a microphone array method. However, other suitable input signal formats include 2-N channels of premixed multi-channel audio analyzed using a channel-based analysis method, a first-order Ambisonics (FOA) or high-order Ambisonics (HOA) signal analyzed according to FOA using a suitable method, or an object-based audio signal including directional metadata and may also include other spatial parameters.

[0091] However, any suitable audio signal may be input, such as a multi-channel (surround sound) audio signal, a synthesized multi-channel audio signal, an Ambisonics audio signal, and / or an object-based audio signal.

[0092] The apparatus also includes a spatial processor 203 configured to receive the first audio signal and generate a transmission audio signal from the first audio signal (which, in some embodiments, may be a downmix of the first audio signal, a selection of one or more channels of audio signals, or any suitable reasonable form of the first audio signal). In addition, the spatial processor may be configured to generate suitable spatial metadata, such as direction parameters (which may include elevation, azimuth, and / or distance), energy ratio parameters, and other suitable spatial parameters.

[0093] The first audio signal parameters and the transmission audio signal may be transmitted to the mixer 215 .

[0094] Figure 2 Also shown is an exemplary second audio input 211, which is configured to receive a second audio signal. The second audio input signal can be, for example, a multi-channel (speaker) audio signal including an audio signal of two or more channels. However, any suitable audio signal can be input, for example, an audio signal captured by a multi-channel microphone, a synthetic multi-channel audio signal, an Ambisonics audio signal, and / or an object-based audio signal.

[0095] The apparatus further comprises another spatial processor 213 configured to receive the second audio signal and generate a transmission audio signal from the second audio signal (which in some embodiments may be a downmix of the second audio signal, a selection of one or more channels of audio signals, or any suitable reasonable form of the second audio signal). In addition, the spatial processor may be configured to generate suitable spatial metadata, such as direction parameters (which may include elevation, azimuth, and / or distance), energy ratio parameters, and other suitable spatial parameters.

[0096] The second audio signal parameters and the transmission audio signal may be transmitted to the mixer 215 .

[0097] In some embodiments, the apparatus includes a mixing controller 205. In some embodiments, the mixing controller can be configured to control the mixing of audio signals, as described in further detail below. Thus, in some embodiments, the mixing controller 205 can be configured to receive user input and / or audio signals and / or parameters (e.g., energy parameters) based on the audio signals.

[0098] The apparatus may further comprise a mixer 215. The mixer 215 may be configured to receive the first and second audio signals, and also receive control from the mixing controller 205, and to directly generate a mix of the two using the examples presented herein.

[0099] The output of the mixer 215 may be passed to an encoder input 221 which is configured to receive the mixed audio signal and, in some embodiments, generate a suitable format to be processed by an encoder.

[0100] The encoder 223 is configured to receive the mixed audio signal, generate a suitable encoded audio signal (including the transmission audio signal and the combined metadata), and output it to the transmitter.

[0101] The transmitter 225 may receive the encoded audio signal and transmit or store it for further use.

[0102] about Figure 3 , shows an exemplary implementation in which mixing is implemented within an encoder, where at least one spatial metadata-based input and at least one channel-based input can be combined into a single spatial metadata representation, which can then be encoded for transmission.

[0103] In some embodiments, mixing can be implemented at any suitable device, such as in a server that receives or obtains audio signals and / or in a decoder that is configured to receive audio signals. Thus, for example, in some embodiments, the encoder can be configured to transmit a partially or fully encoded form of the captured audio signal and an indication of the "background" multi-channel audio signal to be mixed (and / or the "background" multi-channel audio signal to be mixed) in a suitable uplink format. These audio signals are then mixed and encoded in the server and transmitted to the renderer / decoder in a suitable downlink format.

[0104] Similarly, in some embodiments, the encoder may be configured to send the captured audio signal in partially or fully encoded form, along with an indication of a “background” multi-channel audio signal to be mixed (and / or the “background” multi-channel audio signal to be mixed), to a decoder or renderer, which may then mix and store the audio signals for subsequent consumption or render a combined audio signal.

[0105] As in Figure 3 In such an implementation as shown in , a channel-based audio signal may require transmission signal generation and metadata extraction (in other words, spatial analysis). Then, audio mixing and metadata mixing stages may be performed.

[0106] In some embodiments, at least one audio input may be partially or fully decoded in the encoder before mixing occurs and / or before generating a suitable transmission format.

[0107] In this embodiment, the device includes an encoder input 371. The encoder input can be configured to receive a first audio signal. The first audio input signal can be, for example, a captured microphone or microphone array audio signal, in other words, an audio signal including two or more channels. However, any suitable audio signal can be input, for example, a multi-channel (surround sound) audio signal, a synthetic multi-channel audio signal, an Ambisonics audio signal, and / or an object-based audio signal.

[0108] The encoder input 371 can also be configured to receive a second audio signal. The second audio input signal can be, for example, a multi-channel (speaker) audio signal comprising an audio signal of two or more channels. However, any suitable audio signal can be input, for example, an audio signal captured by a multi-channel microphone, a synthetic multi-channel audio signal, an Ambisonics audio signal, and / or an object-based audio signal.

[0109] The encoder 373 may include an input analyzer and classifier 301. The input analyzer and classifier 301 may be configured to receive the first and second audio signals and determine whether spatial metadata associated with the first and second audio signals needs to be generated, whether the first or second audio signals need to be partially or fully decoded, and also generate controls for mixing the first and second audio signals.

[0110] The encoder 373 may also include a partial / full decoder 303 which, when controlled, is configured to partially or fully decode the audio signal and transmit the decoded audio signal to the audio mixer 307 and / or the transport signal generator and metadata generator 305 .

[0111] The encoder may also include a transmission signal generator and metadata generator 305. The transmission signal generator and metadata generator 305 may be configured to receive the first and / or second audio signals and / or partially / fully decoded first and / or second audio signals and generate a suitable transmission audio signal from these signals (in some embodiments, it may be a downmix, a selection of audio signals of one or more channels, or any suitable reasonable form of the audio signals). In addition, the transmission signal generator and metadata generator 305 may be configured to generate suitable spatial metadata, such as direction parameters (which may be elevation, azimuth, and distance), energy ratio parameters, and other suitable spatial parameters of the first and / or second audio signals.

[0112] In some embodiments, the encoder includes an audio mixer 307 configured to mix audio signals and transmit these signals to a core audio encoder 311 .

[0113] The encoder may also include a core audio encoder 311 configured to receive the mixed audio signal and generate a suitable mono / stereo or other core audio encoding.

[0114] In some embodiments, the encoder comprises a metadata mixer 309 configured to mix or merge metadata associated with the first and second audio signals.

[0115] The encoder may also include a metadata compressor 313 that may receive the mixed or merged metadata and perform appropriate metadata compression prior to transmission / storage.

[0116] The apparatus may also include a transmitter / storage device 375 configured to receive the compressed audio and metadata and transmit and / or store them in the form of a payload bitstream 321 or the like.

[0117] Having discussed exemplary devices in which mixing may occur, the following discussion further details three exemplary direct metadata merging (or mixing or combining) methods. These methods are configured to merge multiple independent streams into a single format or stream representation, as discussed in further detail below, and are divided into three main methods:

[0118] A first example may be one in which the metadata is combined by adding an additional analysis direction block to the spatial metadata. This requires that the directional metadata format matches on both directional metadata streams, or a conversion must be performed. In some embodiments, the transport signals may also be combined but sent separately. If the channel signals are sent separately, this approach may be able to produce high quality and require relatively low complexity, but at the potential cost of high bit rate.

[0119] about Figure 4 , a first example direct metadata blending or merging implementation is shown in further detail.

[0120] Combining two streams (in Figure 4 In its simplest form (shown in FIG. 4 as metadata stream 1 401 including common metadata portion 403 and directional metadata block 405, and metadata stream 2 411 including common metadata portion 413 and directional metadata block 415), the directional metadata block is taken from one spatial metadata stream and added to the other spatial metadata stream as an auxiliary directional metadata block. Thus, for example, mixer 421 receives metadata stream 1 401 and metadata stream 2 411 and outputs a combined metadata stream 421. The combined metadata stream includes common metadata portion 423 (which may be one or the other of common metadata portions 403, 413 or a combination thereof), and also includes directional metadata block 405 associated with metadata stream 1 and directional metadata block 415 associated with metadata stream 2.

[0121] The process can be implemented so that no spatial metadata decoding is required, or only partial decoding of the spatial metadata is required, so the computational efficiency is very high. In addition, in some embodiments, the transmission signals can be combined in a separate process, or in some other embodiments, the transmission signals can continue to be sent forward or stored without combination.

[0122] Therefore, about Figure 5 , which generally illustrates the operation of the first direct metadata mixing method.

[0123] As in Figure 5 The operation of receiving stream 1 is shown by step 501.

[0124] As in Figure 5 Another operation of receiving stream 2 is shown by step 502 .

[0125] After streams 1 and 2 have been received, the next operation is to extract the direction metadata block from stream 1 (or in some embodiments, from stream 2), and then insert or add it to the metadata in stream 2 (or in the same other embodiments, insert or add it to stream 1) as an auxiliary direction metadata block, as in Figure 5 As shown in step 503.

[0126] After the metadata has been merged, the next operation is to output stream 2 with the direction metadata block from stream 1 added as an auxiliary direction metadata block (or in the same other embodiment, output stream 1 with the direction metadata block from stream 2 added as an auxiliary direction metadata block), as in Figure 5 As shown in step 505.

[0127] A second example of direct mixing may be one where the metadata is mixed with the help of available channel signals. All underlying transport signals are decoded or available before encoding and are therefore known to the metadata mixer. The advantage of this approach is a potentially even higher perceptual quality and the transport signals can be mixed simultaneously for further transmission, saving downstream bandwidth. However, this approach may require a higher computational complexity of the mixer. The transport signals are analyzed and, in some scenarios, have to be decoded as an additional step. This approach may also result in additional delays if additional decoding / encoding is performed for the spatial metadata mixing, which may occur, for example, in a conference bridge.

[0128] For example, about Figure 6 This method is shown.

[0129] For the first stream, stream 1, and for all subframes and bands, the following operations may be performed by suitable means.

[0130] As in Figure 6 As shown in step 601, the (average) energy of flow 1 is determined.

[0131] As in Figure 6 As shown in step 603, the (energy) ratio parameter of flow 1 is determined.

[0132] In some optional cases, any user preferences for stream 1 are further determined, for example via a suitable user interface or user input, such as in Figure 6 As shown in step 607.

[0133] Additionally, in some optional cases, any server preferences for stream 1 may be further determined, for example, via appropriate server input, such as in Figure 6 As shown in step 609.

[0134] Then, the next operation is to determine the weight value w1 for flow 1, as in Figure 6 As shown in step 605, it can be determined as: signal energy multiplied by signal energy ratio (with the option of mixed weighted combination with user or server).

[0135] For the second stream, stream 2, and for all subframes and bands, the following operations may be performed by suitable means.

[0136] As in Figure 6 As shown in step 611, the average energy of flow 2 is determined.

[0137] As in Figure 6 As shown in step 613, the energy ratio parameter of flow 2 is determined.

[0138] In some optional cases, any user preferences for stream 1 are further determined, for example via a suitable user interface or user input, such as in Figure 6 As shown in step 617.

[0139] Additionally, in some optional cases, any server preferences for stream 1 may be further determined, for example, via appropriate server input, such as in Figure 6 As shown in step 619.

[0140] Then, the next operation is to determine the weight value w2 for flow 2, as in Figure 6 As shown in step 615, it can be determined as: signal energy multiplied by signal energy ratio (with the option of mixed weighted combination with user or server).

[0141] Then, based on the weight values, it is decided how to combine the metadata according to the following criteria.

[0142] First, check whether the weight of signal 1 is a sufficiently large metric (e.g., w1>c w2, where c=1.5) greater than the weight of signal 2, as in Figure 6 As shown in step 621.

[0143] If the check is correct, only metadata from signal 1 is used, as in Figure 6 As shown in step 631.

[0144] Otherwise, check whether the weight of signal 2 is greater than the weight of signal 1 by a sufficiently large metric (e.g., w2>c w1, where c=1.5), as in Figure 6 As shown in step 623.

[0145] If the second check is correct, only metadata from signal 2 is used, as in Figure 6 As shown in step 633.

[0146] Otherwise, the weights are used to compute a weighted average between the parameters, as in Figure 6 As shown in step 635.

[0147] Although described with two signals, this can be applied to more than two signals (in other words, any number of signals can be mixed directly or by chaining). This method is also effective without access to the transmitted signal if a suitable average energy estimate is provided for the mixing process.

[0148] In some embodiments, the average energy per subframe and frequency band is calculated by the following formula:

[0149]

[0150] Where N is the number of time samples in the time frame, Kb and Kt are the bottom and top frequency bins of the current frequency band, and I is the number of transmission channels of the stream. S(i,k,n) is the time-frequency domain representation of the transmission signal.

[0151] In some embodiments, the weight parameter is formed by combining the average energy and energy ratio of the subframe and the frequency band. This is achieved by the following formula:

[0152] w=rE avg p user p server

[0153] Among them, r is the energy ratio parameter, p user is the user preference weight, p server is the server preference weight. As mentioned above, user preference and server preference are optional inclusions.

[0154] The weights as discussed herein are then used to decide how to combine the metadata. If the weight of signal 1 is greater than the weight of signal 2 by a certain threshold, the metadata from the larger weighted signal is used to mix the signals. If the weight of signal 2 is greater than the weight of signal 1, the same approach can be applied. In addition, as discussed above, if the weights are close, a weighted average of the parameters is used. In some embodiments, the weighted average of the parameters can be determined according to the following formula:

[0155]

[0156] Where m(i, k) represents any single parameter in the metadata and c is a predefined limiting constant (e.g., 1.5). Note that this limiting constant allows switching between a strict selection between the two input metadata and a weighted average between the two input metadata. With a suitable choice of c, only one of these operation modes can be used when needed.

[0157] A third example may be one in which the metadata is mixed without access to the transport signal. In this case, the transport signal can either be sent forward simultaneously or mixed completely independently of the spatial metadata mixing. The advantage of this approach is that the spatial metadata mixing is computationally very "light" and requires no additional latency. The perceptual quality of this mixing approach is reasonable, but may not be as good as previous approaches. This approach may be useful in some device types (e.g., conference bridges). For example, in a remote conference bridge mixing implementation, a potentially lower combined audio quality may not be critical, as participants typically try to avoid talking over each other in a remote conference.

[0158] In this approach, two (or more) N-channel+metadata streams are mixed together without accessing the transmitted signal and thus without obtaining energy values. Figure 7 This method is shown.

[0159] For the first stream, stream 1, and for all subframes and bands, the following operations may be performed by suitable means.

[0160] As in Figure 7 As shown in step 703, the energy ratio parameter of flow 1 is determined.

[0161] In some optional cases, any user preferences for stream 1 are further determined, for example via a suitable user interface or user input, such as in Figure 7 As shown in step 707.

[0162] Additionally, in some optional cases, any server preferences for stream 1 may be further determined, for example, via appropriate server input, such as in Figure 7 As shown in step 709.

[0163] Then, the next operation is to determine the weight value w1 for flow 1, as in Figure 7 As shown in step 705, it can be determined as a signal energy ratio (with the option of mixed weighted combination with user or server).

[0164] For the second stream, stream 2, and for all subframes and bands, the following operations may be performed by suitable means.

[0165] As in Figure 7 As shown in step 713, the energy ratio parameter of flow 2 is determined.

[0166] In some optional cases, any user preferences for stream 2 may be further determined, for example, via a suitable user interface or user input, such as in Figure 7 As shown in step 717.

[0167] Additionally, in some optional cases, any server preferences for stream 2 may be further determined, for example, via appropriate server input, such as in Figure 7 As shown in step 719.

[0168] Then, the next operation is to determine the weight value w2 for flow 2, as in Figure 7 As shown in step 715, it can be determined as: signal energy multiplied by signal energy ratio (with the option of mixed weighted combination with user or server).

[0169] Then, based on the weight values, it is decided how to combine the metadata according to the following criteria.

[0170] First, check whether the weight of signal 1 is a sufficiently large metric (e.g., w1>c w2, where c=1.5) greater than the weight of signal 2, as in Figure 7 As shown in step 721.

[0171] If the check is correct, only metadata from signal 1 is used, as in Figure 7 As shown in step 731.

[0172] Otherwise, check whether the weight of signal 2 is greater than the weight of signal 1 by a sufficiently large metric (e.g., w2>c w1, where c=1.5), as in Figure 7 As shown in step 723.

[0173] If the second check is correct, only metadata from signal 2 is used, as in Figure 7 As shown in step 733.

[0174] Otherwise, the weights are used to compute a weighted average between the parameters, as in Figure 7 As shown in step 735.

[0175] Although described with two signals, this can be applied to more than two signals (in other words, any number of signals can be mixed directly or via concatenation). This method is also effective without access to the transmitted signal if a suitable average energy estimate is provided for the mixing process.

[0176] Therefore, this method applies the mixing of metadata similarly to the second method, but with the difference that the weights are formed in the case where energy is not available now. In this case, the following formula forms the weights.

[0177] w=rp user p server

[0178] As described, the apparatus may be implemented as part of a conference bridge device. For example, Figure 8 Such an implementation is shown.

[0179] In this implementation, there can be a server 801, which can be a centralized entity or a cloud entity or a distributed entity, configured to perform the mixing and then transmit the output to participating devices, such as user devices 811, computers 831, and other devices 821, for example, virtual reality or augmented reality or mixed reality capture / display devices or conference call devices.

[0180] In such an embodiment, participants can transmit video and audio signals to participant input terminal 851. These participant inputs can be transmitted to uplink interface 853. Participant inputs can also be encoded. The uplink interface 853 can then transmit the audio signals from the participants and any other audio signals to be mixed (e.g., the selected background audio signal) to the input decoder and audio exporter 855 and the input activity detector 857. They can decode and / or obtain, then output the audio, and determine the spatial audio parameters and transmit them to the stream selector 859. The stream selector can select the streams to be mixed and transmit them to the audio encoder and mixer 861 and the metadata combiner or mixer 363, wherein the audio encoder and mixer 361 is configured to mix streaming (or other suitable format) audio signals, and the metadata combiner or mixer 863 is configured to mix spatial metadata according to the embodiments described herein.

[0181] The audio output from the audio encoder and mixer and the spatial metadata from the metadata combiner and mixer 863 may be communicated to a downlink interface 865 which outputs the combined audio and metadata to the participants via a participant output 867 .

[0182] Thus, the embodiments described herein enable advanced immersive call implementations related to immersive communications and immersive (audio / media) experience enhancement and streaming. Figure 1 As shown, the user may be able to perform two actions, first, initiate an immersive audio enhancement or immersive audio call, where the user is recording at least the immersive audio of himself and his friends in the environment of interest (and may also record video). Secondly, the user adds a background music track or any other second audio (e.g., a 5.0 studio mix). Therefore, this experience now corresponds to a situation where, in addition to recording the real environment, the capture point is surrounded by a virtual speaker setup that provides background music. The user can add a background track by triggering a service on the device to open the second audio (file), process the second audio (processed into a negotiated format), and combine the processed second audio with the first audio into a representation based on common spatial metadata. The combined audio is fed to the codec for encoding and transmission. The addition of background music does not significantly increase the processing overhead of the codec, does not require any renegotiation, and does not cause the recipient to experience discontinuities or other perceived artifacts.

[0183] In examples where mixing occurs in the encoder rather than before the encoder, in some embodiments, the device can be configured to provide a signal to the encoder regarding the role of each audio stream. In some embodiments, such a signal can also be included in the payload and thus allow, for example, individual volume control for the recipient (at least to some extent).

[0184] In some embodiments, the selection and mixing of streams can be selectively performed based on user input or preference. Fig. 9 An exemplary user interface for providing input for mixing is shown. Thus, for example, there may be some controls related to the "position" of the stream relative to the capture location. These position controls 905 may control the orientation and position of the multi-channel input. Thus, for example, Fig. 9 A projected position 909 is shown showing the difference between the captured content at the center of the projected position 909 and the multi-channel (speaker) audio content surrounding the center, and supporting rotation of one stream relative to the other.

[0185] also, Fig. 9 A front view 907 is shown showing the difference between captured content at the center of the front view 907 and multi-channel (speaker) audio content surrounding the center, and supporting tilt and height adjustment of one stream relative to the other.

[0186] in addition, Fig. 9A slider control 903 is shown for adjusting the volume of one stream (captured audio 911) relative to another stream (multi-channel audio file 913).

[0187] Thus, in some embodiments, any suitable user input interface or device may be used to implement examples of user control of alignment and volume levels between captured content (which may be viewed as a fixed stream configuration relative to the capture device) and content from at least one file (which may be viewed as a virtual stream configuration relative to the capture device). In addition to the examples shown, in some embodiments, the relative distances of the audio file virtual speaker sources (e.g., 5.0 or 7.1 configurations) to their center points may be changed or adjusted. This may be particularly meaningful for any parameterized configuration that includes a distance parameter, and certainly for any use case that implements six degrees of freedom.

[0188] In some embodiments, some inputs may not need to be decoded (at least not completely). Audio mixing for the downlink does not always require audio encoding. This is the case, for example, if the uplink audio stream has not yet been decoded (into a waveform). The combination of streams based on independent metadata may take into account information related to the audio signal, or the mixing of metadata may be done independently.

[0189] about Fig.10 , shows an exemplary electronic device that can be used as an analysis or synthesis device. The device can be any suitable electronic device or apparatus. For example, in some embodiments, the device 1400 is a mobile device, a user device, a tablet computer, a computer, an audio playback device, etc.

[0190] In some embodiments, device 1400 includes at least one processor or central processing unit 1407. Processor 1407 may be configured to execute various program codes, such as the methods described herein.

[0191] In some embodiments, the device 1400 includes a memory 1411. In some embodiments, at least one processor 1407 is coupled to the memory 1411. The memory 1411 can be any suitable storage component. In some embodiments, the memory 1411 includes a program code portion for storing program codes that can be implemented on the processor 1407. In addition, in some embodiments, the memory 1411 may also include a storage data portion for storing data (e.g., data that has been processed or will be processed according to the embodiments described herein). Whenever necessary, the processor 1407 can obtain the implementation program code stored in the program code portion and the data stored in the storage data portion via the memory-processor coupling.

[0192] In some embodiments, device 1400 includes a user interface 1405. In some embodiments, user interface 1405 may be coupled to processor 1407. In some embodiments, processor 1407 may control the operation of user interface 1405 and receive input from user interface 1405. In some embodiments, user interface 1405 may enable a user to input commands to device 1400, for example, via a keyboard. In some embodiments, user interface 1405 may enable a user to obtain information from device 1400. For example, user interface 1405 may include a display configured to display information from device 1400 to a user. In some embodiments, user interface 1405 may include a touch screen or touch interface that enables information to be input to device 1400 and also displays information to a user of device 1400.

[0193] In some embodiments, the device 1400 includes an input / output port 1409. In some embodiments, the input / output port 1409 includes a transceiver. In such embodiments, the transceiver may be coupled to the processor 1407 and configured to enable communication with other devices or electronic devices, for example, via a wireless communication network. In some embodiments, the transceiver or any suitable transceiver or transmitter and / or receiver device may be configured to communicate with other electronic devices or devices via a wired or wired coupling.

[0194] The transceiver may communicate with other devices via any suitable known communication protocol. For example, in some embodiments, the transceiver or transceiver component may use a suitable Universal Mobile Telecommunications System (UMTS) protocol, a wireless local area network (WLAN) protocol such as, for example, IEEE 802.X, a suitable short-range radio frequency communication protocol such as Bluetooth, or an infrared data communication path (IRDA).

[0195] The transceiver input / output port 1409 may be configured to receive the speaker signal and, in some embodiments, determine parameters as described herein using the processor 1407 executing appropriate code. In addition, the device may generate appropriate transmission signals and parameter outputs to send to the synthesis device.

[0196] In some embodiments, the apparatus 1400 may be implemented as at least part of a synthesis device. Thus, the input / output port 1409 may be configured to receive the transmission signal, and in some embodiments receive parameters determined at a capture device or processing device as described herein, and generate a suitable audio signal format output using the processor 1407 executing suitable code. The input / output port 1409 may be coupled to any suitable audio output, such as to a multi-channel speaker system and / or headphones or the like.

[0197] As used in this application, the term "circuitry" may refer to one or more or all of the following:

[0198] (a) hardware circuit implementation only (such as analog and / or digital circuit implementation only);

[0199] (b) a combination of hardware circuitry and software such as (where applicable):

[0200] (i) a combination of analog and / or digital hardware circuits and software / firmware; and

[0201] (ii) any portion of a hardware processor with software (including a digital signal processor, software and memory that work together to enable a device such as a mobile phone or server to perform various functions); and

[0202] (c) Hardware circuits and / or processors, such as a microprocessor or portion of a microprocessor, that require software (e.g., firmware) to operate, but may not be present when the software is not required for operation.

[0203] This definition of "circuitry" applies to all uses of this term in this application, including in any claims. As another example, as used in this application, the term "circuitry" also covers an implementation of merely a hardware circuit or processor (or multiple processors) or a portion of a hardware circuit or processor and its accompanying software and / or firmware. The term "circuitry" also covers (for example and if applicable to the specifically claimed element) a baseband integrated circuit or processor integrated circuit for a mobile device, or a similar integrated circuit in a server, cellular network device, or other computing or network device.

[0204] Generally, various embodiments of the present invention may be implemented in hardware or dedicated circuits, software, logic, or any combination thereof. For example, some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device, but the present invention is not limited thereto. Although various aspects of the present invention may be shown and described as block diagrams, flow charts, or illustrated or described using some other graphical representation, it should be fully understood that the boxes, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general hardware or controllers or other computing devices, or some combination thereof as non-limiting examples.

[0205] Embodiments of the present invention may be implemented by computer software, or by hardware, or by a combination of software and hardware, the computer software being executable by a data processor of a mobile device, such as in a processor entity. In addition, in this regard, it should be noted that any box of the logic flow in the accompanying drawings may represent program steps, or interconnected logic circuits, blocks and functions, or a combination of program steps and logic circuits, blocks and functions. The software may be stored on a physical medium, such as a memory chip, or a memory block implemented within a processor, a magnetic medium such as a hard disk or a floppy disk, and an optical medium such as a DVD and its data variants, a CD.

[0206] The memory may be of any type suitable for the local technical environment and may be implemented using any suitable data storage technology, such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory, and removable memory. The data processor may be of any type suitable for the local technical environment and may include, as non-limiting examples, one or more of a general-purpose computer, a special-purpose computer, a microprocessor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a gate-level circuit, and a processor based on a multi-core processor architecture.

[0207] Embodiments of the present invention may be practiced in various components such as integrated circuit modules. The design of integrated circuits is essentially a highly automated process. Complex and powerful software tools are available to convert a logic level design into a semiconductor circuit design ready to be etched and formed on a semiconductor substrate.

[0208] Programs, such as those offered by Synopsys, Inc. of Mountain View, Calif., and Cadence Design, Inc. of San Jose, Calif., can automatically route conductors and position components on a semiconductor chip using well-established design rules and a library of pre-stored design modules. Once the design of a semiconductor circuit is complete, the design, in a standardized electronic format (e.g., Opus, GDSII, etc.), can be transmitted to a semiconductor manufacturing facility or "fab" for fabrication.

[0209] The foregoing description has provided a complete and informative description of exemplary embodiments of the present invention by way of exemplary and non-limiting examples. However, various modifications and adaptations will become apparent to those skilled in the relevant art in view of the foregoing description when read in conjunction with the accompanying drawings and the appended claims. However, all such and similar modifications to the teachings of the present invention will still fall within the scope of the present invention as defined in the appended claims.

Claims

1. A device comprising: at least one processor; as well as at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus to at least perform the following operations: For at least a first audio stream, determining a plurality of first metadata parameters, the plurality of first metadata parameters comprising at least one first energy ratio parameter and at least one first audio signal energy parameter; determining, for at least a second audio stream, a plurality of second metadata parameters, the plurality of second metadata parameters comprising at least one second energy ratio parameter and at least one second audio signal energy parameter; determining a first weight based at least on the determined plurality of first metadata parameters; determining a second weight based at least on the determined plurality of second metadata parameters; as well as Based on a comparison between the first weight and the second weight, the spatial metadata to be output is determined.

2. The device according to claim 1, wherein: Determining the plurality of first metadata parameters comprises: analyzing the first audio stream to determine at least one first metadata parameter of the plurality of first metadata parameters; Determining the plurality of second metadata parameters includes analyzing the second audio stream to determine at least one second metadata parameter of the plurality of second metadata parameters.

3. The device according to claim 1, wherein: Determining the spatial metadata to be output further includes: In response to the first weight being greater than at least the second weight, the first spatial metadata and the second spatial metadata are merged by determining to output at least one first metadata parameter of the plurality of first metadata parameters.

4. The device according to claim 3, wherein: At least one first metadata parameter of the plurality of first metadata parameters determined to be output includes directional metadata for the first audio stream.

5. The device according to claim 3, wherein: At least one first metadata parameter of the plurality of first metadata parameters determined to be output comprises spatial metadata for the first audio stream.

6. The device according to claim 1, wherein: Determining the spatial metadata to be output further includes: In response to the second weight being greater than at least the first weight, the first spatial metadata and the second spatial metadata are merged by determining to output at least one second metadata parameter of the plurality of second metadata parameters.

7. The device according to claim 6, wherein: At least one second metadata parameter of the plurality of second metadata parameters determined to be output includes directional metadata for the second audio stream.

8. The device according to claim 6, wherein: At least one second metadata parameter of the plurality of second metadata parameters determined to be output comprises spatial metadata for the second audio stream.

9. The device according to claim 1, wherein: The first audio stream has an audio signal format that is at least one of the following: object-based audio signals; or Spatial audio signal.

10. The device according to claim 1, wherein: The second audio stream has an audio signal format that is at least one of the following: object-based audio signals; or Spatial audio signal.

11. The device according to claim 1, wherein: Determining the first weight includes: determining the first weight based at least on a product of the determined plurality of first metadata parameters; and Determining the second weight includes determining the second weight based at least on a product of the determined plurality of second metadata parameters.

12. The device according to claim 1, wherein: Determining the spatial metadata to be output includes: combining the spatial metadata from the first audio stream and the spatial metadata from the second audio stream based on a comparison of the first weight and the second weight.

13. A method comprising: For at least a first audio stream, determining a plurality of first metadata parameters, the plurality of first metadata parameters comprising at least one first energy ratio parameter and at least one first audio signal energy parameter; determining, for at least a second audio stream, a plurality of second metadata parameters, the plurality of second metadata parameters comprising at least one second energy ratio parameter and at least one second audio signal energy parameter; determining a first weight based at least on the determined plurality of first metadata parameters; determining a second weight based at least on the determined plurality of second metadata parameters; as well as Based on a comparison between the first weight and the second weight, the spatial metadata to be output is determined.

14. The method according to claim 13, wherein: Determining the plurality of first metadata parameters comprises: analyzing the first audio stream to determine at least one first metadata parameter of the plurality of first metadata parameters; Determining the plurality of second metadata parameters includes analyzing the second audio stream to determine at least one second metadata parameter of the plurality of second metadata parameters.

15. The method according to claim 13, wherein: Determining the spatial metadata to be output further includes: In response to the first weight being greater than at least the second weight, the first spatial metadata and the second spatial metadata are merged by determining to output at least one first metadata parameter of the plurality of first metadata parameters.

16. The method according to claim 13, wherein: Determining the spatial metadata to be output further includes: In response to the second weight being greater than at least the first weight, the first spatial metadata and the second spatial metadata are merged by determining to output at least one second metadata parameter of the plurality of second metadata parameters.

17. The method according to claim 13, wherein: The first audio stream has an audio signal format that is at least one of the following: object-based audio signals; or Spatial audio signal.

18. The method according to claim 13, wherein: The second audio stream has an audio signal format that is at least one of the following: object-based audio signals; or Spatial audio signal.

19. The method of claim 13, wherein: Determining the first weight includes: determining the first weight based at least on a product of the determined plurality of first metadata parameters; and Determining the second weight includes determining the second weight based at least on a product of the determined plurality of second metadata parameters.

20. The method according to claim 13, wherein: Determining the spatial metadata to be output includes: combining the spatial metadata from the first audio stream and the spatial metadata from the second audio stream based on a comparison of the first weight and the second weight.

Images