Spatial audio

Abstract

Apparatus and methods for spatial audio are provided herein. In one embodiment, a method includes capturing a spatial audio signal through a plurality of microphones using a first capture setting, and generating an audio encoder input format data set comprising a representation of the spatial audio signal and associated metadata, the metadata includes, in part, a descriptive metadata set indicating one or more capture parameters associated with the first capture settings. The method further includes providing the audio encoder input format data set to an audio encoder to encode the representation of the spatial audio signal and the associated metadata into a bitstream for the current stream session, and transmit the bitstream to one or more remote devices to render a representation of the spatial audio signal based at least in part on the descriptive metadata set. The method further includes determining, during the current streaming session, that the first capture setting is changed to a second capture setting, and in response to the determination that the first capture setting is changed to the second capture setting and while transmitting the bitstream, capturing the spatial audio signal including capturing the spatial audio signal using the second capture setting, and generating the audio encoder input format data includes changing at least one of the one or more capture parameters in the descriptive metadata set to associate with a second capture setting.

Description

Technical Field

[0001] Example embodiments may relate to systems, methods and / or computer programs for providing spatial audio. In particular, some embodiments relate to transmission of spatial audio signals and associated metadata. Background Art

[0002] Spatial audio signals are used more frequently to produce a more immersive audio experience. Spatial audio refers to 3D audio, i.e., it can provide the perception of hearing sound sources from different directions. Spatial audio can be reproduced, for example, using a speaker setup or through headphones, preferably with head tracking capabilities. Stereo or multi-channel recordings can be passed from a recording or capture device to a listening device and replayed using a suitable multi-channel output (e.g., a multi-channel speaker arrangement) and binaural rendering or virtual surround processing on a pair of stereo headphones or headsets.

[0003] Mobile devices, such as mobile phones, may have more than two microphones. This provides the possibility of recording true multi-channel audio. With advanced signal processing, it is also possible to beamform or directional amplify or process the audio signals captured by the microphones from specific or desired directions.

[0004] The captured audio signal may include metadata to provide greater adaptability of the spatial audio signal. Summary of the invention

[0005] The scope of protection sought by various embodiments of the present invention is determined by the independent claims. Embodiments and features described in this specification that do not belong to the scope of the independent claims (if any) should be interpreted as examples that help understand the various embodiments of the present invention.

[0006] According to a first aspect, an apparatus is described, comprising: means for capturing a spatial audio signal using a first capture setting through a plurality of microphones; means for generating an audio encoder input format data set comprising a representation of the spatial audio signal and associated metadata, the metadata comprising, in part, a descriptive metadata set indicating one or more capture parameters associated with the first capture setting; means for providing the audio encoder input format data set to an audio encoder for encoding the representation of the spatial audio signal and the associated metadata into a bitstream for a current streaming session; means for sending the bitstream to one or more remote devices for rendering the representation of the spatial audio signal based at least in part on the descriptive metadata set; and means for determining, during the current streaming session, that the first capture setting is changed to a second capture setting, wherein, in response to a determination that the first capture setting is changed to the second capture setting, and when sending the bitstream, the means for capturing the spatial audio signal is configured to capture the spatial audio signal using the second capture setting, and the means for generating the audio encoder input format data set is configured to change at least one of the one or more capture parameters of the descriptive metadata set to be associated with the second capture setting.

[0007] In some embodiments, the means for determining that the first capture setting is changed to the second capture setting is configured to determine that the orientation of the apparatus has changed from a first orientation to a second orientation.

[0008] In some embodiments, the means for determining that the first capture setting is changed to the second capture setting is configured to determine that an input request has been received to change the first capture setting to the second capture setting.

[0009] In some embodiments, the apparatus further comprises means for sending the descriptive metadata set indicative of the one or more capture parameters associated with the first capture setting to the one or more remote devices via an out-of-band signal.

[0010] In some embodiments, the one or more capture parameters include at least one of: a directional unit including the number of directions described by the spatial metadata; a channel unit including the number of transmission channels supported by the device; a source format describing the configuration of the device; and a variable description describing at least one of a capture type, an angle between two of the multiple microphones, a device size, and a microphone polarity pattern, wherein the microphone polarity pattern includes an omnidirectional, cardioid, hypercardioid, or supercardioid pattern.

[0011] In some embodiments, the bitstream of the current streaming session includes the one or more capture parameters.

[0012] In some embodiments, the means for generating the audio encoder input format data set is configured to use a Metadata Assisted Spatial Audio (MASA) format.

[0013] In some embodiments, the audio encoder is an Immersive Voice and Audio Services (IVAS) codec.

[0014] In some embodiments, the apparatus comprises user equipment.

[0015] According to a second aspect, an apparatus is described, comprising: a component for receiving a bitstream for a current streaming session; a component for decoding the bitstream for the current streaming session to determine a representation of a spatial audio signal and associated metadata, the metadata partially comprising a descriptive metadata set indicating one or more capture parameters associated with a first capture setting; a component for configuring an audio renderer based on the decoded descriptive metadata; a component for providing the representation of the spatial audio signal to the audio renderer for rendering to produce a rendered audio output signal; a component for outputting the rendered audio output signal through a plurality of speakers; a component for receiving an updated bitstream for the current streaming session and a component for decoding the updated bitstream for the current streaming session to determine an updated representation of the spatial audio signal and associated metadata, the metadata partially comprising a descriptive metadata set indicating one or more capture parameters associated with a second capture setting, wherein the component for configuring the audio renderer based on the decoded descriptive metadata is configured to change at least one of the one or more capture parameters in the descriptive metadata set to be associated with the second capture setting.

[0016] In some embodiments, the apparatus further comprises means for receiving a bitstream for a current streaming session, the bitstream being received.

[0017] In some embodiments, the apparatus further comprises means for receiving the descriptive metadata set indicating one or more capture parameters associated with the first capture setting to the one or more remote devices via an out-of-band signal; and wherein the means for configuring the audio renderer further comprises configuring the audio renderer according to the descriptive metadata set indicating one or more capture parameters associated with the first capture setting.

[0018] In some embodiments, the apparatus comprises user equipment.

[0019] According to a third aspect, a method is described, comprising: capturing a spatial audio signal using a first capture setting through multiple microphones; generating an audio encoder input format data set including a representation of the spatial audio signal and associated metadata, the metadata partially including a descriptive metadata set indicating one or more capture parameters associated with the first capture setting; providing the audio encoder input format data set to an audio encoder to encode the representation of the spatial audio signal and the associated metadata into a bitstream for a current streaming session; sending the bitstream to one or more remote devices to render the representation of the spatial audio signal based at least in part on the descriptive metadata set; and determining during the current streaming session that the first capture setting is changed to a second capture setting, wherein, in response to a determination that the first capture setting is changed to the second capture setting, and when sending the bitstream, capturing the spatial audio signal includes capturing the spatial audio signal using the second capture setting, and generating the audio encoder input format data includes changing at least one of the one or more capture parameters of the descriptive metadata set to be associated with the second capture setting.

[0020] According to a fourth aspect, a method is described, comprising: receiving a bitstream for a current streaming session; decoding the bitstream for the current streaming session to determine a representation and associated metadata of a spatial audio signal, the metadata partially comprising a descriptive metadata set indicating one or more capture parameters associated with a first capture setting; configuring an audio renderer according to the decoded descriptive metadata; providing the representation of the spatial audio signal to the audio renderer for rendering to produce a rendered audio output signal; outputting the rendered audio output signal through a plurality of speakers; receiving an updated bitstream for the current streaming session and decoding the updated bitstream for the current streaming session to determine an updated representation and associated metadata of the spatial audio signal, the metadata partially comprising a descriptive metadata set indicating one or more capture parameters associated with a second capture setting, wherein configuring the audio renderer according to the decoded descriptive metadata comprises changing at least one of the one or more capture parameters in the descriptive metadata set to be associated with the second capture setting.

[0021] According to a fifth aspect, there is provided a computer program product comprising a set of instructions which, when executed on an apparatus, are configured to cause the apparatus to perform a method as defined in any preceding method.

[0022] According to a sixth aspect, a non-transitory computer-readable medium is provided, comprising program instructions stored thereon for performing a method, the method comprising: capturing a spatial audio signal using a first capture setting through multiple microphones; generating an audio encoder input format data set comprising a representation of the spatial audio signal and associated metadata, the metadata partially comprising a descriptive metadata set indicating one or more capture parameters associated with the first capture setting; providing the audio encoder input format data set to an audio encoder to encode the representation of the spatial audio signal and the associated metadata into a bitstream for a current streaming session; sending the bitstream to one or more remote devices to render the representation of the spatial audio signal based at least in part on the descriptive metadata set; and determining during the current streaming session that the first capture setting is changed to a second capture setting, wherein, in response to a determination that the first capture setting is changed to the second capture setting, and when sending the bitstream, capturing the spatial audio signal comprises capturing the spatial audio signal using the second capture setting, and generating the audio encoder input format data comprises changing at least one of the one or more capture parameters of the descriptive metadata set to be associated with the second capture setting.

[0023] According to a seventh aspect, a non-transitory computer-readable medium is provided, comprising program instructions stored thereon for performing a method, the method comprising: receiving a bitstream for a current streaming session; decoding the bitstream for the current streaming session to determine a representation and associated metadata of a spatial audio signal, the metadata partially comprising a descriptive metadata set indicating one or more capture parameters associated with a first capture setting; configuring an audio renderer according to the decoded descriptive metadata; providing the representation of the spatial audio signal to the audio renderer for rendering to produce a rendered audio output signal; outputting the rendered audio output signal through multiple speakers; receiving an updated bitstream for the current streaming session and decoding the updated bitstream for the current streaming session to determine an updated representation and associated metadata of the spatial audio signal, the metadata partially comprising a descriptive metadata set indicating one or more capture parameters associated with a second capture setting, wherein configuring the audio renderer according to the decoded descriptive metadata comprises changing at least one of the one or more capture parameters in the descriptive metadata set to be associated with the second capture setting. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Example embodiments will now be described by way of non-limiting examples with reference to the accompanying drawings, in which:

[0025] Figure 1A is a plan view of a first apparatus in a first orientation in which one or more example embodiments may be applied;

[0026] Figure 1B yes Figure 1A a plan view of the first device in a second orientation;

[0027] Figure 2 is the view of the first device in use by the user;

[0028] Figure 3A is a plan view of a first apparatus in a first orientation in which one or more example embodiments may be applied;

[0029] Figure 3B is a plan view of a first apparatus in a second orientation in which one or more example embodiments may be applied;

[0030] Figure 3C is a plan view of a first apparatus in a third orientation in which one or more example embodiments may be applied;

[0031] Figure 4 is a view of a first device held by a user;

[0032] Figure 5 is a plan view of a third apparatus to which one or more example embodiments may be applied;

[0033] Figure 6 is a flow chart illustrating method steps that may be performed by an apparatus according to one or more example embodiments;

[0034] Figure 7 is a flow chart illustrating method steps that may be performed by an apparatus according to one or more further example embodiments;

[0035] Figure 8 is a flowchart illustrating processing operations that may be performed by an apparatus according to one or more example embodiments;

[0036] Fig. 9 is a flowchart illustrating processing operations that may be performed by an apparatus according to one or more further example embodiments;

[0037] Fig.10 is a flowchart illustrating processing operations that may be performed by an apparatus according to one or more further example embodiments;

[0038] Fig.11 An apparatus according to one or more example embodiments is shown; and

[0039] Fig.12 A non-transitory medium for storing computer readable code or instructions that, when executed by an apparatus such as FIG. 1 , may perform the processing operations described herein is shown. DETAILED DESCRIPTION

[0040] Example embodiments relate to apparatus, methods and computer programs for outputting stereo or spatial audio. The stereo or spatial audio may be represented by any suitable form of data, whether in the form of one or more data files, or in the case of streaming data, data packets or any other suitable format. The stereo or spatial audio may relate to voice and other audio communications.

[0041] In its basic form, stereo audio data includes two channels, a left channel and a right channel, for output by first and second speakers, respectively. More advanced formats include 2.1, 5.1, and 7.1, with 2.1 adding lower frequencies for output to a third subwoofer speaker, and 5.1 and 7.1 being generally referred to as "surround sound" formats. Spatial audio data, also known as three-dimensional audio or immersive audio, may describe audio data that enables a user to perceive sounds from all around them; for a fully immersive experience, the spatial audio data may include cues so that the user can perceive other attributes, such as the direction of sound emitted by one or more sound sources or objects, the trajectory of the sound sources or objects, changes in the amplitude of the sound based on the distance from the sound sources or objects, and other sound effects. For example, if a user moves their user device (e.g., a smartphone), this may change the perception of the audio.

[0042] As used herein, the apparatus may include a user device having three or more microphones. The user device may be a portable user device, such as a smartphone, a tablet, a digital assistant, a wearable computer, or a head mounted device (HMD). This list is not exhaustive. The user device may also include a speaker.

[0043] User devices may have different form factors. For example, some user devices have multiple screens, some have three or more microphones, and / or some may be foldable, i.e., have a foldable body that carries a foldable screen for open and closed configurations, and may be used for one or more intermediate configurations in which the screen is at an angle between the open and closed configurations. Some user devices may be used in different orientations, for example, changing the user interface from portrait mode to landscape mode upon detecting that the user device is rotated more than 45 degrees or approximately 45 degrees from the horizontal plane. The user device may be configured to receive and decode different types of audio data, including mono audio data, stereo (stereo) audio data including two channels, other forms of multi-channel audio data (e.g., 2.1, 5.1, and 7.1), and spatial audio data (e.g., Ambisonics or metadata-assisted spatial audio (MASA)).

[0044] The user equipment may be able to establish a communication session with one or more other user equipment, servers and / or nodes through a communication network. The user equipment may be configured to send and receive data using protocols for 3G, 4G, LTE, 5G or any future generation communication protocols. The user equipment may include components for short-range communication using, for example, Bluetooth, Zigbee or WiFi. The user equipment may include one or more antennas for communicating with external devices, such as one or more other remote user equipment and / or one or more remote servers and / or one or more communication nodes of a network.

[0045] In use, the user device can process and output different types of audio data. For example, the user device can output stereo audio data associated with a music track or a movie to the first and second speakers. When other audio data (i.e., audio data that is not currently output to the first and second speakers) is received, this is usually output by one or both of the first and second speakers. For example, when a new text or multimedia message is received, an audible notification can be output to one or both of the first and second speakers. The two types of audio data are mixed at least for a period of time. For other types of data (such as incoming calls or meeting notifications), there may be the same or similar situation. In fact, sometimes the output of other audio data may pause, mute or reduce the volume of stereo audio data at least for a period of time. Example embodiments are intended to improve the flexibility and user experience of user devices with three or more speakers and / or microphones. For example, example embodiments can enable one or more speakers that are not currently in use to be utilized. For example, example embodiments can enable optimized audio output, such as stereo expansion effects, enhanced immersion and / or increased volume. Other advantages will become apparent.

[0046] Spatial audio refers to 3D audio. Spatial audio includes the perception of the location of sound sources heard from different directions and replicating the audio to the user. For example, spatial audio can be reproduced using a speaker setup or through headphones. Optionally, spatial audio rendering can utilize head tracking capabilities.

[0047] Spatial capture can be performed using various methods. For example, spatial audio can be captured using a multi-microphone device (such as a smartphone). Figure 1A is a plan view of the first device 100 in a first orientation, according to Figure 1A One or more exemplary embodiments may be applied. Figure 1A The device 100 is a smartphone shown from the back. The smartphone device 100 includes a back side 102, a front side 104, microphones 106A-106F, and a camera 108. Figure 1B yes Figure 1AThe first device is in a plan view in a second orientation. Figure 1B The device 100 is Figure 1A The smartphone is flipped so that the front side 104 is exposed. The front side 104 of the smartphone may include a touch screen. Microphones 106A, 106B, and 106C are located on the back side 102 of the device, while microphones 106D, 106E, and 106F are located on the front side 104 of the device. Figure 1A and Figure 1B The device shown in is only an example, and the microphones may be arranged in other configurations, as shown in Figures 3 to 5. The camera 108 is located on the back 102 of the device, and optionally, there may also be a camera (not shown) on the front 104 of the device. At least one microphone 106C may be located next to the camera 108.

[0048] Figure 1A and 1B The device may use a parametric capture component to analyze and process audio input signals received at microphones 106A-106F to derive a parametric spatial audio format or synthesize some other spatial audio format. The smartphone has integrated microphones 106A-106F, which are placed in a position that allows good voice pickup in different orientations and usage situations. For good spatial audio capture and processing, microphone separation along each of the three principal axes is preferred, however, in thin devices, microphone separation along only two axes is desired.

[0049] The device may be a user device or user equipment (UE), which generally refers to a portable computing device, including wireless mobile communication devices operating with or without a subscriber identity module (SIM), including but not limited to the following types of devices: mobile stations (mobile phones), smartphones, personal digital assistants, handheld devices, devices using wireless modems (alarm or measurement devices, etc.), laptops and / or touch screen computers, tablets, game consoles, notebooks, and multimedia devices. It should be understood that a user device may also be an almost dedicated uplink-only device, an example of which is a camera or video camera that loads images or video clips to a network. A user device may also be a device with the ability to operate in an Internet of Things (IoT) network, which is a scenario in which objects are given the ability to transmit data over a network without human-to-human or human-to-computer interaction.

[0050] Known from the ongoing 3GPP standardization of the Immersive Voice and Audio Services (IVAS) codec and specified in Annex A of 3GPP TS26.258, Metadata Assisted Spatial Audio (MASA) is a parameterized spatial audio format. For example, Atmos is a non-parametric spatial audio format. MASA is a spatial audio format, while metadata can be obtained by analyzing microphone signals. The MASA format consists of an audio signal and metadata. The audio signal can be mono or stereo (i.e. 1-2 transmission channels). The metadata includes descriptive metadata and spatial metadata. Examples of descriptive metadata and spatial metadata are given in detail below:

[0051]

[0052] Table A.1: MASA format descriptive common metadata parameters

[0053]

[0054]

[0055] Table A.2a: MASA format spatial metadata parameters (depending on the number of directions)

[0056]

[0057] Table A.2b: MASA format spatial metadata parameters (independent of the number of directions)

[0058] Table A.1 lists the MASA descriptive common metadata parameters in the order in which they are written.

[0059] Table A.2a and Table A.2b list the MASA spatial metadata parameters that are dependent and independent of the number of directions, respectively.

[0060] The MASA spatial metadata describes the spatial audio characteristics corresponding to one or two transmitted audio signals. Therefore, a spatial audio scene can be rendered for listening based on the combination of the transmitted audio signal and the spatial metadata.

[0061] The following describes the definition and use of MASA spatial metadata parameters in order.

[0062] Direction index: spatial direction

[0063] Spatial directions represent the directional energy flow in the sound scene. Each spatial direction together with the corresponding direction to total energy ratio describes how much of the total energy for each time-frequency patch comes from that specific direction. In general, this parameter can also be considered as the direction of arrival (DOA).

[0064] The input metadata can have one or two spatial directions for each time-frequency tile. Each spatial direction is represented using a 16-bit direction index. This is an efficient way to represent directions as spherical grid points with an accuracy of about 1 degree in any direction.

[0065] The direction index corresponds to the audio direction angle value (azimuth and elevation angle θ) into an index, and an inverse function that converts the index into an audio direction angle value.

[0066] Each pair of values ​​containing elevation and azimuth is first quantized on a spatial spherical point grid and the index of the corresponding point is constructed. First the structure of the spherical grid is defined, then the quantization function, and finally the index formation, followed by the corresponding de-indexing function.

[0067] The spherical grid is defined as a series of horizontal circles of points. These circles are distributed on the sphere and they correspond to several elevation values. The index function associates the angle (elevation and azimuth) corresponding to each point on the grid with a 16-bit index.

[0068] A spherical mesh is located on a sphere of unit radius, defined by the following elements:

[0069] -Elevation values ​​are equidistant between -90 and +90 degrees; a value of 0 represents and corresponds to a circle located at the equator. The values ​​are symmetric with respect to the origin. The number of positive elevation values ​​is N θ =122.

[0070] -For each elevation value, there are several equidistant azimuth values. A point on the grid is given by an elevation and an azimuth value.

[0071] The number of azimuth values ​​n(i) is calculated as follows:

[0072] οAt the equator of the spherical grid (θ = 0), it is set to

[0073] n(1)=430

[0074] οThere is a point at each pole (θ = ±90 degrees)

[0075] n(N θ )=1

[0076] ο Calculate the elevation index i=2,...,N on the grid for other elevation indexes θ The function of the number of points n(i) with respect to -1 is defined as follows:

[0077]

[0078] where cumN(1) = 0 and

[0079]

[0080] where δ is i=1,...,N θ -1 uniform quantization step, 2round i (x / 2) is a function that rounds to the nearest even number (greater than x for i==2, and closest for i>2). The term cumN(i) gives the cumulative cardinality (i.e., the cumulative number of points in the spherical grid) in the spherical region from the first non-zero elevation value to the i-th elevation value. This cumulative cardinality comes from the relative area on the sphere, assuming the number of remaining points is 2 16 -432 (not to mention the equator and the poles) has a (close to) uniform distribution of points.

[0081] - Azimuth values ​​go from 0 to 2π in trigonometric order starting from the head-on direction.

[0082] - For odd values ​​of i, the quantized azimuth values ​​are equally spaced, starting from 0.

[0083] - For even values ​​of i, the quantized azimuth values ​​are equally spaced, starting from start.

[0084] - For the same absolute value elevation codeword, the number of quantized azimuth values ​​is the same.

[0085] Quantization in a spherical grid is performed as follows:

[0086] - The elevation angle value is quantized to the two closest values ​​θ1, θ2 in a uniform scalar quantizer

[0087] - The azimuth values ​​are quantized in the azimuth scalar quantizer corresponding to the elevation values ​​θ1, θ2

[0088] - Calculate the spherical distance between the input elevation-azimuth pair and each quantized pair (θ1, φ1), (θ2, φ2)

[0089] d i =-(sinθsinθ i +cosθ i cos(φ-φ i )), i=1:2

[0090] -Select the pair with smaller distance as the quantization direction.

[0091] The resulting quantized direction index is obtained by enumerating the points on the spherical grid, starting with the point for zero elevation, then the point corresponding to the smallest positive elevation codeword, the point corresponding to the first negative elevation codeword, followed by the point at the next positive elevation codeword, and so on.

[0092] Direction to total energy ratio

[0093] Directional to total energy ratios work in conjunction with the spatial directions described above. Each directional to total energy ratio corresponds to a specific spatial direction and describes how much energy comes from that specific spatial direction compared to the total energy.

[0094] Extended Coherence

[0095] Extended coherence is a parameter that further describes the directional energy flow. It represents the situation where coherent directional sound energy comes from multiple directions at the same time. This is represented by a single extended coherence parameter that describes how the sound should be synthesized.

[0096] In synthesis, this parameter should be used so that a value of 0 means that the sound is synthesized into a single direction pointed by the spatial direction, a value of 0.5 means that the sound is synthesized into the spatial direction and the two surrounding directions as coherent directions, and a value of 1 means that the sound is synthesized into the two surrounding directions surrounding the spatial direction.

[0097] Diffusion to total energy ratio

[0098] The diffuse to total energy ratio represents the non-directional energy flow in the sound scene. This is the complement of the directional to total energy ratio, which in an ideal capture without undesired signals (or a synthetic sound scene) would always be

[0099] r diff =1-∑r dir .

[0100] Surround coherence

[0101] Surround coherence is a parameter that describes the flow of non-directional energy. It indicates how much non-directional energy should be presented as a coherent reproduction rather than a decorrelated reproduction.

[0102] Residue to total energy ratio

[0103] The residual to total energy ratio represents all the energy that does not "belong" to the captured sound scene according to the model used. This includes possible microphone noise and other capture artefacts that were not removed from the signal in preprocessing. This means that by considering the directional to total energy ratio, the diffuse to total energy ratio, and the residual to total energy ratio, we end up with a complete energy ratio model when any residual energy is present.

[0104]

[0105] Spatial metadata describes how spatial rendering is done, i.e. from which direction a certain time-frequency component should be rendered etc. This defines the sound scene (together with the audio). Descriptive metadata provides additional information, e.g. about how spatial capture or spatial audio generation was done. In some cases this information can be used to optimize the encoding, and when the information is available at the receiver it can be used to improve the rendering quality in advanced rendering. For this purpose this information needs to be delivered via the transport channels, e.g. using the codec bitstream, RTP signaling or any other suitable method.

[0106] Channel Audio Format in MASA Descriptive Metadata defines the number of directions (1-2), the number of channels (1-2 transmitted channels), and the source format (including unknown source formats for mixdowns (default); microphone grids, e.g. smartphones or other such UEs; channel-based sources (e.g. 5.1); or Atmos) in the MASA metadata. Additionally, there are so-called variable descriptions. All this information within a session can be constant / fixed, however, the information is provided in every frame, i.e. every 20 ms, so it may change frequently. Depending on the exact source format value (and number of transmitted channels), the variable description provides, for example, transmitted channel polar pattern information, channel angles for the directional pattern, and channel distance information.

[0107] Spatial audio capture can be done in many different usage scenarios. For example, regular spatial audio calls, spatial audio with audio focus, and spatial audio capture in rich ambience often require slightly different means and settings for optimal capture, encoding, and rendering. In capture, it is up to the implementer and many solutions may exist. On the other hand, anything related to encoding needs to be available to the codec, and for anything related to rendering, the information needs to be available to the receiving UE.

[0108] If these optimizations and adaptations are not fully considered, the final user experience (such as the spatial audio quality of the rendering) will always be compromised to some extent. Therefore, comprehensive improvements are needed. It is worth noting that the IVAS MASA format provides a mechanism to adapt several rendering-related aspects of descriptive metadata at a 50Hz update rate, and the IVAS codec can then be used to provide immersive voice and audio capture optimizations in a suitable manner.

[0109] The present disclosure relates to spatial audio capture, encoding, transmission and rendering, wherein a method is proposed to achieve rendering quality improvement in spatial audio communication and real-time streaming by configuring and adding descriptive metadata parameters corresponding to a selected spatial audio capture mode and / or target effect, thereby providing real-time information describing specific properties of spatial audio capture settings and / or target spatial audio effects to a spatial audio renderer. The utilization of the sent descriptive metadata parameters is based on the specific renderer implementation of the receiving device. Therefore, the present disclosure enables the provision of audio capture parameters synchronized with the corresponding spatial audio representation to the renderer via a codec bitstream or out-of-band signaling.

[0110] Adapting audio capture parameters may, in an example, for example indicate separation of an audio focused source or multiple audio focused sources to improve rendering control in playback.

[0111] Adapting the audio capture parameters may in further examples for example indicate a change in at least one of: the capture microphone polar pattern, the angle between beams, and the distance between stereo channels. This information may be used in the rendering to, for example, modify the rendering so that the user cannot hear the changes, so that the rendering is maintained regardless of any changes in the capture, or to make the changes more noticeable so that the effect is stronger.

[0112] Therefore, rendering control is significantly improved, thereby improving the user experience. Specifically, the current disclosure is described in the context of the IVAS MASA input format, and the disclosure can be implemented as part of an enhanced MASA audio capture system. According to the provided IVAS implementation, based on the IVAS frame size and MASA metadata definition, the adaptation rate of the system is 50Hz. For example, out-of-band signaling, in a preferred example, a real-time transport protocol (RTP) can be used to send information at least substantially synchronously with the audio codec bit stream. Although the adaptation rate at input can be 50Hz, the update rate sent can be the same or lower.

[0113] The present disclosure relates to real-time adaptation of descriptive metadata in a parameterized spatial audio format for encoding and transmission in spatial audio communication and streaming use cases and services. Specifically, the codec may be a 3GPP IVAS codec, the parameterized audio format may be a MASA format, and the adapted portion of the descriptive metadata may include a channel audio format including a source format description, the source format description including a transport definition, channel angles and channel distances, while the number of channels may be (at least primarily) 2, and the number of directions may be 1 or 2.

[0114] Figure 2 is a view 200 of a first device 300 used by a user 202 in a first capture setting. The first capture setting is Figure 2, shown in phone mode, where the device 100 is held against the user's ear during a call. Figure 3A is a plan view of the device 100 in a first orientation 300 . Figure 3B is a plan view of the first device in a second orientation 310 . Figure 3C is a plan view of the first device in a third orientation 320. Figure 2 and Figures 3A-3C As shown, user 202 is using device 100 in mobile mode to make a media call. Spatial audio is captured and transmitted according to the first capture setting. Figure 3A As shown, microphones 302A-302D may be used for spatial audio analysis. Figure 3B The same microphones are shown, labeled 312A-312D, Figure 3C 322A-322D. At least two channels 314 received by the microphones are selected as transmission channels, thereby allowing good voice pickup and left-right separation of the transmission channels. The two channels 314 selected as transmission channels can be the channels closest to the mouth when the user speaks to the device 300. The two channels selected as transmission channels can change depending on the orientation of the device.

[0115] In phone mode, device 100 can have several different orientations (e.g. Figures 3A-3C 300, second orientation 310, and third orientation 320 are shown in FIG. 300. In general, a slightly upright orientation (portrait) is typical. Figures 3A-3C All display devices are in phone mode, so the first capture setting is used. In some examples, phone mode can cover an orientation range angle within 90 degrees.

[0116] The user 202 may first place the device 100 (e.g., a mobile phone) against his ear for a media call, i.e., the user captures a spatial audio signal without video in mobile phone mode, and in this case uses a first capture setting. The placement of microphones 302A-302D and the resulting selections are to be understood as an example only. In the example with four microphones 302A-302D, one microphone may be placed in the center of each side of the device. The example capture first selects and maintains the following parameters in the MASA descriptive metadata:

[0117] - Set the number of directions (1 bit) to "0" or "1", depending on the capture implementation. (We note that capturing a single direction, i.e., a value of "0", may be sufficient regardless of the device capabilities in this scenario.)

[0118] - According to the capture system supporting 2 transmission channels, the number of channels (1 bit) is set to "2".

[0119] - Source format (2 bits) according to the specific system settings. For smartphones, the configuration might be, for example, "10" or "00", depending on the exact configuration and manufacturer design choice.

[0120] - Set the variable description (12 bits) to, for example, "001000011100", where:

[0121] о "001" indicates that omnidirectional capture is selected, which is common in smartphones, for example for voice-centric capture,

[0122] o "000" indicates an unspecified angle between the transmitted channels, and

[0123] о "011100" indicates that the channel distance is approximately 7 cm, corresponding to this particular example device dimensions and its microphone placement for the transmission channel in a (at least relatively) upright configuration in phone mode.

[0124] It will be appreciated that many MASA formats are contemplated for use, but the above represents one example MASA format.

[0125] When the media call is completed, the user 202 then changes the orientation of the device 100 to capture further audio (e.g., landscape orientation, see Figure 4 ) and introduce a video component to their media call. For example, a user wants to show a musical performance to a recipient. Figure 4 and Figure 5 The representation device is held in alternative orientations 400, 500. The placement of microphones 502A-502D and the resulting selection is to be understood as just one example and other possible use case scenarios are conceivable. The system now determines a new configuration suitable for the change in use case, the new configuration being referred to as the second capture setup.

[0126] In the second capture setup, the spatial audio signal and the corresponding metadata are adapted. The parameters in the MASA descriptive metadata may now be as follows, for example:

[0127] - Set the number of directions (1 bit) to "0" or "1", depending on the capture implementation. (We note that if the capture device is able to capture both directions, i.e. value "1", this generally provides better quality.)

[0128] - According to the capture system supporting 2 transmission channels, the number of channels (1 bit) is set to "2".

[0129] - Source format (2 bits) according to the specific system settings. For smartphones, the configuration may be "10" or "00", depending on the exact configuration and manufacturer design choice.

[0130] - Set the variable description (12 bits) to "011010100111", where:

[0131] о "011" indicates the selection of cardioid capture,

[0132] ο "010" indicates that the angle between the transmission channels in this configuration is + / -70°, and

[0133] o "100111" indicates that the channel distance is approximately 16 cm, corresponding to the dimensions of this example device and its microphone placement for transmitting the channel in the landscape configuration that the user is currently using.

[0134] Thus, a user triggering recording of a video as part of a media call can result in a different selection of a second capture setting, which in turn drives a new selection of a microphone polar pattern configuration, and ultimately results in a corresponding change in the descriptive metadata sent to the receiving user. Furthermore, the orientation of the device can change the optimal transmission signal selection. Again, this change is visible to the receiver as part of the descriptive metadata in a parameterized format.

[0135] Thus, in this example, it is the first two subfields in the "Variable Description" that adapt based on the selection driven by the mode selection (audio only, audio video). In addition, a change in device orientation results in a different transmission channel selection for microphones 502A and 502B, which is primarily reflected in the last subfield value. The new transmission channel selection for microphones 502A and 502B is based on the most useful microphone to be used.

[0136] When transmitted to a receiver, this new adaptation behavior of the captured information can be used to optimize the rendering in a suitable renderer implementation. For example, a wider sound stage can be rendered for a music video section. During a phone mode call, a softer spatial scene may be of greater interest.

[0137] A second example use case (not shown) may also be considered. Audio focus is a feature that allows directional sound sources (e.g., speakers) in a scene to be clearly captured by beamforming. Thus, audio focus may, for example, suppress some background noise or ambience signals in order to more clearly capture and reproduce the speaker. For example, a user may wish to use audio focus when recording a spatial scene.

[0138] Consider that the audio focus algorithm on the user's smartphone detects two talkers (or other sound sources of interest) in different parts of the scene, and the activity signals from these two sound sources overlap at least partially in time. For example, during rendering, it would be useful to be able to focus more on one of the audio focus sources at a given time. According to the present disclosure, this can be better achieved through appropriate description of the capture scene and the related capture configuration, which can be achieved by adapting descriptive metadata (i.e., in the second capture setup) and sending it to the receiver.

[0139] According to the present disclosure, the system now adapts descriptive metadata according to the second capture setting in the MASA format during a specific portion of the audio-focused spatial audio capture, as follows:

[0140] - Set the direction number (1 bit) to "0" or "1" depending on the capture implementation. (We note that in this use case, implementations that implement analysis for both directions, i.e. using a value of "1", generally provide higher quality output.)

[0141] - According to the capture system that supports 2 transmission channels, the number of channels (1 bit) is set to "2".

[0142] - Source format (2 bits) according to the specific system settings. For smartphones, the configuration may be "10" or "00", depending on the exact configuration and manufacturer design choice.

[0143] - Set the variable description (12 bits) to, for example, "101100100111", where:

[0144] о "101" indicates selection of strong cardioid beam or polar pattern depending on the audio focusing effect applied,

[0145] ο "100" indicates to select an angle of + / -45° for audio focus effect, and

[0146] о "100111" indicates that the channel distance is about 16 cm, corresponding to the size of the device and the placement of its microphone for transmitting the channel.

[0147] Therefore, it is specifically the first two subfields in the "variable description" that are adapted according to the selection of the audio focus processing driver.

[0148] This information set, when transmitted to a receiver, can be used to optimize the rendering in a suitable renderer implementation.For example, the listener may have the ability to control the rendering of the two main sound sources detected and focused by the audio focus.

[0149] A third example use case (not shown) can also be considered. For example, the user is in a nice ambience in a forest with birds singing in the trees. The spatial audio capture algorithm on the user's tablet device detects the rich ambience in the 3D sound scene. For example, there can be an AI component to understand in which environment the user is capturing audio. Good settings are selected for the scene, for example, smooth playback with good 3D overlay is appropriate.

[0150] According to the present disclosure, the system chooses to capture the configuration and adapt the descriptive metadata of the MASA format. It is understood that many MASA formats can be used, but as an example, the recommended MASA format is as follows:

[0151] - Set the direction number (1 bit) to "0" or "1" depending on the capture implementation. (We note that in this use case, implementations that implement analysis for both directions, i.e. using a value of "1", generally provide higher quality output.)

[0152] - According to the capture system that supports 2 transmission channels, the number of channels (1 bit) is set to "2".

[0153] -Set the source format according to the specific system (2 bits). For smartphones, the configuration can be "10" or "00", depending on the exact configuration and manufacturer design choice.

[0154] - Variable description (12 bits) is set to "010001101000", where:

[0155] ο "010" indicates the selection of the subcardioid polar pattern for smooth and wide ambience playback,

[0156] ο "001" indicates an angle of + / - 90° depending on the desired effect, and

[0157] ο "101000" indicates a channel distance of approximately 29 cm, corresponding to the example device dimensions and the position of its microphone in the selected transmission channel.

[0158] Therefore, the first two subfields in the "Variant Description" are specifically adapted according to the selection driven by the desired effect of smoothly playing the vast immersive atmosphere. This generates a second capture setting, which may be different from the first initial capture setting that the device initially used.

[0159] This information set, when transmitted to a receiver, can be used to optimize the rendering in a suitable renderer implementation. For example, listeners can make better use of their multi-channel speaker setup, where more content is now played from behind the user and in the height channels.

[0160] Figure 6A flowchart of a method according to an example embodiment is shown by way of example. Each unit of the flowchart may include one or more operations. These operations may be performed in hardware, software, firmware, or a combination thereof. For example, these operations may be performed individually or collectively by components, wherein the components may include at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the execution of the operations.

[0161] The method 600 includes a first operation 601 of capturing a spatial audio signal through a plurality of microphones using a first capture setup.

[0162] The method 600 includes a second operation 602 of generating an audio encoder input format data set including a representation of a spatial audio signal and associated metadata, the metadata including, in part, a descriptive metadata set indicating one or more capture parameters associated with the first capture setting. Generating the audio encoder input format data set may optionally include generating the audio encoder input format data set configured to use a MASA format.

[0163] The method 600 comprises a third operation 603 of providing the audio encoder input format data set to an audio encoder to encode the representation of the spatial audio signal and the associated metadata into a bitstream for the current streaming session. Optionally, the audio encoder may be an IVAS codec.

[0164] The method 600 includes a fourth operation 604 of transmitting the bitstream to one or more remote devices to render a representation of the spatial audio signal based at least in part on the set of descriptive metadata.

[0165] The method 600 includes a fifth operation 605 of determining whether the first capture setting is changed to a second capture setting during the current streaming session.

[0166] In response to determining that the first capture setting is changed to a second capture setting, and when sending the bitstream, capturing the spatial audio signal includes capturing the spatial audio signal using the second capture setting, and generating the audio encoder input format data includes changing at least one of the one or more capture parameters of the descriptive metadata set to be associated with the second capture setting. Optionally, determining that the first capture setting is changed to the second capture setting may include determining that the orientation of the device has changed from the first orientation to the second orientation. For example, the device may have been changed from a first orientation to a second orientation. Figure 2 The mobile phone mode shown in Figure 3 is changed to Figure 4 and Figure 5Additionally, or alternatively, determining that the first capture setting is changed to the second capture setting may include determining that an input request has been received to change the first capture setting to the second capture setting, for example, in the scenarios discussed in the second and third use cases. Or alternatively, the AI ​​system may determine that the first capture setting is changed to the second capture setting.

[0167] The method 600 may also include sending a descriptive metadata set indicating one or more capture parameters associated with the first capture setup to one or more remote devices. The sending may be performed via an out-of-band signal.

[0168] The capture parameters may include at least one of: a direction unit including the number of directions described by the spatial metadata, a channel unit including the number of transmission channels supported by the device, a source format describing the configuration of the device; and a variable description describing at least one of a capture type, an angle between two microphones of a plurality of microphones, a device size, and a microphone polar pattern (including an omnidirectional, cardioid, hypercardioid, or hypercardioid pattern). Optionally, the bitstream of the current streaming session includes one or more capture parameters.

[0169] Figure 7 A flow chart of a method according to an example embodiment is shown by way of example. Each unit of the flow chart may include one or more operations. The operations may be performed using hardware, software, firmware, or a combination thereof. For example, the operations may be performed individually or collectively by components, wherein the components may include at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the execution of the operations.

[0170] The method 700 includes a first operation 701 of receiving a bitstream for a current streaming session. Receiving a bitstream for a current streaming session may include receiving a bitstream from the apparatus 100 of FIGS. 1-5.

[0171] The method 700 includes a second operation 702 of decoding a bitstream for a current streaming session to determine a representation of a spatial audio signal and associated metadata, the metadata including in part a descriptive metadata set indicating one or more capture parameters associated with a first capture setup.

[0172] The method 700 comprises a third operation 703 of configuring the audio renderer according to the decoded descriptive metadata.

[0173] The method 700 comprises a fourth operation 704 of providing the representation of the spatial audio signal to an audio renderer for rendering to produce a rendered audio output signal.

[0174] The method 700 includes a fifth operation 705 of outputting the rendered audio output signal through a plurality of speakers.

[0175] The method 700 includes a sixth operation 706 of receiving an updated bitstream for the current streaming session and decoding the updated bitstream for the current streaming session to determine an updated representation of the spatial audio signal and associated metadata, the metadata including, in part, a descriptive metadata set indicating one or more capture parameters associated with the second capture setup. Configuring the audio renderer according to the decoded descriptive metadata includes changing at least one of the one or more capture parameters in the descriptive metadata set to be associated with the second capture setup.

[0176] The method may optionally further include receiving a descriptive metadata set indicating one or more capture parameters associated with the first capture setting to one or more remote devices via an out-of-band signal. In addition, configuring the audio renderer may also include configuring the audio renderer according to the descriptive metadata set indicating one or more capture parameters associated with the first capture setting.

[0177] Figure 8 800 is a flow chart of a method according to an example embodiment. Each unit of the flow chart may include one or more operations. The operations may be performed using hardware, software, firmware, or a combination thereof. For example, the operations may be performed individually or collectively by components, wherein the components may include at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the execution of the operations. Operations 801-807 may be performed, for example, on an apparatus as shown in FIGS. 1-5.

[0178] At step 801, a user starts a spatial audio capture, such as a spatial audio call. The capture system responds by defining an initial capture mode with a first capture setting set. At step 802, the capture system captures the spatial audio and generates an audio encoder input format based on the captured audio (and, for example, based on other system settings, such as the negotiated encoding mode of the codec supported by the spatial audio call). For example, the capture system generates a MASA format for an IVAS encoder.

[0179] In step 803, the capture system configures descriptive metadata parameters for the audio encoder format based on the initially selected capture mode and settings. These parameters are inserted as part of the encoder format to complete the input for the current time segment. For example, this will insert the MASA descriptive metadata parameters, as shown in the IVAS encoder time segment (i.e., 20 millisecond frames) in the above example.

[0180] In step 804, proceed to the next time period and process (e.g., 20 millisecond frames for IVAS). The capture system detects any input to adapt the spatial audio capture. For example, this can be user input, application input, etc. For example, the user can add a video recording, or select audio focus, or any other related input that can change at least some aspects related to the spatial audio capture. In step 806, if input is detected, the spatial audio capture is updated according to the new mode settings. If no input is detected, the previously selected mode settings are maintained. Again, in step 802, the system captures spatial audio and generates an audio encoder input format based on the captured audio. The spatial audio capture may now be different depending on whether the spatial audio capture is updated. In step 803, the capture system again configures the descriptive metadata parameters for the audio encoder format based on the currently selected capture mode and settings.

[0181] The audio codec format is encoded by the running audio codec at step 806, and the encoded audio data (bitstream) is transmitted at step 807. The encoded bitstream is received at a separate device at 808. The separate device may be any type of user equipment, as is the device that transmits the encoded bitstream.

[0182] On the receiving end, at step 809, the bitstream is received and decoded using a decoder. At step 810, the audio renderer is configured based on the descriptive metadata parameters sent and decoded. For example, in IVAS, these are MASA descriptive metadata parameters, as shown in the example above. There may also be other inputs, such as determining the output audio target based on the recipient's rendering device (e.g., headphone or speaker rendering). In addition, there may be some user input related to the rendering mode.

[0183] At step 811 , output audio is rendered based on at least the received spatial audio and the selected configuration, and finally at step 812 the audio is output.

[0184] Fig. 9 900 is a flowchart of a method according to an example embodiment. Each unit of the flowchart may include one or more operations. The operations may be performed using hardware, software, firmware, or a combination thereof. For example, the operations may be performed individually or collectively by components, wherein the components may include at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the execution of the operations.

[0185] Steps 901-912 correspond to Figure 8Steps 801-812 of the embodiment of the present invention are described in detail. In addition, there are new steps 906a and 908a. In step 906a, the descriptive metadata is provided to an external module that creates suitable out-of-band signaling, for example using RTP. Thus, step 906 is extended and the actual packetization will be different. In addition, in step 908a, on the receiving side, the descriptive metadata is received from the external module to obtain suitable out-of-band signaling, for example using RTP.

[0186] Figure 8 and Fig. 9 The capture system disclosed in the disclosure adapts spatial audio capture according to a selected effect or operating mode. In the present disclosure, the encoder input format is adaptively generated not only based on the selected spatial audio capture and the audio representation generated therefrom, but also using descriptive metadata parameters. Depending on the codec implementation, the encoder and / or a suitable external control module can utilize descriptive metadata, which can be related to, for example, an RTP payload format implementation (including RTP packetization in the example). The descriptive metadata parameter configuration according to the present disclosure can correspond to the physical properties of the spatial audio capture, or at least partially to the desired effect corresponding to the selected operating mode. For example, the descriptive metadata parameter selection guides the renderer implementation to process and render the spatial audio in a different manner, such as to make the effect stronger, clearer, etc.

[0187] The IVAS codec, or any other suitable immersive audio codec, may provide means for transmitting the necessary descriptive metadata to the receiver. Means may be provided to encode the descriptive metadata as part of the codec bitstream (as part of the "audio data"). Metadata signaling as part of the bitstream / payload may be referred to as in-band signaling.

[0188] The IVAS codec or any other suitable immersive audio codec may provide means for transmitting the necessary descriptive metadata to the receiver according to suitable out-of-band signaling. For example, in a preferred embodiment, descriptive metadata parameters or changes thereof may be sent using RTP. For example, an RTP header extension mechanism may be used. Fig.10 A block diagram is provided to aid in understanding the processing used in RTP.

[0189] Example Device

[0190] Fig.11An apparatus according to some example embodiments is shown, which may include an apparatus 100. The apparatus may be configured to perform the operations described herein, such as the operations described with reference to any disclosed process. The apparatus includes at least one processor 1100 and at least one memory 1101 directly or closely connected to the processor. The memory 1101 includes at least one random access memory (RAM) 1101a and at least one read-only memory (ROM) 1101b. Computer program code (software) 1105 is stored in the ROM 1101b. The apparatus may be connected to a transmitter (TX) and a receiver (RX). The apparatus may optionally be connected to a user interface (UI) for indicating the apparatus and / or for outputting data. At least one processor 1100, at least one memory 1101 and computer program code 1105 are arranged such that the apparatus performs at least a method according to any of the previous processes, such as as described with respect to Figure 6 And / or the flowchart of 7 and its related features disclosed.

[0191] Fig.12 A non-transitory medium 1200 according to some embodiments is shown. The non-transitory medium 1200 is a computer readable storage medium. It may be, for example, a CD, a DVD, a USB stick, a Blu-ray disc, etc. The non-transitory medium 1200 stores computer program code causing the apparatus to perform a method of any of the previous processes, for example as disclosed with respect to the flow charts and their related features.

[0192] The names of network elements, protocols and methods are based on current standards. In other versions or other technologies, the names of these network elements and / or protocols and / or methods may be different, as long as they provide the corresponding functions. For example, embodiments may be deployed in 2G / 3G / 4G / 5G networks and subsequent generations of 3GPP, and may also be deployed in non-3GPP radio networks, such as WiFi.

[0193] The memory can be volatile or non-volatile. It can be, for example, RAM, SRAM, Flash, FPGA block RAM, DCD, CD, USB stick, and Blu-ray disc.

[0194] Unless otherwise stated or clear from the context, the statement that two entities are different means that they perform different functions. This does not necessarily mean that they are based on different hardware. That is, each entity described in this specification can be based on different hardware, or some or all entities can be based on the same hardware. This does not necessarily mean that they are based on different software. That is, each entity described in this specification can be based on different software, or some or all entities can be based on the same software. Each entity described in this specification can be embodied in the cloud.

[0195] Implementations of any of the above blocks, devices, systems, techniques, or methods include, as non-limiting examples, implementations of hardware, software, firmware, dedicated circuits or logic, general purpose hardware or controllers or other computing devices, or some combination thereof. Some embodiments may be implemented in the cloud.

[0196] It should be understood that the above is what is presently considered to be the preferred embodiment. However, it should be noted that the description of the preferred embodiment is given by way of example only and that various modifications may be made without departing from the scope defined in the appended claims.

Claims

1. A device comprising: means for capturing a spatial audio signal using a first capture setup with a plurality of microphones; means for generating an audio encoder input format data set comprising a representation of said spatial audio signal and associated metadata, said metadata comprising in part a descriptive metadata set indicative of one or more capture parameters associated with said first capture setup; means for providing said audio encoder input format data set to an audio encoder for encoding said representation of said spatial audio signal and said associated metadata into a bitstream for a current streaming session; means for transmitting the bitstream to one or more remote devices for rendering the representation of the spatial audio signal based at least in part on the set of descriptive metadata; as well as means for determining during said current streaming session that said first capture setting is changed to a second capture setting, wherein, in response to a determination that the first capture setting is changed to a second capture setting, and when sending the bitstream, the means for capturing the spatial audio signal is configured to capture the spatial audio signal using the second capture setting, and the means for generating the audio encoder input format data is configured to change at least one of the one or more capture parameters of the descriptive metadata set to be associated with the second capture setting.

2. The device according to claim 1, wherein: The means for determining a change of the first capture setting to the second capture setting is configured to determine that the orientation of the apparatus has changed from a first orientation to a second orientation.

3. The device according to claim 1 or claim 2, wherein: The means for determining that the first capture setting is changed to the second capture setting is configured to determine that an input request to change the first capture setting to the second capture setting has been received.

4. The device according to any one of the preceding claims, further comprising: Means for transmitting the descriptive metadata set indicative of the one or more capture parameters associated with the first capture setting to the one or more remote devices via an out-of-band signal.

5. A device according to any one of the preceding claims, wherein: The one or more capture parameters include at least one of the following: a direction unit including the number of directions described by the spatial metadata; a channel unit including the number of transmission channels supported by the device; a source format describing the configuration of the device; as well as A description of variables describing at least one of a capture type, an angle between two microphones of the plurality of microphones, a device size, and a microphone polar pattern, the microphone polar pattern comprising an omnidirectional, cardioid, hypercardioid, or supercardioid pattern.

6. The device according to claim 5, wherein: The bitstream for the current streaming session includes the one or more capture parameters.

7. Apparatus according to any one of the preceding claims, wherein The means for generating the audio encoder input format data set is configured to use a Metadata Assisted Spatial Audio (MASA) format.

8. Apparatus according to any one of the preceding claims, wherein The audio encoder is an immersive voice and audio services codec.

9. The device according to any one of the preceding claims, wherein: The apparatus comprises user equipment.

10. An apparatus comprising: means for receiving a bitstream for a current streaming session; means for decoding the bitstream for the current streaming session to determine a representation of a spatial audio signal and associated metadata, the metadata comprising, in part, a descriptive metadata set indicative of one or more capture parameters associated with a first capture setup; A component for configuring an audio renderer based on decoded descriptive metadata; means for providing said representation of said spatial audio signal to said audio renderer for rendering to produce a rendered audio output signal; means for outputting said rendered audio output signal through a plurality of speakers; Means for receiving an updated bitstream for the current streaming session and means for decoding the updated bitstream for the current streaming session to determine an updated representation of a spatial audio signal and associated metadata, the metadata comprising, in part, a set of descriptive metadata indicating one or more capture parameters associated with a second capture setup, wherein the means for configuring the audio renderer based on the decoded descriptive metadata is configured to change at least one of the one or more capture parameters in the set of descriptive metadata to be associated with the second capture setup.

11. The device according to claim 10, wherein: The means for receiving a bitstream of a current streaming session receives the bitstream from an apparatus according to any one of claims 1 to 9.

12. The apparatus according to any one of claims 10 or 11, further comprising: means for receiving, via an out-of-band signal, the set of descriptive metadata indicative of one or more capture parameters associated with the first capture setting to the one or more remote devices; And wherein the means for configuring the audio renderer further comprises configuring the audio renderer according to the descriptive metadata set indicating one or more capture parameters associated with the first capture setting.

13. An apparatus according to any one of claims 10 to 12, wherein the apparatus comprises user equipment.

14. A method comprising: capturing a spatial audio signal using a first capture setup with a plurality of microphones; generating an audio encoder input format data set comprising a representation of the spatial audio signal and associated metadata, the metadata comprising in part a descriptive metadata set indicating one or more capture parameters associated with the first capture setup; providing the audio encoder input format data set to an audio encoder to encode the representation of the spatial audio signal and the associated metadata into a bitstream for a current streaming session; transmitting the bitstream to one or more remote devices to render the representation of the spatial audio signal based at least in part on the set of descriptive metadata; as well as determining during the current streaming session that the first capture setting changes to a second capture setting, wherein, in response to a determination to change the first capture setting to a second capture setting, and when sending the bitstream, capturing the spatial audio signal comprises capturing the spatial audio signal using the second capture setting, and generating the audio encoder input format data comprises changing at least one of the one or more capture parameters of the descriptive metadata set to be associated with the second capture setting.

15. A method comprising: receiving a bitstream for a current streaming session; decoding the bitstream for the current streaming session to determine a representation of a spatial audio signal and associated metadata, the metadata comprising, in part, a descriptive metadata set indicative of one or more capture parameters associated with a first capture setup; Configure the audio renderer based on the decoded descriptive metadata; providing the representation of the spatial audio signal to the audio renderer for rendering to produce a rendered audio output signal; outputting the rendered audio output signal through a plurality of speakers; receiving an updated bitstream for the current streaming session and decoding the updated bitstream for the current streaming session to determine an updated representation of a spatial audio signal and associated metadata, the metadata comprising, in part, a descriptive metadata set indicating one or more capture parameters associated with a second capture setup, wherein configuring the audio renderer based on the decoded descriptive metadata comprises changing at least one of the one or more capture parameters in the descriptive metadata set to be associated with the second capture setup.

Images