Complexity reduction in multi-stream audio

Abstract

A method comprising: obtaining at least one audio stream for multi-stream audio transmission; obtaining an indication related to the importance of the at least one audio stream in the multi-stream audio transmission; determining an importance value of the at least one audio stream based at least on the indication; comprising the at least one audio stream in a payload of a plurality of RTPs, and the importance values in headers of the RTPs; and transmitting the at least one audio stream to one or more devices participating in the multi-stream audio transmission.

Description

Technical Field

[0001] The present invention relates to reducing complexity in multi-stream audio environments, particularly in immersive audio. Background Art

[0002] Current audio codec developments, such as the 3GGP IVAS codec, aim to provide new immersive voice and audio services based on various network technologies, including 4G LTE (Long Term Evolution) and 5G NR (New Radio). Such immersive services include, for example, immersive voice and audio for communications and augmented / virtual / extended reality (AR / VR / XR). Another example of an immersive service is teleconferencing, which allows other participants to be positioned around the listener in the audio scene.

[0003] However, when delivering several streams, the immersive nature of multi-party conferencing and any other similar multi-stream transmission increases the burden for decoding and for rendering. It is expected that the IVAS codec supports more than one audio format used as input (i.e., component signals) to the encoding process (using one or more encoders). At the receiving end, each component signal used for the final rendering needs to be decoded, rendered, and combined separately. Therefore, it is usually necessary to activate multiple decoder instances. Switching from using one decoder instance to using two decoder instances will double the average computational complexity, and it can be expected that further switching from two decoder instances, for example, to using four decoder instances will further double the computational complexity. Therefore, the impact is very significant.

[0004] On the other hand, it can be assumed that not all audio streams are important at all times.Therefore, a method for reducing complexity in audio stream processing is needed. Summary of the Invention

[0005] Now, an improved method and a technical device for implementing the method have been invented, which alleviate the aforementioned problems. Various aspects include a method, an apparatus, and a non-transitory computer-readable medium (including a computer program or signal stored therein), characterized by what is stated in the independent claims. Various details of the embodiments are disclosed in the dependent claims, as well as in the corresponding figures and description.

[0006] The scope of protection sought by various embodiments of the present invention is defined by the independent claims. Embodiments and features described in this specification that do not fall within the scope of the independent claims (if any) should be interpreted as useful examples for understanding various embodiments of the present invention.

[0007] According to a first aspect, an apparatus is provided, comprising: means for obtaining at least one audio stream for a multi-stream audio transmission; means for obtaining an indication related to the importance of the at least one audio stream in the multi-stream audio transmission; means for determining an importance value of the at least one audio stream based at least on the indication; means for including the at least one audio stream in payloads of a plurality of real-time transport protocol packets and including the importance value in headers of the real-time transport protocol packets; and means for sending the at least one audio stream to one or more apparatuses participating in the multi-stream audio transmission.

[0008] According to an embodiment, the importance value is provided with a value range configured to indicate at least one value for important flows and at least one value for unimportant flows.

[0009] According to an embodiment, the importance value is configured as a plurality of values indicating different importances for the flow.

[0010] According to an embodiment, the importance value is configured to indicate the absolute importance of the flow.

[0011] According to an embodiment, the importance value is configured to indicate a relative importance of a stream among multiple streams of a multi-stream audio transmission.

[0012] According to an embodiment, the apparatus comprises means for sending said importance value using a Real-time Transport Protocol header extension.

[0013] According to an embodiment, the apparatus comprises means for including information about the jointly encoded multiple audio inputs in said real-time transport protocol header extension.

[0014] According to an embodiment, the apparatus comprises means for obtaining said indication related to the importance of said at least one audio stream from one or more of: a user input, an application input, a service input, or a device input.

[0015] A method according to a second aspect comprises: obtaining at least one audio stream for a multi-stream audio transmission; obtaining an indication related to importance of the at least one audio stream in the multi-stream audio transmission; determining an importance value of the at least one audio stream based at least on the indication; including the at least one audio stream in payloads of a plurality of real-time transport protocol packets and including the importance value in headers of the real-time transport protocol packets; and sending the at least one audio stream to one or more devices participating in the multi-stream audio transmission.

[0016] According to the third aspect, an apparatus includes: at least one processor and at least one memory, wherein the at least one memory has computer program code stored thereon, and the at least one memory and the computer program code are configured to, together with the at least one processor, cause the apparatus to at least perform the following operations: obtain at least one audio stream for multi-stream audio transmission; obtain an indication related to the importance of the at least one audio stream in the multi-stream audio transmission; determine an importance value of the at least one audio stream based at least on the indication; include the at least one audio stream in payloads of multiple real-time transport protocol packets and include the importance value in headers of the real-time transport protocol packets; and send the at least one audio stream to one or more apparatuses participating in the multi-stream audio transmission.

[0017] According to a fourth aspect, an apparatus is provided, comprising: means for receiving at least one audio stream in a payload of a plurality of real-time transport protocol packets; means for obtaining an importance value of the at least one audio stream from a header of the real-time transport protocol packets; and means for determining, based at least on the importance value, one or more parameters defining processing of the at least one audio stream for multi-stream audio transmission.

[0018] According to an embodiment, the apparatus comprises means for setting the decoding of at least one audio stream having a lower importance value to an inactive state.

[0019] According to an embodiment, the apparatus comprises means for obtaining at least one transmission parameter value; means for comparing the at least one transmission parameter value with corresponding transmission parameter values of at least one audio stream; and means for determining a processing option for the at least one audio stream based on at least one importance value and at least one transmission parameter value.

[0020] According to an embodiment, the processing options include at least one of the following:

[0021] - forwarding at least one audio stream intact to the receiver;

[0022] - Encoding at least one audio stream and packetizing the encoded audio stream for transmission to a receiver.

[0023] According to an embodiment, the device is an audio conferencing server or a bridge.

[0024] According to a fifth aspect, a method comprises: receiving at least one audio stream in a payload of a plurality of real-time transport protocol packets; obtaining an importance value of the at least one audio stream from a header of the real-time transport protocol packets; and determining one or more parameters defining processing of the at least one audio stream for multi-stream audio transmission based at least on the importance value.

[0025] According to the sixth aspect, an apparatus comprises: at least one processor and at least one memory, wherein the at least one memory has computer program code stored thereon, and the at least one memory and the computer program code are configured to, together with the at least one processor, cause the apparatus to at least perform: receiving at least one audio stream in payloads of a plurality of real-time transport protocol packets; obtaining an importance value of the at least one audio stream from a header of the real-time transport protocol packet; and determining, based at least on the importance value, one or more parameters defining processing of the at least one audio stream for multi-stream audio transmission.

[0026] According to a further aspect, a computer-readable storage medium comprises code for use by an apparatus, which, when executed by a processor, causes the apparatus to perform the above-described method. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] For a more complete understanding of the exemplary embodiments, reference will now be made to the following description taken in conjunction with the accompanying drawings, in which:

[0028] Figure 1a and Figure 1b Examples of multi-party conferencing implementations as a server-based approach and a peer-to-peer approach are shown respectively;

[0029] Figure 2 An exemplary case is shown, where the input to a multi-stream codec comprises four different component signals;

[0030] Figure 3 shows a flow chart for indicating the importance of an audio stream according to an embodiment;

[0031] Figure 4 shows a flow chart for determining the importance of a received audio stream according to an embodiment;

[0032] Figure 5 An example of a transmitting side of a teleconference server receiving at least two audio streams according to an embodiment is shown;

[0033] Figure 6 An example of a receiving side according to an embodiment is shown, wherein the receiving Figure 5 The transport configuration shown in ;

[0034] Figure 7 a flowchart illustrating an example decoding method according to an embodiment; and

[0035] Figure 8 A flow chart illustrating an example processing method according to an embodiment. DETAILED DESCRIPTION

[0036] The 3GGP IVAS (Immersive Voice and Audio Services) codec is an extension of the 3GPP EVS (Enhanced Voice Services) codec and is intended to provide new immersive voice and audio services based on contemporary and future communication networks such as 4G LTE (Long Term Evolution), 5G NR (New Radio), and 6G networks. Such immersive services include, for example, immersive voice and audio for augmented / virtual / extended reality (AR / VR / XR). This versatile audio codec is expected to handle encoding, decoding, and rendering of speech, music, and general audio. It will support channel-based audio input and scene-based audio input, including spatial information about the sound field and sound sources. It is also expected to operate with low latency to enable conversational services and support high error robustness under various transmission conditions.

[0037] Immersive audio capture and encoding can be achieved, for example, through microphone capture processing or equivalent preprocessing that generates an input signal based on multi-microphone mobile device audio capture, Ambisonics capture, or any other related capture. In addition, an audio file can be used to provide at least a portion of the input to the codec. The input signal is presented to the IVAS encoder in at least one of the supported audio formats, but the IVAS encoder can process multiple various allowed audio format combinations as input.

[0038] One of the audio formats supported by the IVAS encoder is the Metadata Assisted Spatial Audio (MASA) audio format. The MASA audio format consists of channels and spatial metadata. For example, there can be one or two channels (mono, stereo) and spatial metadata. The directional parameters of the audio source (such as its azimuth and elevation) and the energy ratio obtained by multi-channel analysis in the time-frequency domain are used to represent the spatial metadata. On the other hand, the directional metadata of the individual audio sources and audio objects can be processed in separate processing chains.

[0039] An important use case for modern communication codecs (including IVAS) is multi-party conferencing. Typically, multi-party conferencing can be achieved using server-based approaches such as Figure 1a ) or a point-to-point approach (as shown in Figure 1b In a server-based approach, a conference server / bridge, such as a multipoint control unit (MCU), controls the delivery and mixing of upstream and downstream audio signals between participant sites A, B, and C. Each participant site may have one or more users and one or more audio signal sources.

[0040] In the point-to-point approach, each participant site has a direct connection to every other participant site for delivering upstream and downstream audio signals, and the mixing of signals from different sites is performed by equipment at the participant sites.

[0041] Traditional voice conferencing provides a mono upstream signal and a mono downstream signal between connections (server or point-to-point). A mono upstream signal and a stereo (or other spatialized) downstream signal is also becoming a common option. With IVAS, each upstream and downstream signal can be spatial and therefore include several audio sources that can be distinguished by their direction. As mentioned above, more than one audio format can be used as input to the encoding process (using one or more encoders), for example, two mono audio objects and a MASA ambience signal that describes the overall spatial audio scene.

[0042] Upstream and downstream signals can be sent in a server-based as well as point-to-point manner using the Real-time Transport Protocol (RTP). RTP is designed to provide end-to-end, real-time transmission or streaming media, and provides functions / facilities for jitter compensation and detection of packet loss and out-of-order delivery. RTP allows data to be transmitted to multiple destinations via IP multicast or to a specific destination via IP unicast. Most RTP implementations are built on top of the User Datagram Protocol (UDP). Other transport protocols can also be used. RTP is used in conjunction with other protocols such as H.323 and the Real Time Streaming Protocol (RTSP).

[0043] The RTP specification describes two protocols: RTP and RTCP. RTP is used to transmit multimedia data, and RTCP is used to periodically send control information and QoS parameters.

[0044] An RTP session can be initiated between a client and a server using a signaling protocol such as H.323, Session Initiation Protocol (SIP), or RTSP. These protocols can use the Session Description Protocol (RFC 8866) to specify parameters for the session.

[0045] RTP is designed to carry a large number of multimedia formats, and this permits the development of new formats without the need to revise the RTP standard. For this reason, the information required for the specific application of the protocol is not included in the general RTP header. For a class of applications (for example, audio, video), the RTP configuration file can be defined. For media formats (for example, specific video encoding formats), the associated RTP payload format can be defined. Each instantiation of RTP in a specific application may require configuration files and payload format specifications.

[0046] The profile defines the codec used to encode the payload data and its mapping to the payload format codec in the protocol field Payload Type (PT) of the RTP header.

[0047] For example, the RTP profile with minimal controls for audio and video conferencing is defined in RFC 3551. This profile defines a set of static payload type assignments and a dynamic mechanism for mapping payload formats to PT values using the Session Description Protocol (SDP). The IVAS RTP payload and signaling will subsequently be specified in 3GPP TS 26.253, which is expected to provide a detailed algorithmic description of the IVAS codec, including its RTP payload format and SDP parameter definitions.

[0048] An RTP session is established for each multimedia stream. Audio (and video) streams can use separate RTP sessions, allowing the receiver to selectively receive components of a particular stream. The RTP specification recommends using even-numbered port numbers for RTP and using the next odd-numbered port number of the associated RTCP session. In applications where multiplexing protocols is used, a single port can be used for both RTP and RTCP.

[0049] Each RTP stream consists of RTP packets, which in turn consist of an RTP header and a payload portion.

[0050] The Session Description Protocol (SDP) is a format for describing multimedia communication sessions for the purpose of announcements and invitations. It is primarily used to support conversational and streaming applications. SDP itself does not deliver any media streams, but is used between endpoints to negotiate network metrics, media types, and other related characteristics. This set of characteristics and parameters is called a session profile. SDP is extensible to support new media types and formats.

[0051] The new audio codec features can significantly improve the user experience because the inherent audio quality is very high, creating a better sense of immersion, and it is possible to manipulate the audio scene by controlling the position, orientation, and relative volume of each source at the renderer.

[0052] Immersive multi-party conferencing and other multi-stream use cases will be crucial for the upcoming 3GPP IVAS codec, which is expected to be used on a wide variety of UEs, including devices with more constraints in terms of complexity and battery life, such as connected (audio) headsets, earbuds, etc.

[0053] However, the immersive nature of multi-party conferencing and any other similar multi-stream transmissions increases the burden for decoding and rendering when delivering several streams. Figure 2The diagram shows an exemplary case where the codec input contains four different component signals, namely, two mono audio object signals, a MASA ambient signal, and a first-order ambisonics (FOA) signal. Each component signal needs to be decoded, rendered, and combined separately for the final rendering. Therefore, in this example, at least three, possibly four, decoder instances need to be activated. Switching from one decoder instance to two decoder instances doubles the average computational complexity. It can be expected that switching from two decoder instances to four decoder instances will double the computational complexity again. Therefore, this effect is very significant.

[0054] On the other hand, it can be assumed that not all audio streams are important at all times.

[0055] In the following, an enhancement method for enabling reduction of computational complexity in audio signal processing and / or decoding will be described in more detail according to various embodiments.

[0056] exist Figure 3 The method disclosed in the disclosure includes: obtaining (300) at least one audio stream for a multi-stream audio transmission; obtaining (302) an indication related to the importance of the at least one audio stream in the multi-stream audio transmission; determining (304) an importance value of the at least one audio stream based at least on the indication; including (306) the at least one audio stream in payloads of a plurality of real-time transport protocol packets and including (306) the importance value in headers of the real-time transport protocol packets; and sending (308) the at least one audio stream to one or more devices participating in the multi-stream audio transmission.

[0057] Therefore, the method originates from the fact that not all audio streams are important at all times in multi-stream audio transmission (such as in an audio conference). Therefore, by determining the importance value of the audio stream and including the importance value in the header of the RTP packet carrying the audio stream, the receiving side (whether it is a conference server / bridge for multi-stream audio transmission or other participant device) can process the audio stream according to its importance, thereby reducing the computational complexity of decoding, for example.

[0058] Another aspect relates to the processing of such an audio stream upon reception, wherein the method is as follows Figure 4 The method includes receiving (400) at least one audio stream in payloads of a plurality of real-time transport protocol packets; obtaining (402) an importance value of the at least one audio stream from a header of the real-time transport protocol packets; and determining (404) one or more parameters defining processing of the at least one audio stream for multi-stream audio transmission based at least on the importance value.

[0059] Thus, based on an audio stream specific importance value, the receiving device can define appropriate processing for said audio stream.

[0060] In the following, these embodiments will be described by way of example with reference to audio conferencing as a suitable platform for implementing these methods and their embodiments. However, it should be noted that while audio conferencing is a common example of multi-stream (spatial) audio transmission / sending, receiving, decoding / rendering, it is only one example and not the only relevant use case. Even in a one-to-one call, multiple streams can be sent, or the service can add at least one stream in some use cases.

[0061] According to an embodiment, the method comprises setting the decoding of at least one audio stream having a lower importance value to an inactive state.

[0062] Therefore, the computational complexity at the receiving side (e.g., at the UE) can be reduced by using fewer decoders and rendering instances without degrading the quality of experience. In addition, the complexity of the rendered audio scene can be reduced at the receiving UE by setting the decoding of at least one less important stream to an inactive state to help the listener focus on more important audio scene components.

[0063] According to an embodiment, the importance value is provided with a value range configured to indicate at least one value for important flows and at least one value for unimportant flows.

[0064] Therefore, in the simplest case, only two importance values may be used: one for indicating important streams and another for indicating unimportant streams. In a simple implementation example, the important stream may refer to speech signals and the unimportant stream may refer to non-speech signals.

[0065] According to an embodiment, the importance value is configured as a plurality of values indicating different importances for the flow.

[0066] Providing a wider range of importance values enables more flexible processing of audio streams according to their importance.

[0067] According to an embodiment, the importance value is configured to indicate the absolute importance of the flow.

[0068] The absolute importance of a stream can be indicated by multiple code values. The absolute code indicates a specific importance value for each stream. The value can have an inherent meaning, specifying the kind of content the stream is currently carrying. Since no knowledge of the other streams of the multi-stream audio conference is required and defining importance can be based directly on the source signal or input from the user or service, an absolute value can always be provided. For example, an active voice signal can be considered important by default in any communication scenario, while a silent channel can always be considered unimportant since it will not affect the rendering anyway. Therefore, providing absolute values in the simplest way can be implemented with only two code values.

[0069] According to an embodiment, the importance value is configured to indicate a relative importance of a stream among multiple streams of a multi-stream audio transmission.

[0070] The relative importance of streams can also be indicated by multiple code values. Relative codes indicate the order of importance of a group of streams (not necessarily the group itself). Providing relative values essentially requires some knowledge of all streams. For example, sufficient side information transmission between nodes in a server-based architecture or a peer-to-peer architecture may be suitable for the use of relative codes. Relative codes can be used for subsets of streams, but it will be understood that ambiguity still exists between streams in different subsets.

[0071] According to an embodiment, the method comprises sending said importance value using a Real-time Transport Protocol header extension.

[0072] In the context of multi-stream audio conferencing, a single stream may also be referred to as a stream component. The stream component importance may be set in a codec-specific RTP header (such as an IVAS RTP header) using, for example, an RTP header extension mechanism as described in RFC8285. The RTP header extension mechanism enables the use of multiple identified extensions in an RTP packet without the need to formally register these extensions, but still avoids conflicts. Multiple header extension elements may be provided in a single RTP packet, wherein the extension elements are named by a URI. The Session Description Protocol (SDP) method may be used to map between named URIs and identifier values carried in RTP packets.

[0073] Here, either a one-byte or two-byte header can be used. The one-byte header format allows data lengths between 1 and 16 bytes (maximum rate 6.4 kbps), while the two-byte header format has data lengths between 0 and 255 bytes (maximum rate 102 kbps). An RTP packet with an RTP header extension will indicate whether it uses a one-byte or two-byte header extension. In the following, examples are described using a one-byte header format.

[0074] Each extension element begins with an identifier (ID) and a length value (len). In the following example, len=0. In the first example, the stream component importance information can be encoded using a one-byte header format as follows:

[0075]

[0076] The stream component importance information (i.e., CompImport) is an 8-bit code (256 possible values) that indicates the importance of a stream ID. The stream can define at least a portion of a spatial audio scene and it includes at least one component. Therefore, the importance value has been derived based on the highest importance of the current stream. The stream component importance information (i.e., CompImport) can be configured to indicate the absolute or relative importance of a stream.

[0077] Table 1 and Table 2 show examples of field code values for CompImport for absolute importance values and relative importance values, respectively.

[0078]

[0079] Table 1

[0080]

[0081] Table 2

[0082] In these examples, only the first 64 values and the first 128 values, respectively, are used.

[0083] According to an embodiment, the method comprises including information about the jointly encoded multiple audio inputs in said real-time transport protocol header extension.

[0084] Therefore, in an alternative implementation, a format-specific internal stream component importance flag F can be included in the RTP header extension. The value of this flag is "0" or "1", and it indicates further classification within the codec payload. Although CompImport (now reduced to 7 bits) indicates the overall stream importance, there may be additional information available that requires (at least partial) decoding of the payload. For example, a stereo signal may have different importance for the left (L) and right (R) channels. For example, a stereo signal may consist of two (essentially) unrelated signals, where the L channel carries speech and the R channel carries background audio. Obviously, this information cannot be used without first decoding the L and R channels, because actually using them separately requires access to the decoded channels. Therefore, by setting F=1, it can indicate that further decisions are possible through decoding. The overall importance of the stream is carried by a 7-bit indicator corresponding to the first example above (resulting in 128 possible values instead of 256).

[0085] In this second example, the stream component importance information may be encoded using a one-byte header format as follows:

[0086]

[0087] According to an embodiment, the method comprises obtaining said indication related to the importance of said at least one audio stream from one or more of: a user input, an application input, a service input, or a device input.

[0088] The code value may be determined at any relevant device / node, but is typically determined in the capture / sending UE and conference mixer / server. Decisions about code values may be based on factors such as, for example, user input via a user interface on the UE or conference application. For example, a user may indicate on their device which of a plurality of microphones used as input for a call is to be used for the user's voice. Such microphone input (defining, for example, a channel that is part of a stream or a complete stream) will be given a high default importance. On the other hand, the decision may also be based on signal analysis, for example, signal or voice activity detection. Thus, sometimes, a voice input that by default has a high importance may receive a lower importance setting due to inactivity. Such a change may also be caused by user input, for example, muting the audio or indicating a setting that is not present.

[0089] Typically, it can be expected that the stream component importance values are not updated frequently during a call / session. In particular, very large updates (e.g., from a value of "1" to a value of "63" in the table above) are typically infrequent.

[0090] In the following, some use case examples are given to further illustrate these embodiments. Figure 5 The example of the sending side of a teleconference server (such as an MCU) is shown, which receives at least two audio streams with optional side information. Bob and Jon are participants in a multi-user voice conference. The devices used by Bob and Jon (such as UEs) are both sending audio streams to the server. Bob's audio stream has side information indicating that Bob is speaking. The audio stream from Jon consists of two components: the voice stream (which is indicated as muted by the side information, i.e., Jon's microphone is muted) and the room environment signal (which is active).

[0091] Here, side information can be used as an indication of the importance of the audio stream. For example, the MCU can obtain side information from a conferencing application running on the MCU. Alternatively, the MCU can obtain side information, such as that analyzed locally by Bob and Jon's UEs. In another example, the MCU does not receive side information, or it only receives partial side information, based on which the MCU can perform audio analysis per component signal. This analysis can be performed independently or by a designated provider, such as in the cloud.

[0092] Based on side information or signal analysis, the MCU determines the importance value for each stream component (audio stream). The stream component importance value is set in the RTP header and is thus sent to the receiving user. In this example, two simple values "important" and "unimportant" are used.

[0093] In an alternative example, Bob and Jon may be connected to the same UE, such as a local conferencing system with multiple audio inputs. Separate stream component importance values may be set for the different audio inputs (such as microphones) used by Bob and Jon, respectively. Thus, stream component importance values may be set in any sending module or device.

[0094] Figure 6 An example of the receiving side is shown, where the received Figure 5 . The UE receives packets of Bob's audio stream and Jon's audio stream. The control module reads the RTP packet headers and, in this example, specifically, information related to the stream component importance of each stream being received. The control module determines, for each stream individually or for a group of streams collectively, whether each stream is to be decoded for rendering. The decision is based at least on the stream component importance values. Furthermore, at least one additional input (e.g., a local input requesting to limit power consumption and therefore the number of decoder instances) may be used as a trigger for making a decision about decoding a stream.

[0095] Here, Bob's stream (which is indicated as "important") will be decoded and rendered. Jon's audio stream will be discarded, and the decoder instance that would otherwise be needed will not be started. Thus, decoding complexity is reduced. However, completely discarding the audio stream from Jon will result in the receiving user also not being able to hear the ambient audio from Jon.

[0096] Therefore, when receiving packets, the control unit reads at least the stream component importance value CompImport from each RTP header. In some embodiments, based at least in part on this value, the receiving UE can drop certain streams from the decoding queue (i.e., the decoder instance for the stream can remain uninitialized / can be closed / can not be used). This simplifies decoding and rendering, and thus significantly reduces the associated complexity. On the other hand, at least one audio stream is not part of the rendering.

[0097] If an audio stream of a certain importance level is never decoded and rendered, then there's obviously no reason to send it. To reduce decoding and audio scene complexity, you can specify that certain frames be discarded based at least in part on the value of CompImport. In other words, further conditions can often be assumed. Such conditions can be input from the device, application, service, or user. If such conditions are not met, all frames are decoded normally.

[0098] Typically, the first three inputs (device, application, service) are related to reducing computational complexity, for example, to save battery. User input (and in some cases, application input) can also be related to reducing the complexity of the rendered scene. For example, a user can do other things while listening to a conference call, i.e., not paying special attention to the conference call. In this case, the user may want to have a listening experience that is as simple as possible. A simple way to achieve this is to remove less important audio streams, where the less important audio streams can be determined using stream component importance values.

[0099] In some cases, the application may receive information about other ongoing audio renderings. The user can control the audio scene, for example, using a slider or software button, to simplify the scene. Some application implementations may take this into account and automatically reduce the complexity of the rendered audio scene to make it more convenient for the user. At the same time, the computational load is also reduced.

[0100] Figure 7 A flowchart of an example decoding method is presented, wherein a receiving device (such as a UE) obtains (700) packets of at least one audio stream. It further obtains (702) at least one of the following from an RTP header of the packets of the at least one audio stream: an absolute importance value or a relative importance value. As a further condition, the device obtains an input (704) that is at least one of the following: user input, application input, service input, or device input. Furthermore, based on the at least one importance value and the at least one input, it is determined (706) whether to discard at least one audio stream corresponding to the at least one importance value. If the audio stream is not discarded, it is decoded and rendered (708). If the audio stream is discarded, the corresponding decoder instance can be closed (710).

[0101] Some embodiments can be used to handle constraints in conference scenarios, such as transmission constraints. For example, at least one audio source sends a stream to a conference bridge (MCU), which can mix and re-encode independent streams or forward independent streams to at least one other party. At least one of the connections to the other parties may involve bit rate limitations, for example due to network congestion. In this case, the MCU can process the content in one of two ways according to some of the above embodiments (or a combination thereof):

[0102] 1) The MCU obtains the value of CompImport for each stream. The MCU obtains the value of the available limited resources (e.g., total bitrate). The MCU determines to allocate the streams into at least two categories: 1) stream forwarding for streams of higher importance, and 2) re-encoding for streams of lower importance. As a subcategory for re-encoding, re-encoding may involve audio down-mixing, i.e., at least two streams are first decoded and then mixed together before being re-encoded. Based on this process, the available bitrate is allocated so that the more important streams are generally forwarded with the original encoding quality, while the less important streams are generally compressed into lower bitrate streams. Re-encoding also produces additional delay, which may or may not be compensated.

[0103] 2) The MCU obtains the value of CompImport for each stream. The MCU obtains the value of the available limited resources (e.g., total bit rate). The MCU determines to allocate the streams into at least two categories: 1) for streams of higher importance, the stream is sent to the receiver, and 2) for streams of lower importance, the stream is dropped. This is a similar process to that previously considered for the UE.

[0104] Note that both approaches above also benefit from reduced complexity. This reduction is achieved both at the conference bridge (fewer decoding and encoding operations) and at the receiving UE (fewer streams to decode and render). However, for the UE, the goal is simply to allow the highest possible quality for the most important streams when transmission resources are limited.

[0105] Figure 8A flowchart of an example processing method is presented, wherein the processing unit can be, for example, the MCU of FIG. 1 . The processing unit obtains (800) packets of at least one audio stream. It also obtains (802) at least one of the following from the RTP header of the packets of the at least one audio stream: an absolute importance value or a relative importance value. The processing unit further obtains (804) at least one transmission parameter value, which can be, for example, a total audio payload bit rate allocated for at least one downstream. The processing unit compares (806) the at least one transmission parameter value with corresponding transmission parameter values of the at least one audio stream. Furthermore, based on the at least one importance value and the at least one transmission parameter value, an allocation for processing the at least one audio stream is determined (808). The allocation for processing can result in forwarding the at least one audio stream to a receiver as is (810) or encoding and packetizing the at least one audio stream for transmission to the receiver (812).

[0106] Thus, the embodiments described herein enable reducing computational complexity at a receiving UE by allowing the receiving UE to inactive the decoding of at least one less important stream, thereby resulting in the use of fewer decoders and rendering instances and thus a reduced computational load. Furthermore, the complexity of the rendered audio scene can be reduced at the receiving UE by inactively decoding at least one less important stream to help the listener focus on more important audio scene components. Furthermore, audio stream mixing, encoding, and stream forwarding can be controlled at the receiving processing unit to allocate available bitrates without the need to decode and analyze each stream content individually.

[0107] According to one aspect of the present invention, an apparatus is configured to implement the above-described sending-side method, and possibly one or more embodiments related thereto. Thus, the apparatus comprises: means for obtaining at least one audio stream for multi-stream audio transmission; means for obtaining an indication related to the importance of the at least one audio stream in the multi-stream audio transmission; means for determining an importance value of the at least one audio stream based at least on the indication; means for including the at least one audio stream in payloads of a plurality of real-time transport protocol packets and including the importance value in headers of the real-time transport protocol packets; and means for transmitting the at least one audio stream to one or more apparatuses participating in the multi-stream audio transmission.

[0108] According to an embodiment, the importance value is provided with a value range configured to indicate at least one value for important flows and at least one value for unimportant flows.

[0109] According to an embodiment, the importance value is configured as a plurality of values indicating different importances for the flow.

[0110] According to an embodiment, the importance value is configured to indicate the absolute importance of the flow.

[0111] According to an embodiment, the importance value is configured to indicate a relative importance of a stream among multiple streams of a multi-stream audio transmission.

[0112] According to an embodiment, the apparatus comprises means for sending said importance value using a Real-time Transport Protocol header extension.

[0113] According to an embodiment, the apparatus comprises means for including information about the jointly encoded multiple audio inputs in said real-time transport protocol header extension.

[0114] According to an embodiment, the apparatus comprises means for obtaining said indication related to the importance of said at least one audio stream from one or more of: a user input, an application input, a service input, or a device input.

[0115] According to another aspect, an apparatus includes at least one processor and at least one memory having computer program code stored thereon, the at least one memory and the computer program code being configured to, together with the at least one processor, cause the apparatus to at least perform: obtaining at least one audio stream for multi-stream audio transmission; obtaining an indication related to importance of the at least one audio stream in the multi-stream audio transmission; determining an importance value of the at least one audio stream based at least on the indication; including the at least one audio stream in payloads of multiple real-time transport protocol packets and including the importance value in headers of the real-time transport protocol packets; and sending the at least one audio stream to one or more apparatuses participating in the multi-stream audio transmission.

[0116] According to an embodiment, the importance value is provided with a value range configured to indicate at least one value for important flows and at least one value for unimportant flows.

[0117] According to an embodiment, the importance value is configured as a plurality of values indicating different importances for the flow.

[0118] According to an embodiment, the importance value is configured to indicate the absolute importance of the flow.

[0119] According to an embodiment, the importance value is configured to indicate a relative importance of a stream among multiple streams of a multi-stream audio transmission.

[0120] According to an embodiment, the apparatus comprises code configured to cause the apparatus to transmit the importance value using a Real-time Transport Protocol header extension.

[0121] According to an embodiment, the apparatus comprises code configured to cause the apparatus to include information about the jointly encoded multiple audio inputs in the real-time transport protocol header extension.

[0122] According to an embodiment, the apparatus comprises code configured to cause the apparatus to obtain the indication related to the importance of the at least one audio stream from one or more of: user input, application input, service input, or device input.

[0123] An apparatus configured to implement a receiving-side method may include: a component for receiving at least one audio stream in a payload of a plurality of real-time transport protocol packets; a component for obtaining an importance value of the at least one audio stream from a header of the real-time transport protocol packet; and a component for determining one or more parameters defining processing of the at least one audio stream for a multi-stream audio conference based at least on the importance value.

[0124] According to an embodiment, the apparatus comprises means for setting the decoding of at least one audio stream having a lower importance value to an inactive state.

[0125] According to an embodiment, the apparatus comprises means for obtaining at least one transmission parameter value; means for comparing the at least one transmission parameter value with corresponding transmission parameter values of at least one audio stream; and means for determining a processing option for the at least one audio stream based on at least one importance value and at least one transmission parameter value.

[0126] According to an embodiment, the processing options include at least one of the following:

[0127] - forwarding at least one audio stream intact to the receiver;

[0128] - Encoding at least one audio stream and packetizing the encoded audio stream for transmission to a receiver.

[0129] According to an embodiment, the device is an audio conferencing server or a bridge.

[0130] According to another aspect, an apparatus includes at least one processor and at least one memory having computer program code stored thereon, the at least one memory and the computer program code being configured to, together with the at least one processor, cause the apparatus to at least perform: receiving at least one audio stream in payloads of a plurality of real-time transport protocol packets; obtaining an importance value of the at least one audio stream from a header of the real-time transport protocol packets; and determining, based at least on the importance value, one or more parameters defining processing of the at least one audio stream for a multi-stream audio conference.

[0131] According to an embodiment, the apparatus comprises code configured to cause the apparatus to set decoding of at least one audio stream having a lower importance value to an inactive state.

[0132] According to an embodiment, the apparatus comprises code configured to cause the apparatus to: obtain at least one transmission parameter value; compare the at least one transmission parameter value with corresponding transmission parameter values of at least one audio stream; and determine a processing option for the at least one audio stream based on at least one importance value and the at least one transmission parameter value.

[0133] According to an embodiment, the apparatus comprises code configured to cause the apparatus to: process at least one audio stream according to at least one of the following:

[0134] - forwarding at least one audio stream intact to the receiver;

[0135] - Encoding at least one audio stream and packetizing the encoded audio stream for transmission to a receiver.

[0136] In the above, some embodiments have been described with reference to encoders or encoding methods, it is to be understood that the resulting bitstream and decoder or decoding method may have corresponding elements. Similarly, where example embodiments have been described with reference to decoders, it is to be understood that the encoder may have a structure and / or computer program for generating a bitstream to be decoded by the decoder.

[0137] In general, various embodiments of the present invention may be implemented using hardware or dedicated circuits, or any combination thereof. Although various aspects of the present invention may be illustrated and described as block diagrams or using some other graphical representations, it is well known that the blocks, devices, systems, techniques, or methods described herein may be implemented, as non-limiting examples, using hardware, software, firmware, dedicated circuits or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0138] Embodiments of the present invention may be practiced in various components such as integrated circuit modules. The design of integrated circuits is generally a highly automated process. Complex and powerful software tools are available to convert a logic-level design into a semiconductor circuit design ready to be etched and formed on a semiconductor substrate.

[0139] Programs, such as those offered by Synopsys, Inc. of Mountain View, Calif., and Cadence Design, Inc. of San Jose, Calif., use well-established design rules and a library of pre-stored design blocks to automatically route conductors and position components on a semiconductor chip. Once the design of a semiconductor circuit is complete, the resulting design, in a standardized electronic format (e.g., Opus, GDSII, etc.), can be transferred to a semiconductor fabrication facility, or "fab," for fabrication.

[0140] The foregoing description has provided a complete and informative description of exemplary embodiments of the present invention by way of exemplary and non-limiting examples. However, various modifications and adaptations will become apparent to those skilled in the relevant arts in view of the foregoing description when read in conjunction with the accompanying drawings and examples. Nevertheless, all such and similar modifications of the teachings of the present invention will still fall within the scope of the present invention.

Claims

1. A device for sending, comprising: means for obtaining at least one audio stream for multi-stream audio transmission; means for obtaining an indication relating to the importance of said at least one audio stream in said multi-stream audio transmission; means for determining an importance value of the at least one audio stream based at least on the indication; means for including said at least one audio stream in payloads of a plurality of real-time transport protocol packets and including said importance value in headers of said real-time transport protocol packets; as well as Means for sending the at least one audio stream to one or more devices participating in the multi-stream audio transmission.

2. The device according to claim 1, wherein The importance value is provided with a value range configured to indicate at least one value for important flows and at least one value for unimportant flows.

3. The device according to claim 1 or 2, wherein: The importance value is configured as a plurality of values indicating different importances for a flow.

4. The device according to any one of the preceding claims, wherein The importance value is configured to indicate the absolute importance of the flow.

5. The device according to any one of the preceding claims, wherein The importance value is configured to indicate a relative importance of a stream among multiple streams of the multi-stream audio transmission.

6. The device according to any one of the preceding claims, comprising: Means for sending said importance value using a real-time transport protocol header extension.

7. The device according to any one of the preceding claims, comprising: Means for including information regarding the jointly encoded multiple audio inputs in said real-time transport protocol header extension.

8. The device according to any one of the preceding claims, comprising: Means for obtaining said indication related to said importance of said at least one audio stream from one or more of: user input, application input, service input, or device input.

9. An apparatus comprising at least one processor and at least one memory, wherein the at least one memory has computer program code stored thereon, the at least one memory and the computer program code being configured to, together with the at least one processor, cause the apparatus to at least perform: obtaining at least one audio stream for multi-stream audio transmission; obtaining an indication related to importance of the at least one audio stream in the multi-stream audio transmission; determining an importance value for the at least one audio stream based at least on the indication; including the at least one audio stream in payloads of a plurality of real-time transport protocol packets and including the importance value in headers of the real-time transport protocol packets; as well as The at least one audio stream is sent to one or more devices participating in the multi-stream audio transmission.

10. A method for sending, comprising: obtaining at least one audio stream for multi-stream audio transmission; obtaining an indication related to importance of the at least one audio stream in the multi-stream audio transmission; determining an importance value for the at least one audio stream based at least on the indication; including the at least one audio stream in payloads of a plurality of real-time transport protocol packets and including the importance value in headers of the real-time transport protocol packets; as well as The at least one audio stream is sent to one or more devices participating in the multi-stream audio transmission.

11. An apparatus for receiving, comprising: means for receiving at least one audio stream in a payload of a plurality of real-time transport protocol packets; means for obtaining an importance value of said at least one audio stream from a header of said real-time transport protocol packet; as well as Means for determining, based at least on the importance value, one or more parameters defining processing of the at least one audio stream for a multi-stream audio transmission.

12. The apparatus according to claim 11, comprising: Means for setting decoding of at least one audio stream having a lower importance value to an inactive state.

13. The device according to claim 11 or 12, comprising: means for obtaining a value of at least one transmission parameter; means for comparing said at least one transmission parameter value with a corresponding transmission parameter value of said at least one audio stream; as well as Means for determining a processing option for said at least one audio stream based on said at least one importance value and said at least one transmission parameter value.

14. The device according to claim 13, wherein The processing options include at least one of the following: - forwarding the at least one audio stream intact to a receiver; - encoding the at least one audio stream and packetizing the encoded audio stream for transmission to a receiver.

15. The device according to claim 13 or 14, wherein The device is an audio conferencing server or bridge.

16. An apparatus comprising at least one processor and at least one memory, the at least one memory having computer program code stored thereon, the at least one memory and the computer program code being configured to, together with the at least one processor, cause the apparatus to at least perform: receiving at least one audio stream in a payload of a plurality of real-time transport protocol packets; Obtaining an importance value of the at least one audio stream from a header of the real-time transport protocol packet; as well as Based at least on the importance value, one or more parameters defining processing for the at least one audio stream of the multi-stream audio conference are determined.

17. A method for receiving, comprising: receiving at least one audio stream in a payload of a plurality of real-time transport protocol packets; Obtaining an importance value of the at least one audio stream from a header of the real-time transport protocol packet; as well as Based at least on the importance value, one or more parameters defining processing of the at least one audio stream for a multi-stream audio transmission are determined.

Images