Immersive conversational audio

Abstract

An apparatus for controlling an immersive audio transmission comprising at least partially a bidirectional signalling, the apparatus comprising means configured to: obtain at least one immersive audio payload comprising at least one immersive audio coded bitstream; obtain at least one reverse direction processing information, the at least one reverse direction processing information for providing at least in part the bidirectional signalling; and packetize the at least one reverse direction processing information in a transport protocol payload.

Description

[0001] IMMERSIVE CONVERSATIONAL AUDIO

[0002] Field

[0003] The present application relates to apparatus and methods for bidirectional signalling for immersive conversational audio, but not exclusively for implementing bidirectional signaling for immersive conversational audio employing immersive voice and audio services (IVAS) real-time transport protocol (RTP) payload.

[0004] Background

[0005] Immersive audio codecs are being implemented supporting a multitude of operating points ranging from a low bit rate operation to transparency. An example of such a codec is the immersive voice and audio services (IVAS) codec which is being designed to be suitable for use over a communications network such as a 3GPP 4G / 5G network. Such immersive services include uses for example in immersive voice and audio for applications such as virtual reality (VR), augmented reality (AR) and mixed reality (MR) as well as spatial voice communication including teleconferencing. This audio codec is expected to handle the encoding, decoding and rendering of speech, music and generic audio. It is furthermore expected to support channel-based audio and scene-based audio inputs including spatial information about the sound field and sound sources. The codec is also expected to operate with low latency to enable conversational services as well as support high error robustness under various transmission conditions.

[0006] The input signals are presented to the IVAS encoder in one of the supported formats (and in some allowed combinations of the formats). Similarly, it is expected that the decoder can output the audio in supported formats. A pass-through mode has been proposed, where the audio could be provided in its original format after transmission (encoding / decoding).

[0007] Additionally RTP (Real-Time Transport Protocol) is intended for an end-to- end, real-time transfer of streaming media and provides facilities for jitter compensation and detection of packet loss and out-of-order delivery. RTP allows data transfer to multiple destinations through IP multicast or to a specific destination through IP unicast. The majority of the RTP implementations are built on top of the User Datagram Protocol (UDP). Other transport protocols may also be utilized. RTP is used in together with other protocols such as H.323 and Real Time Streaming Protocol (RTSP).

[0008] The RTP specification describes two protocols: RTP and RTCP. RTP is used for the transfer of multimedia data, and its companion protocol (RTCP) is used to periodically send control information and QoS (Quality of Service) parameters.

[0009] RTP sessions are typically initiated between client and server or between client and another client (or a multi-party topology) using a signalling protocol, such as H.323, the Session Initiation Protocol (SIP), or RTSP. These protocols typically use the Session Description Protocol (SDP), such as defined by RFC 8866 to specify parameters for the sessions.

[0010] Summary

[0011] There is provided according to a first aspect an apparatus for controlling an immersive audio transmission comprising at least partially a bidirectional signalling, the apparatus comprising means configured to: obtain at least one immersive audio payload comprising at least one immersive audio coded bitstream; obtain at least one reverse direction processing information, the at least one reverse direction processing information for providing at least in part the bidirectional signalling; and packetize the at least one reverse direction processing information in a transport protocol payload.

[0012] The at least one immersive audio payload may further comprise at least one processing information frame for assisting the processing of the at least one immersive audio coded bitstream.

[0013] The at least one reverse direction processing information may be associated with the at least one immersive audio payload.

[0014] The means configured to packetize the at least one reverse direction processing information in a transport protocol payload may be configured to packetize the at least one reverse direction processing information in at least one of: a real-time transport protocol payload; a real-time transport protocol header extension; and a real-time transport control protocol payload.

[0015] The means may be further configured to transmit the packetized at least one reverse direction processing information to a further apparatus. The means configured to transmit the packetized at least one reverse direction processing information may be configured to transmit the packetized at least one reverse direction processing information as part of a packet comprising at least one further at least one immersive audio coded bitstream.

[0016] The means configured to transmit the packetized at least one reverse direction processing information may be configured to transmit the packetized the at least one reverse direction processing information separate to a further packet comprising at least one further at least one immersive audio coded bitstream.

[0017] The means may be further configured to obtain at least one response processing information with an identifier header, the at least one response processing information being obtained in response to the at least one reverse direction processing information.

[0018] The means configured to obtain the at least one response processing information may be configured to obtain the at least one response processing information within at least one of: a real-time transport protocol payload; a real-time transport protocol header extension; and a real-time transport control protocol payload.

[0019] The means configured to obtain the at least one reverse direction processing information may be configured to generate at least one associated identifier header.

[0020] The at least one reverse direction processing information may comprise at least one of: an action request; a request to change an IVAS input format; a request to change a coded format; a request for an orientation of the further apparatus; a request for configuration information relating to a microphone array constellation of the further apparatus; a request to switch a bitrate from the further apparatus to the apparatus; a request for binauralization data that can be used to revert the binauralization at the apparatus; information relating to the apparatus, for assisting the at least one further apparatus in encoding the at least one immersive audio coded bitstream; information relating to a class of the reverse direction processing information; information relating to the apparatus orientation and / or position; information relating to a head rotation of a listener of the at least one immersive audio coded bitstream; information relating to a position of a listener of the at least one immersive audio coded bitstream; information relating to a loudspeaker constellation associated with the apparatus; information relating to a change in an output audio format; information relating to a combined orientation applied at the apparatus; and information related to reverberation parameters applied at the apparatus.

[0021] The at least one reverse direction processing information may further comprise at least one of: a time value indicating a time of the reverse direction processing information, to cause the further apparatus to process the reverse direction processing information based on the time value; an urgency value, to cause the further apparatus to process the reverse direction processing information relative to a further reverse direction processing information based on the urgency value; a priority value, to cause the further apparatus to process the reverse direction processing information relative to a further reverse direction processing information based on the priority value.

[0022] The at least one reverse direction processing information may further comprise at least one usage value, the usage value caused to identify whether the information expects or does not expect a response.

[0023] The means configured to packetize the at least one reverse direction processing information in a transport protocol payload may be configured to perform one of: attach an identifier to the header of the transport protocol payload; attach at least one reverse direction processing information frame to the transport protocol payload; and attach at least one response PI frame and a related payload header to an IVAS real-time transport protocol payload.

[0024] The at least one reverse direction processing information may comprise at least one of: information identifying an orientation of further apparatus or capturing apparatus; information identifying a microphone array constellation configuration of further apparatus or capturing apparatus; information identifying an orientation applied to the at least one immersive audio coded bitstream; information identifying a subsequent input format change for the at least one immersive audio coded bitstream; information identifying a subsequent coded format change for the at least one immersive audio coded bitstream; information identifying a specified number of frames or number of seconds until a change in an input format for the at least one immersive audio coded bitstream; information identifying binauralization data that can be used to revert the binauralization at the apparatus; and information about an acoustic environment. The means may be further configured to negotiate with a further apparatus support for the reverse direction processing information.

[0025] The means configured to negotiate with the further apparatus may be configured to negotiate employing a session description file.

[0026] According to a second aspect there is provided an apparatus for controlling bidirectional audio transmission comprising at least partially a bidirectional signalling, the apparatus comprising means configured to: transmit at least one immersive audio payload comprising at least one immersive audio coded bitstream; obtain packetized at least one reverse direction processing information, the packetized at least one reverse direction processing information for providing at least in part the bidirectional signalling; and process the packetized at least one reverse direction processing information.

[0027] The at least one immersive audio payload may further comprise at least one processing information frame for assisting the processing of the at least one immersive audio coded bitstream.

[0028] The at least one reverse direction processing information may be associated with the at least one immersive audio payload.

[0029] The means configured to obtain the packetized at least one reverse direction processing information may be further configured to obtain the packetized at least one reverse direction processing information as part of at least one of: a real-time transport protocol payload; a real-time transport protocol header extension; and a real-time transport control protocol payload.

[0030] The means may be further configured to obtain the packetized at least one reverse direction processing information from a further apparatus.

[0031] The means configured to obtain the packetized at least one reverse direction processing information may be configured to obtain the packetized at least one reverse direction processing information as part of a packet comprising at least one further at least one immersive audio coded bitstream.

[0032] The means configured to obtain the packetized at least one reverse direction processing information may be configured to obtain the packetized at least one reverse direction processing information separate to a further packet comprising at least one further at least one immersive audio coded bitstream. The means may be further configured to transmit at least one response processing information with an identifier header, the at least one response processing information being generated in response to the processing of the at least one reverse direction processing information.

[0033] The means configured to transmit the at least one response processing information may be configured to transmit the at least one response processing information within at least one of: a real-time transport protocol payload; a real-time transport protocol header extension; and a real-time transport control protocol payload.

[0034] The means configured to obtain the at least one reverse direction processing information may be configured to obtain at least one associated identifier header configured to identify the at least one reverse direction processing information associated with the at least one immersive audio payload.

[0035] The reverse direction processing information may comprise at least one of: an action request; a request to change an IVAS input format; a request to change a coded format; a request for an orientation of the apparatus; a request for configuration information relating to a microphone array constellation of the apparatus; a request to switch a bitrate from the apparatus to the further apparatus; a request for binauralization data that can be used to revert the binauralization at the further apparatus; information relating to the further apparatus, for assisting the apparatus in encoding the at least one immersive audio coded bitstream; information relating to the apparatus orientation and / or position; information relating to a class of the reverse direction processing information; information relating to a head rotation of a listener of the at least one immersive audio coded bitstream; information relating to a position of a listener of the at least one immersive audio coded bitstream; information relating to a loudspeaker constellation associated with the further apparatus; information relating to a change in an output audio format; information relating to a combined orientation applied at the further apparatus; and information related to reverberation parameters applied at the further apparatus.

[0036] The reverse direction processing information may further comprise at least one of: a time value indicating a time of the reverse direction processing information, to cause the apparatus to process the reverse direction processing information based on the time value; an urgency value, to cause the apparatus to process the reverse direction processing information relative to a further reverse direction processing information based on the urgency value; a priority value, to cause the apparatus to process the reverse direction processing information relative to a further reverse direction processing information based on the priority value.

[0037] The at least one reverse direction processing information may further comprise at least one usage value, the usage value caused to identify whether the information expects or does not expect a response.

[0038] The reverse direction processing information may comprise at least one of: information identifying an orientation of the apparatus or a capturing apparatus; information identifying a microphone array constellation configuration of the apparatus or a capturing apparatus; information identifying an orientation applied to the at least one immersive audio coded bitstream; information identifying a subsequent changes input format for the at least one immersive audio coded bitstream; information identifying a subsequent coded format change for the at least one immersive audio coded bitstream; information identifying a specified number of frames or number of seconds until a change in an input format for the at least one immersive audio coded bitstream; information identifying binauralization data that can be used to revert the binauralization at the apparatus; and information about an acoustic environment.

[0039] The means may be further configured to negotiate with a further apparatus support for the reverse direction processing information.

[0040] The means configured to negotiate with the further apparatus may be configured to negotiate employing a session description file.

[0041] According to a third aspect there is provided a method for an apparatus for controlling an immersive audio transmission comprising at least partially a bidirectional signalling, the method comprising: obtaining at least one immersive audio payload comprising at least one immersive audio coded bitstream; obtaining at least one reverse direction processing information, the at least one reverse direction processing information for providing at least in part the bidirectional signalling; and packetizing the at least one reverse direction processing information in a transport protocol payload. The at least one immersive audio payload may further comprise at least one processing information frame for assisting the processing of the at least one immersive audio coded bitstream.

[0042] The at least one reverse direction processing information may be associated with the at least one immersive audio payload.

[0043] Packetizing the at least one reverse direction processing information in a transport protocol payload may comprise packetizing the at least one reverse direction processing information in at least one of: a real-time transport protocol payload; a real-time transport protocol header extension; and a real-time transport control protocol payload.

[0044] The method may further comprise transmitting the packetized at least one reverse direction processing information to a further apparatus.

[0045] Transmitting the packetized at least one reverse direction processing information may comprise transmitting the packetized at least one reverse direction processing information as part of a packet comprising at least one further at least one immersive audio coded bitstream.

[0046] Transmitting the packetized at least one reverse direction processing information may comprise transmitting the packetized the at least one reverse direction processing information separate to a further packet comprising at least one further at least one immersive audio coded bitstream.

[0047] The method may further comprise obtaining at least one response processing information with an identifier header, the at least one response processing information being obtained in response to the at least one reverse direction processing information.

[0048] Obtaining the at least one response processing information may comprise obtaining the at least one response processing information within at least one of: a real-time transport protocol payload; a real-time transport protocol header extension; and a real-time transport control protocol payload.

[0049] Obtaining the at least one reverse direction processing information may comprise generating at least one associated identifier header.

[0050] The at least one reverse direction processing information may comprise at least one of: an action request; a request to change an IVAS input format; a request to change a coded format; a request for an orientation of the further apparatus; a request for configuration information relating to a microphone array constellation of the further apparatus; a request to switch a bitrate from the further apparatus to the apparatus; a request for binauralization data that can be used to revert the binauralization at the apparatus; information relating to the apparatus, for assisting the at least one further apparatus in encoding the at least one immersive audio coded bitstream; information relating to a class of the reverse direction processing information; information relating to the apparatus orientation and / or position; information relating to a head rotation of a listener of the at least one immersive audio coded bitstream; information relating to a position of a listener of the at least one immersive audio coded bitstream; information relating to a loudspeaker constellation associated with the apparatus; information relating to a change in an output audio format; information relating to a combined orientation applied at the apparatus; and information related to reverberation parameters applied at the apparatus.

[0051] The at least one reverse direction processing information may further comprise at least one of: a time value indicating a time of the reverse direction processing information, to cause the further apparatus to process the reverse direction processing information based on the time value; an urgency value, to cause the further apparatus to process the reverse direction processing information relative to a further reverse direction processing information based on the urgency value; a priority value, to cause the further apparatus to process the reverse direction processing information relative to a further reverse direction processing information based on the priority value.

[0052] The at least one reverse direction processing information may further comprise at least one usage value, the usage value caused to identify whether the information expects or does not expect a response.

[0053] Packetizing the at least one reverse direction processing information in a transport protocol payload may comprise one of: attaching an identifier to the header of the transport protocol payload; attaching at least one reverse direction processing information frame to the transport protocol payload; and attaching at least one response processing information frame and a related payload header to an IVAS real-time transport protocol payload. The at least one reverse direction processing information may comprise at least one of: information identifying an orientation of further apparatus or capturing apparatus; information identifying a microphone array constellation configuration of further apparatus or capturing apparatus; information identifying an orientation applied to the at least one immersive audio coded bitstream; information identifying a subsequent input format change for the at least one immersive audio coded bitstream; information identifying a subsequent coded format change for the at least one immersive audio coded bitstream; information identifying a specified number of frames or number of seconds until a change in an input format for the at least one immersive audio coded bitstream; information identifying binauralization data that can be used to revert the binauralization at the apparatus; and information about an acoustic environment.

[0054] The method may further comprise negotiating with a further apparatus support for the reverse direction processing information.

[0055] Negotiating with the further apparatus may comprise negotiating employing a session description file.

[0056] According to a fourth aspect there is provided a method for an apparatus for controlling bidirectional audio transmission comprising at least partially a bidirectional signalling, the method comprising: transmitting at least one immersive audio payload comprising at least one immersive audio coded bitstream; obtaining packetized at least one reverse direction processing information, the packetized at least one reverse direction processing information for providing at least in part the bidirectional signalling; and processing the packetized at least one reverse direction processing information.

[0057] The at least one immersive audio payload may further comprise at least one processing information frame for assisting the processing of the at least one immersive audio coded bitstream.

[0058] The at least one reverse direction processing information may be associated with the at least one immersive audio payload.

[0059] Obtaining the packetized at least one reverse direction processing information may further comprise obtaining the packetized at least one reverse direction processing information as part of at least one of: a real-time transport protocol payload; a real-time transport protocol header extension; and a real-time transport control protocol payload.

[0060] The method may further comprise obtaining the packetized at least one reverse direction processing information from a further apparatus.

[0061] Obtaining the packetized at least one reverse direction processing information may comprise obtaining the packetized at least one reverse direction processing information as part of a packet comprising at least one further at least one immersive audio coded bitstream.

[0062] Obtaining the packetized at least one reverse direction processing information may comprise obtaining the packetized at least one reverse direction processing information separate to a further packet comprising at least one further at least one immersive audio coded bitstream.

[0063] The method may further comprise transmitting at least one response processing information with an identifier header, the at least one response processing information being generated in response to the processing of the at least one reverse direction processing information.

[0064] Transmitting the at least one response processing information may comprise transmitting the at least one response processing information within at least one of: a real-time transport protocol payload; a real-time transport protocol header extension; and a real-time transport control protocol payload.

[0065] Obtaining the at least one reverse direction processing information may comprise obtaining at least one associated identifier header configured to identify the at least one reverse direction processing information associated with the at least one immersive audio payload.

[0066] The reverse direction processing information may comprise at least one of: an action request; a request to change an IVAS input format; a request to change a coded format; a request for an orientation of the apparatus; a request for configuration information relating to a microphone array constellation of the apparatus; a request to switch a bitrate from the apparatus to the further apparatus; a request for binauralization data that can be used to revert the binauralization at the further apparatus; information relating to the further apparatus, for assisting the apparatus in encoding the at least one immersive audio coded bitstream; information relating to the apparatus orientation and / or position; information relating to a class of the reverse direction processing information; information relating to a head rotation of a listener of the at least one immersive audio coded bitstream; information relating to a position of a listener of the at least one immersive audio coded bitstream; information relating to a loudspeaker constellation associated with the further apparatus; information relating to a change in an output audio format; information relating to a combined orientation applied at the further apparatus; and information related to reverberation parameters applied at the further apparatus.

[0067] The reverse direction processing information may further comprise at least one of: a time value indicating a time of the reverse direction processing information, to cause the apparatus to process the reverse direction processing information based on the time value; an urgency value, to cause the apparatus to process the reverse direction processing information relative to a further reverse direction processing information based on the urgency value; a priority value, to cause the apparatus to process the reverse direction processing information relative to a further reverse direction processing information based on the priority value.

[0068] The at least one reverse direction processing information may further comprise at least one usage value, the usage value caused to identify whether the information expects or does not expect a response.

[0069] The reverse direction processing information may comprise at least one of: information identifying an orientation of the apparatus or a capturing apparatus; information identifying a microphone array constellation configuration of the apparatus or a capturing apparatus; information identifying an orientation applied to the at least one immersive audio coded bitstream; information identifying a subsequent changes input format for the at least one immersive audio coded bitstream; information identifying a subsequent coded format change for the at least one immersive audio coded bitstream; information identifying a specified number of frames or number of seconds until a change in an input format for the at least one immersive audio coded bitstream; information identifying binauralization data that can be used to revert the binauralization at the apparatus; and information about an acoustic environment.

[0070] The method may further comprise negotiating with a further apparatus support for the reverse direction processing information. Negotiating with the further apparatus may comprise negotiating employing a session description file.

[0071] According to a fifth aspect there is provided an apparatus for controlling an immersive audio transmission comprising at least partially a bidirectional signalling, the apparatus comprising at least one processor and at least one memory including a computer program code, the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus at least to: obtain at least one immersive audio payload comprising at least one immersive audio coded bitstream; obtain at least one reverse direction processing information, the at least one reverse direction processing information for providing at least in part the bidirectional signalling; and packetize the at least one reverse direction processing information in a transport protocol payload.

[0072] The at least one immersive audio payload may further comprise at least one processing information frame for assisting the processing of the at least one immersive audio coded bitstream.

[0073] The at least one reverse direction processing information may be associated with the at least one immersive audio payload.

[0074] The apparatus caused to packetize the at least one reverse direction processing information in a transport protocol payload may be caused to packetize the at least one reverse direction processing information in at least one of: a realtime transport protocol payload; a real-time transport protocol header extension; and a real-time transport control protocol payload.

[0075] The apparatus may be further caused to transmit the packetized at least one reverse direction processing information to a further apparatus.

[0076] The apparatus caused to transmit the packetized at least one reverse direction processing information may be further caused to transmit the packetized at least one reverse direction processing information as part of a packet comprising at least one further at least one immersive audio coded bitstream.

[0077] The apparatus caused to transmit the packetized at least one reverse direction processing information may be caused to transmit the packetized the at least one reverse direction processing information separate to a further packet comprising at least one further at least one immersive audio coded bitstream. The apparatus may be further caused to obtain at least one response processing information with an identifier header, the at least one response processing information being obtained in response to the at least one reverse direction processing information.

[0078] The apparatus caused to obtain the at least one response processing information may be caused to obtain the at least one response processing information within at least one of: a real-time transport protocol payload; a real-time transport protocol header extension; and a real-time transport control protocol payload.

[0079] The apparatus caused to obtain the at least one reverse direction processing information may be caused to generate at least one associated identifier header.

[0080] The at least one reverse direction processing information may comprise at least one of: an action request; a request to change an IVAS input format; a request to change a coded format; a request for an orientation of the further apparatus; a request for configuration information relating to a microphone array constellation of the further apparatus; a request to switch a bitrate from the further apparatus to the apparatus; a request for binauralization data that can be used to revert the binauralization at the apparatus; information relating to the apparatus, for assisting the at least one further apparatus in encoding the at least one immersive audio coded bitstream; information relating to a class of the reverse direction processing information; information relating to the apparatus orientation and / or position; information relating to a head rotation of a listener of the at least one immersive audio coded bitstream; information relating to a position of a listener of the at least one immersive audio coded bitstream; information relating to a loudspeaker constellation associated with the apparatus; information relating to a change in an output audio format; information relating to a combined orientation applied at the apparatus; and information related to reverberation parameters applied at the apparatus.

[0081] The at least one reverse direction processing information may further comprise at least one of: a time value indicating a time of the reverse direction processing information, to cause the further apparatus to process the reverse direction processing information based on the time value; an urgency value, to cause the further apparatus to process the reverse direction processing information relative to a further reverse direction processing information based on the urgency value; a priority value, to cause the further apparatus to process the reverse direction processing information relative to a further reverse direction processing information based on the priority value.

[0082] The at least one reverse direction processing information may further comprise at least one usage value, the usage value caused to identify whether the information expects or does not expect a response.

[0083] The apparatus caused to packetize the at least one reverse direction processing information in a transport protocol payload may be caused to perform one of: attach an identifier to the header of the transport protocol payload; attach at least one reverse direction processing information frame to the transport protocol payload; and attach at least one response processing information frame and a related payload header to an IVAS real-time transport protocol payload.

[0084] The at least one reverse direction processing information may comprise at least one of: information identifying an orientation of further apparatus or capturing apparatus; information identifying a microphone array constellation configuration of further apparatus or capturing apparatus; information identifying an orientation applied to the at least one immersive audio coded bitstream; information identifying a subsequent input format change for the at least one immersive audio coded bitstream; information identifying a subsequent coded format change for the at least one immersive audio coded bitstream; information identifying a specified number of frames or number of seconds until a change in an input format for the at least one immersive audio coded bitstream; information identifying binauralization data that can be used to revert the binauralization at the apparatus; and information about an acoustic environment.

[0085] The apparatus may be further caused to negotiate with a further apparatus support for the reverse direction processing information.

[0086] The apparatus caused to negotiate with the further apparatus may be caused to negotiate employing a session description file.

[0087] According to a sixth aspect there is provided an apparatus for controlling bidirectional audio transmission comprising at least partially a bidirectional signalling, the apparatus comprising at least one processor and at least one memory including a computer program code, the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus at least to: transmit at least one immersive audio payload comprising at least one immersive audio coded bitstream; obtain packetized at least one reverse direction processing information, the packetized at least one reverse direction processing information for providing at least in part the bidirectional signalling; and process the packetized at least one reverse direction processing information.

[0088] The at least one immersive audio payload may further comprise at least one processing information frame for assisting the processing of the at least one immersive audio coded bitstream.

[0089] The at least one reverse direction processing information may be associated with the at least one immersive audio payload.

[0090] The apparatus caused to obtain the packetized at least one reverse direction processing information may be further caused to obtain the packetized at least one reverse direction processing information as part of at least one of: a real-time transport protocol payload; a real-time transport protocol header extension; and a real-time transport control protocol payload.

[0091] The apparatus may be further caused to obtain the packetized at least one reverse direction processing information from a further apparatus.

[0092] The apparatus caused to obtain the packetized at least one reverse direction processing information may be caused to obtain the packetized at least one reverse direction processing information as part of a packet comprising at least one further at least one immersive audio coded bitstream.

[0093] The apparatus caused to obtain the packetized at least one reverse direction processing information may be caused to obtain the packetized at least one reverse direction processing information separate to a further packet comprising at least one further at least one immersive audio coded bitstream.

[0094] The apparatus may be caused to transmit at least one response processing information with an identifier header, the at least one response processing information being generated in response to the processing of the at least one reverse direction processing information.

[0095] The apparatus caused to transmit the at least one response processing information may be caused to transmit the at least one response processing information within at least one of: a real-time transport protocol payload; a real-time transport protocol header extension; and a real-time transport control protocol payload.

[0096] The apparatus caused to obtain the at least one reverse direction processing information may be caused to obtain at least one associated identifier header configured to identify the at least one reverse direction processing information associated with the at least one immersive audio payload.

[0097] The reverse direction processing information may comprise at least one of: an action request; a request to change an IVAS input format; a request to change a coded format; a request for an orientation of the apparatus; a request for configuration information relating to a microphone array constellation of the apparatus; a request to switch a bitrate from the apparatus to the further apparatus; a request for binauralization data that can be used to revert the binauralization at the further apparatus; information relating to the further apparatus, for assisting the apparatus in encoding the at least one immersive audio coded bitstream; information relating to the apparatus orientation and / or position; information relating to a class of the reverse direction processing information; information relating to a head rotation of a listener of the at least one immersive audio coded bitstream; information relating to a position of a listener of the at least one immersive audio coded bitstream; information relating to a loudspeaker constellation associated with the further apparatus; information relating to a change in an output audio format; information relating to a combined orientation applied at the further apparatus; and information related to reverberation parameters applied at the further apparatus.

[0098] The reverse direction processing information may further comprise at least one of: a time value indicating a time of the reverse direction processing information, to cause the apparatus to process the reverse direction processing information based on the time value; an urgency value, to cause the apparatus to process the reverse direction processing information relative to a further reverse direction processing information based on the urgency value; a priority value, to cause the apparatus to process the reverse direction processing information relative to a further reverse direction processing information based on the priority value. The at least one reverse direction processing information may further comprise at least one usage value, the usage value caused to identify whether the information expects or does not expect a response.

[0099] The reverse direction processing information may comprise at least one of: information identifying an orientation of the apparatus or a capturing apparatus; information identifying a microphone array constellation configuration of the apparatus or a capturing apparatus; information identifying an orientation applied to the at least one immersive audio coded bitstream; information identifying a subsequent changes input format for the at least one immersive audio coded bitstream; information identifying a subsequent coded format change for the at least one immersive audio coded bitstream; information identifying a specified number of frames or number of seconds until a change in an input format for the at least one immersive audio coded bitstream; information identifying binauralization data that can be used to revert the binauralization at the apparatus; and information about an acoustic environment.

[0100] The apparatus may be further caused to negotiate with a further apparatus support for the reverse direction processing information.

[0101] The apparatus caused to negotiate with the further apparatus may be caused to negotiate employing a session description file.

[0102] According to a seventh aspect there is provided an apparatus for controlling an immersive audio transmission comprising at least partially a bidirectional signalling, the apparatus comprising: obtaining circuitry configured to obtain at least one immersive audio payload comprising at least one immersive audio coded bitstream; obtaining circuitry configured to obtain at least one reverse direction processing information, the at least one reverse direction processing information for providing at least in part the bidirectional signalling; and packetizing circuitry configured to packetize the at least one reverse direction processing information in a transport protocol payload.

[0103] According to an eighth aspect there is provided an apparatus for controlling bidirectional audio transmission comprising at least partially a bidirectional signalling, the apparatus comprising: transmitting circuitry configured to: transmit at least one immersive audio payload comprising at least one immersive audio coded bitstream; obtaining circuitry configured to obtain packetized at least one reverse direction processing information, the packetized at least one reverse direction processing information for providing at least in part the bidirectional signalling; and processing circuitry configured to process the packetized at least one reverse direction processing information.

[0104] According to a ninth aspect there is provided a computer program comprising instructions [or a computer readable medium comprising program instructions] for causing an apparatus for controlling an immersive audio transmission comprising at least partially a bidirectional signalling to perform at least the following: obtain at least one immersive audio payload comprising at least one immersive audio coded bitstream; obtain at least one reverse direction processing information, the at least one reverse direction processing information for providing at least in part the bidirectional signalling; and packetize the at least one reverse direction processing information in a transport protocol payload.

[0105] According to a tenth aspect there is provided a computer program comprising instructions [or a computer readable medium comprising program instructions] for causing an apparatus for controlling bidirectional audio transmission comprising at least partially a bidirectional signalling to perform at least the following: transmit at least one immersive audio payload comprising at least one immersive audio coded bitstream; obtain packetized at least one reverse direction processing information, the packetized at least one reverse direction processing information for providing at least in part the bidirectional signalling; and process the packetized at least one reverse direction processing information.

[0106] According to an eleventh aspect there is provided a non-transitory computer readable medium comprising program instructions for causing an apparatus for controlling an immersive audio transmission comprising at least partially a bidirectional signalling to perform at least the following: obtain at least one immersive audio payload comprising at least one immersive audio coded bitstream; obtain at least one reverse direction processing information, the at least one reverse direction processing information for providing at least in part the bidirectional signalling; and packetize the at least one reverse direction processing information in a transport protocol payload.

[0107] According to a twelfth aspect there is provided a non-transitory computer readable medium comprising program instructions for causing an apparatus for controlling bidirectional audio transmission comprising at least partially a bidirectional signalling to perform at least the following: transmit at least one immersive audio payload comprising at least one immersive audio coded bitstream; obtain packetized at least one reverse direction processing information, the packetized at least one reverse direction processing information for providing at least in part the bidirectional signalling; and process the packetized at least one reverse direction processing information.

[0108] According to a thirteenth aspect there is provided an apparatus for controlling an immersive audio transmission comprising at least partially a bidirectional signalling, the apparatus comprising: means for obtaining at least one immersive audio payload comprising at least one immersive audio coded bitstream; means for obtaining at least one reverse direction processing information, the at least one reverse direction processing information for providing at least in part the bidirectional signalling; and means for packetizing the at least one reverse direction processing information in a transport protocol payload

[0109] According to a fourteenth aspect there is provided an apparatus for controlling bidirectional audio transmission comprising at least partially a bidirectional signalling, the apparatus comprising: means for transmitting at least one immersive audio payload comprising at least one immersive audio coded bitstream; means for obtaining packetized at least one reverse direction processing information, the packetized at least one reverse direction processing information for providing at least in part the bidirectional signalling; and means for processing the packetized at least one reverse direction processing information.

[0110] According to a fifteenth aspect there is provided a computer readable medium comprising program instructions for causing an apparatus for controlling an immersive audio transmission comprising at least partially a bidirectional signalling to perform at least the following: obtain at least one immersive audio payload comprising at least one immersive audio coded bitstream; obtain at least one reverse direction processing information, the at least one reverse direction processing information for providing at least in part the bidirectional signalling; and packetize the at least one reverse direction processing information in a transport protocol payload. According to a sixteenth aspect there is provided a computer readable medium comprising program instructions for causing an apparatus for controlling bidirectional audio transmission comprising at least partially a bidirectional signalling to perform at least the following: transmit at least one immersive audio payload comprising at least one immersive audio coded bitstream; obtain packetized at least one reverse direction processing information, the packetized at least one reverse direction processing information for providing at least in part the bidirectional signalling; and process the packetized at least one reverse direction processing information.

[0111] An apparatus comprising means for performing the actions of the method as described above.

[0112] An apparatus configured to perform the actions of the method as described above.

[0113] A computer program comprising program instructions for causing a computer to perform the method as described above.

[0114] A computer program product stored on a medium may cause an apparatus to perform the method as described herein.

[0115] An electronic device may comprise apparatus as described herein.

[0116] A chipset may comprise apparatus as described herein.

[0117] Embodiments of the present application aim to address problems associated with the state of the art.

[0118] Summary of the Figures

[0119] For a better understanding of the present application, reference will now be made by way of example to the accompanying drawings in which:

[0120] Figure 1 shows schematically example server and peer-to-peer teleconferencing systems within which embodiments may be implemented;

[0121] Figure 2 shows schematically an example packet format with Enhanced Voice Services (EVS) Header-Full payload structure;

[0122] Figure 3 shows schematically an example Table of Content (ToC) byte structure for a Header-Full IVAS frame according to some embodiments;

[0123] Figure 4 shows schematically an example processing information (PI) frame in an IVAS RTP payload according to some embodiments; Figure 5 shows schematically an example CMR frame in a IVAS RTP payload according to some embodiments;

[0124] Figure 6 shows a schematic view of an example exchange of different PI frames and IVAS frames between two UEs in an IVAS session;

[0125] Figure 7 shows schematically an example structure for an input format change PI frame;

[0126] Figure 8 shows schematically an example structure for an output format change PI frame;

[0127] Figure 9 shows schematically a flow diagram session negotiation offer and answer operations according to some embodiments;

[0128] Figure 10 shows schematically a flow diagram session for packetization and de-packetization of bi-directional PI frames according to some embodiments; and

[0129] Figure 11 shows an example device suitable for implementing the apparatus shown.

[0130] Embodiments of the Application

[0131] The following describes in further detail suitable apparatus and possible mechanisms for the provision of efficient IVAS audio.

[0132] An example system within which embodiments may be implemented is shown in Figure 1 .

[0133] Figure 1 , for example, shows an example teleconferencing system within which some embodiments can be implemented. In this example there is shown two sites or rooms, Room A 100 and Room B 102. Room A 100 comprises a ‘talker’ or user, Talker 1 103. Room B 102 comprises one ‘talker’ or user, Talker RX 141.

[0134] In the following example within room A is a suitable teleconference apparatus (or more generally telecommunications apparatus 110) configured to spatially capture and encode the audio environment and furthermore is configured to render a spatial audio signal to the room. The apparatus can in some embodiments be implemented by a user equipment (UE) operating within a cellular communications system or accessing any suitable access network. Within each of the other rooms may be a suitable teleconference apparatus (or more generally telecommunications apparatus such as apparatus 120 within room B) configured to render a spatial audio signal to the room and furthermore is configured to capture and encode at least a mono audio and optionally configured to spatially capture and encode the audio environment.

[0135] In the following examples each room is provided with the means to spatially capture, encode spatial audio signals, receive spatial audio signals and render these to a suitable listener. It would be understood that there may be other embodiments where the system comprises some apparatus configured to only capture and encode audio signals (in other words the apparatus is a ‘transmit’ only apparatus), and other apparatus configured to only receive and render audio signals (in other words the apparatus is a ‘receive’ only apparatus). In such embodiments the system within which embodiments may be implemented may comprise apparatus with varying abilities to capture / render audio signals.

[0136] The teleconference apparatus (for each site or room) 110, 120 can be configured to call into a teleconference controlled by and implemented over a server 111.

[0137] In some embodiments the communications or teleconferencing system comprises a (peer-to-peer) communications system (rather than the server based system shown in Figure 1 ) within which some embodiments can be implemented. Thus, for example, two or more UEs can be configured to interact directly with each other (for example to implement an immersive audio phone call between users). In such a scenario one of the UEs can be configured to deliver spatial ambience as one stream and employ a close-up microphone (for example a Lavalier microphone) to capture the speech as an audio object or audio source. The sender UE can be configured to encode the spatial ambience audio signals in a MASA format stream and the close-up microphone audio signal as an object format stream. The two audio streams can then be delivered as separated IVAS streams. The sender UE, in addition, can be configured to encode processing information during the encoding to deliver the PI frames together with the IVAS frames to the receiver UE.

[0138] The teleconference apparatus can be configured to spatially capture and encode the audio environment and furthermore can be configured to render a spatial audio signal to the room. In this example only the communications or signalling path from the Room A 100 to the Room B 102 is shown for simplicity but a duplex or multipoint communication system comprising multiple signalling paths can be implemented using the methods as described herein without significant inventive input.

[0139] The teleconference apparatus (for each site or room) 110, 120 is further configured to communicate with each other to implement a teleconference function.

[0140] As shown in Figure 1 , the apparatus 110, 120 and server 111 can comprise suitable encoder and decoder functionality. For example, the apparatus 110 is shown comprising an (IVAS) encoder 101 , the server 111 is shown comprising a (IVAS) decoder and encoder 121 and the apparatus 120 is shown comprising an (IVAS) decoder 131. In such a manner an object 120 (the audio signals representing the user or talker 1 111 ) can be encoded by the encoder 101 which generates a bitstream 106 to be passed to a server 111. The server 111 can then decode, (optionally then mix with other objects and otherwise process the audio signals) and encode then to generate the bitstream 108 to be passed to the apparatus 120. The apparatus 120 can then decode the audio signals and present them to the user or talker ‘Talker RX’ 141 .

[0141] Although this example shows a teleconference application the encoder / decoder functionality can be applied to the streaming of any suitable media.

[0142] The IVAS decoder / renderer for each of the teleconference apparatus 102 can be furthermore configured to handle multiple input streams that may each originate from a different encoder.

[0143] As discussed previously RTP is intended for an end-to-end, real-time transfer of streaming media and provides facilities for jitter compensation and detection of packet loss and out-of-order delivery. RTP is furthermore designed to carry a multitude of multimedia formats, which permit the transport of new formats without revising the RTP standard. To this end, the information required by a specific application of the protocol is not included in the generic RTP header. For a class of applications (e.g., audio, video), an RTP profile may be defined. For a media format (e.g., a specific video coding format), an associated RTP payload format may be defined. Every instantiation of RTP in a particular application may therefore require a profile and payload format specifications. The profile is configured to define the codec used to encode the payload data and the mapping to payload format codes in the protocol field Payload Type (PT) of the RTP header.

[0144] For example, the RTP profile for audio and video conferences with minimal control is defined in RFC 3551 . The profile defines a set of static payload type assignments, and a dynamic mechanism for mapping between a payload format, and a PT value using Session Description Protocol (SDP). The latter mechanism is used for newer video codec such as RTP payload format for H.264 Video defined in RFC 6184 or RTP Payload Format for High Efficiency Video Coding (HEVC) defined in RFC 7798.

[0145] An RTP session can be established for each multimedia stream. Audio and video streams may be implemented which use separate RTP sessions, enabling a receiver to selectively receive components of a particular stream. The RTP specification can furthermore be configured to recommend port numbers for RTP, and furthermore to recommend the use of the next odd port number for the associated RTCP session. A single port can be used for RTP and RTCP in applications that multiplex the protocols.

[0146] Each RTP stream can comprise RTP packets, and the RTP packet in turn can comprise a RTP header and payload pair.

[0147] Enhanced Voice Services (EVS) is a mono voice codec standardized in 3GPP and described in the TS 26.445 specification document. The codec can have two operating modes: EVS Primary and EVS AMR-WB IO (Adaptive Multi Rate Wideband Inter-Operable).

[0148] The IVAS codec can be considered to be an extension to the EVS codec and as such the IVAS and EVS codecs can have some similarities in terms of design and implementation. The RTP payload structure, while not having been specified for the IVAS codec, is envisioned to have similarities to the RTP payload structure in EVS.

[0149] The RTP payload format of EVS is described in 3GPP TS 26.445 Annex A. In EVS, the RTP payload format is divided into two different embodiments: a Compact format and a Header-Full format. In the EVS Compact payload format, a RTP packet includes only a single EVS speech frame for EVS Primary mode. For EVS AMR-WB IO mode, the compact RTP packet also includes a 3-bit Codec Mode Request (CMR) field in front of the speech frame. In the EVS Compact format, the different modes and bitrates for the speech frames are identified by the size of the RTP payload. For example, an RTP packet of size 328 bits is assigned for EVS Primary mode with 16.4 kbps bitrate, as is shown in Table A.1 in TS 26.445 Annex A.

[0150] In EVS Header-Full format, the RTP payload consists of the speech frame(s) accompanied by an optional CMR byte and Table of Content (ToC) bytes. The CMR byte is used to request a change in bitrate or coding mode that the receiver wants to receive. The request is sent as a CMR byte as part of the Header-Full EVS packet. In EVS AMR-WB IO Compact format, the CMR functionality is also present as a 3-bit signaling at the beginning of the packet.

[0151] The ToC bytes in EVS Header-Full format describe the mode and bitrate for the accompanied EVS coded speech frames. RTP payload structures for EVS Header-Full format are furthermore shown in Figure 2.

[0152] Figure 2 shows, for example, a payload structure of Header-Full format with ToC single frame 201 , a payload structure of Header-Full format with ToC multiple frames 203, a payload structure of Header-Full format with CMR and ToC single frame 205 and a payload structure of Header-Full format with CMR and ToC multiple frames 207.

[0153] Each of the payload structures comprises ToC / CMR bytes where for each of these bytes there is a first bit 211 , 221 , 231 at the beginning of the ToC / CMR bytes which can be used to differentiate the bytes between ToC (where the bit value is 0) and CMR (where the bit value is 1 ).

[0154] In the payload structure 201 where there is a single speech frame in the payload, the speech data 215 is preceded by a single ToC byte 213 (with optional zero padding 217 at the end of the payload).

[0155] In examples where the single speech frame payload structure can have an optional CMR byte, such as shown in payload structure 205. The payload structure 205 differs from the payload structure 201 in that there is a CMR byte 232 (with associated indicator bit 231 value of 1 ) positioned before the ToC byte 213.

[0156] In some example structures there can be two speech frames in the payload structure 203, 207. In this example an additional ToC byte 223 (with associated indicator bit 221 value of 0) and the ToC byte 213 is positioned at the beginning of the payload followed by the speech data frames 215 and 225 respectively to the order of the ToC bytes.

[0157] Furthermore the multiple speech frame payload structure can have an optional CMR byte, such as shown in payload structure 207. The payload structure 207 differs from the payload structure 203 in that there is a CMR byte 232 (with associated indicator bit 231 value of 1 ) positioned before the ToC bytes 213, 223.

[0158] The IVAS codec supports stereo and several spatial / immersive formats at wide range of bitrates. Particularly at the lowest IVAS bitrates (as low as 13.2 kbps) the encoding quality of the audio signals is reduced due to insufficient bitrate. There is thus generally a very limited capability to provide audio processing, rendering related, or even non-audio related, auxiliary information within the codec payload.

[0159] During an IVAS session setup, the UEs may be arranged to agree on an immersive format which is suitable for their usage. For example, the immersive format may be agreed for headphone or multichannel loudspeaker setup. This can happen for a user wearing headphones or a user consuming IVAS audio via multichannel speaker configurations in their car. However, when the same user removes their headphones or steps out of their car and switches to consuming audio via the mono loudspeaker on his mobile device, the high bitrate and high complexity immersive audio being delivered is not required for a credible mono loudspeaker playback.

[0160] A quick switch of input format (for example from binaural / multispeaker high quality to economical mono consumption mode) or indication of the consumption mode is currently not supported for IVAS based conversational audio sessions. There is a need to enable signaling of output type information or input format change requests via a suitable low latency path in order to provide quick responses in consumption modes.

[0161] As presented in GB2313472.9, PI (processing information) frames can be used in an IVAS session to transmit (non-audio) related information between session parties. The PI frames enable maintaining appropriate temporal alignment between the audio data and the associated information. However, PI frames operate in a unidirectional manner. In other words data is transmitted from the sender to the receiver. In an IVAS session, there are no defined mechanisms for bi-directional communication between the session parties within the payload. There is furthermore support for rate adaptation control (e.g., CMR in EVS codec RTP payload), however this support for rate adaptation control is limited to modifying the codec mode.

[0162] Currently PI frames support only Header-Full IVAS format. The following embodiments describe support for PI frames in IVAS Compact format. Additionally PI frames currently require content headers (ToC) for each PI frame, and the embodiments described herein show compound headers which can identify multiple PI frames.

[0163] In other words, the following examples describe apparatus and methods configured to enable bidirectional signaling between the transmitter entity (e.g., sender UE) and the receiver entity (e.g., receiver UE) in order to enable bitstream adaptation capability during the immersive conversational audio session.

[0164] The concept relates to apparatus and methods for signalling output format indication and format change request in-band by leveraging the bidirectional audio transmission streams in an immersive conversational audio session. This can in some embodiments be achieved by adding format change and output format selection indications as real time transport protocol payload headers. In some embodiments, this information can also be signalled as RTCP feedback.

[0165] These embodiments therefore relate to apparatus and methods for transporting processing information in an immersive conversational audio session where there is provided a method for bi-directional communication between the session parties to enable a support for transmitting feedback and request processing information data from a decoding entity to an encoding entity and a support for transmitting response processing information data from an encoding entity to a decoding entity.

[0166] Furthermore, in some embodiments the headers for the processing information frames can be combined into a single compound header to signal the presence of multiple processing information frames in an immersive conversational audio payload with a single header.

[0167] Additionally in some embodiments, there is provided support for the processing information frames with header-less immersive conversational audio frames. The support is achieved by transmitting the processing information frames either as separate packets (e.g., RTP packets) from the audio data (e.g., IVAS frames) and ensuring that the sizes of the processing information payloads do not collide with the sizes of the protected header-less immersive conversational audio payloads. In case of delivering processing information as RTP header extension avoids the risk of collision with protected sizes of the compact format.

[0168] The bi-directional processing information communication can in some embodiments be achieved as follows:

[0169] Decoder B (receiver UE) to Encoder A (sender UE):

[0170] Form a processing information request or feedback frame and an associated identifier header in an immersive voice codec (IVAS) session;

[0171] Attach the identifier header to the processing information frame;

[0172] Packetize or include the processing information (header and frame) in an IVAS (RTP) payload and transmit the payload to the sender UE. Another possibility in some embodiments is to provide processing information frame as RTP header extension data or via RTCP (real-time transport control protocol).

[0173] Encoder A (sender UE):

[0174] (The encoder with respect to Encoder A handling of feedback from receiver UE to sender UE)

[0175] Receive an immersive voice codec (IVAS) payload;

[0176] Read the identifier header(s) and identify the processing information frames from the payload;

[0177] Detect a feedback frame via indication in the processing information frame;

[0178] For a feedback frame:

[0179] Read the feedback frame;

[0180] Take the feedback data or information into account in the encoding or capturing process or in any other suitable action.

[0181] (Transmission from sender UE to receiver UE)

[0182] Detect a request frame via indication in the processing information frame;

[0183] For a request frame:

[0184] Read the request frame; Act on the request by either:

[0185] Create a response processing information frame with the requested data and transmit the data to the receiver B;

[0186] OR change the encoding or capturing processing or take any other suitable action based on the request

[0187] In some embodiments, the processing information can be signaled as part of the RTP header extension.

[0188] In some embodiments, the session negotiation between the sender and the receiver UE for immersive conversational audio session can be negotiated via SIP / SDP. The SIP / SDP message further can comprise one or more of the following signaling capability:

[0189] Bitstream status or property indication (from sender UE to receiver UE);

[0190] Format change request or feedback (from receiver UE to sender UE);

[0191] Bitrate level indication for rate adaptation feedback;

[0192] Output format indication (from receiver to sender UE);

[0193] Capture device orientation; and

[0194] Request Response.

[0195] As shown in Figure 2, the Header-Full IVAS frame comprises one or several Table-of-Content (ToC) indicators 213, 223 and their associated IVAS speech / audio frames 215, 225. With respect to Figure 3 is shown an example Table-of-Content (ToC) byte structure 301 for an IVAS frame.

[0196] The (F) bit 303 indicates whether more frames follow this entry. The F (1 bit) 303: If set to 1 , the bit indicates that the corresponding frame is followed by another speech or PI frame in this payload, implying that another ToC byte follows this entry. If set to 0, the bit indicates that this frame is the last frame in this payload and no further header entry follows this entry.

[0197] The Bitrate Level Indication (BLI) 307 describes the content of the associated frame, e.g., the bitrate of an IVAS frame. The Supplemental Signaling Bits (SSB) 305 can be used to describe various things, for example IVAS input format specific signaling. The BLI (4-5 bits) 307 can, for example, be configured to indicate the bitrate or other frame content indication for the frame. From the content indication, the receiver can determine the size of the received frame either directly from the bitrate or from pre-determined frame sizes, e.g., for the SPEECH_LOST and NO_DATA frames. The BLI (or in some implementations the Frame Type or FT) bits can indicate, for example, the bitrate of an IVAS speech frame, a SPEECH_LOST, NO_DATA or comfort noise (SID) frames.

[0198] An example for BLI bit values (for 5 bits) are presented in the following table. At this point of the codec development, it is not decided if the aforementioned three types (SPEECH_LOST, NO_DATA, SID) are supported in the final codec. The SPEECH_LOST, NO_DATA and SID frames are part of the EVS codec and at it is likely that at least the SID and NO_DATA frame types are being incorporated into the IVAS specification. It is understood that the final IVAS codec specification, however, might have different frame types present than presented here. In some embodiments, 4 bits are reserved for the BLI part. The frame type bit values and content indications when using 4 bits are presented below. In these embodiments the BLI part identifies the bitrate where the BLI bits are bit values other than a defined value (for example 1111 ) and can be used to identify other aspects, for example SPEECH_LOST, NO_DATA and SID frames with the combination of the BLI indicator OTHER (the defined value) and the extra bits.

[0199] As shown above there are 15 available bit allocations for the BLI bits reserved for future use (bit allocations 10001 - 11111 ), when 5 bits are reserved for the BLI indicator. In some embodiments when 4 bits are reserved for the BLI indicator, there are 5 available bit allocations for future use, when BLI value 1111 is used. BLI value 1110 provides an additional 8 bit allocations for future use when combined with the extra bits. These available bit allocations can be used in the future to indicate frame contents that will be defined, such as PI frames described hereafter.

[0200] These bit allocation values are examples only and it would be appreciated that in some embodiments the bit allocation values can be otherwise configured.

[0201] The number of bits in the SSB 305 can vary between 2-3 depending on how many bits are reserved for the BLI bits 307 in the IVAS ToC byte 301 . The SSB (2- 3 bits) 305 can be bits reserved for future use. If 4 bits are reserved for the BLI indicator, the extra 3 bits can be partly used to identify other frame content than bitrates / frame-size (e.g., SPEECH_LOST, NO_DATA, SID) as demonstrated in further detail in GB2313472.9, where the BLI bits are referred to as FT bits (frame type index) and SSB as extra bits.

[0202] In order to enable rendering or consumption of the IVAS audio bitstream data, additional (non-audio) data can be added to streamed IVAS RTP packets. Specifically, there can be inserted data that requires maintaining a sufficient or exact alignment with the IVAS audio bitstream data. The alignment can be considered, for example, relative to an IVAS audio frame. For example, any external orientations from the sender UE side could be included in these (audio) processing information (PI) frames.

[0203] A RTP payload with respect to these embodiments can comprise one or more of the following: immersive audio coded bitstream; the immersive audio coded bitstream payload header. The immersive audio coded bitstream payload header can be a ToC header byte, which can comprise supplemental signaling bits or frame type indication, bitrate level indication bits, etc; and processing information frame.

[0204] Packetization furthermore with respect to these embodiments can be a process of including the immersive audio coded bitstream header, and at least one of the immersive audio coded bitstream or processing information frame as RTP payload in order to deliver the immersive audio coded bitstream over real-time transport protocol. In some embodiments, the packetization can also be performed for inclusion and carriage of control information as payload of the real-time transport control protocol.

[0205] An example PI frame structure is shown with respect to Figure 4.

[0206] The example PI frame structure 401 comprises a “format” field 403. The format field 403 is configured to describe the type of the PI data, for example, orientation data.

[0207] The example PI frame structure 401 can further comprise a “usage” field 405. The usage field 405 is configured to describe how the PI data should be used. For example, the usage field 405 is configured to define whether the data should be applied to the next IVAS frame, to all frames in the RTP packet or if the data is more general information to be sent to the receiver. In other words the usage field can describe that the PI data should not be taken account in the rendering, i.e. , that the data is more general information sent to the receiver.

[0208] Additionally the example PI frame structure 401 can further comprise a “validity” field 407. The validity field 407 in some embodiments describes how long the PI data is valid at the receiver end. For example, the field could describe that for X amount of processing frames the receiver should apply the PI data described in the frame, and if no new PI data is received within X frames, stop applying the data. The validity could also be indicated in other time formats than number of processing frames, e.g., in milliseconds. Furthermore, some audio frame values may be “hold” whereas other values may be “instantaneous” only. This enables the renderer to perform rendering accordingly.

[0209] Furthermore the example PI frame structure 401 can comprise an optional “size” field 409. The size field 409 in some embodiments is configured to define the size of the PI data 411. In some embodiments to minimize the number of additional bits introduced by the PI frames, the maximum size of the PI frames can be restricted. For example, the frames can be restricted to have a maximum size of K bits that is common for all IVAS bitrates. Alternatively, the maximum size of the PI frames could be linked to the used bitrate of the IVAS speech frames, e.g., by having the maximum number of bits for PI frames be some percentage share of the bits used for the speech frames. In addition, the use of PI frames could be restricted to be used only in the highest IVAS bitrates to reduce the risk of adding too much load to the sent packets. In some embodiments the “usage” and “validity” fields could be also combined into a single field, for example, to a “scope” field. The “scope” field would indicate both how the data should be applied and how long the data is valid.

[0210] Additionally the PI frame structure comprises the PI data 411 .

[0211] CMR (Codec Mode Request) can furthermore be signaled through an IVAS ToC byte, as described in GB2314333.2. Another approach for supporting CMR in an IVAS session is to indicate the change request through a PI frame. An example CMR PI frame structure is shown in Figure 5.

[0212] The CMR PI frame example comprises a CMR type 503, a “usage” field 405, a “validity” field 407 and “size” field 409 as described above.

[0213] Additionally, the CMR PI frame 501 indicates the stream which is affected by the request (via “Requested_bitrate” field 511 ). The “Requested_bitrate” 511 field indicates the bitrate that is requested. In some embodiments, the “Requested_bitrate” field 511 can also be used to request mode change, for example, from EVS Primary to EVS AMR-WB IO, as is possible with the CMR byte in the EVS specification. The usage of CMR PI frame type can in some embodiments be used as a replacement to the ToC CMR approach. The “usage” field for a CMR PI frame could be set as REQUEST, which is explained in more detail in the examples below.

[0214] As presented in further detail in GB2313472.9, the PI (processing information) frames can be employed in an IVAS session to transmit (non-audio) related information between the session parties. The information can, for example, relate to the capturing device orientation and labelling the data. The PI frames enable maintaining appropriate temporal alignment between the audio data and the associated information. The transmission described in the examples presented in GB2313472.9 is unidirectional, i.e., from the sender to the receiver. However, as described herein in further detail the PI frames could also be utilized to transmit bidirectional information. That is, the transmitted data from the sender to the receiver can also have an effect on the other transmit direction.

[0215] For example Figure 6 shows an example scenario according to some embodiments where two UEs 110, 120 are communicating through an IVAS session. These examples focus on aspects where the encoder / sender UE 110 or UE A is sending IVAS frames 601 to the decoder / receiver UE 120 or UE B. The PI forward frames 603 can be interpreted as processing information data indication or indicators from the sender UE 110 to the receiver UE 120. The type of data within the PI forward frames 603 can in some embodiments relate to the capturing device orientation and labelling the data, for example, as mentioned above and in further detail as described in GB2313472.9.

[0216] Additionally as shown in Figure 6, PI request frames can be action requests transmitted from the receiver UE 120 to the sender UE 110. The action requests can be, for example, a request to change the IVAS input format from the sender A to the receiver B. The PI request frames 605 can comprise indicators or other signaling that the sender UE 110 is caused to take some action in response to the received PI request frame, for example, change the used input format.

[0217] In some embodiments the request can be a request to change a coded format. For example the input format could be an input format OMASA which is a combination of Objects + MASA formats whereas the coded format of the IVAS frame is OMASA.

[0218] Furthermore, there can comprise PI feedback frames 607, as shown in Figure 6 transmitted from the receiver UE 120 to the sender UE 110. The feedback frames can comprise information from the receiver UE 120 to the sender UE 110. The PI feedback frames 607 can differ from the PI request frames 605 in that the PI feedback frames 607 do not cause the sender UE 110 to switch format but can for example comprise information relating to the receiver UE which the sender UE can choose whether or not to process. For example the feedback information can be the playback device orientation or the head rotation of the listener at the receiver UE which the sender UE 110 can, in some circumstances use to improve the encoding or processing of the capture audio signals. In some embodiments, as shown in Figure 6, there can also be PI response frames 609. The PI response frames 609 can be PI frames in response to the PI request frames 605. For example, if the receiver UE 120 is requesting the capture device orientation of the sender A, the sender UE 110 can send the orientation to the receiver UE 120 as a PI response frame 609.

[0219] In some embodiments each request can be identified with an ID and the same ID can also be present in the response. Additionally in some embodiments the request is identified by a unique ID which is echoed by the same ID in the PI response frame 609.

[0220] Additionally the receiver UE 120 can be configured to insert other information into the PI request frame 605 (or PI feedback frame 607) information which enables the sender UE 110 to identify which request the sender UE 110 is responding to. This other information can for example comprise time information, priority information or other suitable information such as urgency information identifying whether the request is an urgent or real-time request.

[0221] For example, the receiver UE 120 can send multiple requests with similar type to the sender UE 110. The sender UE 110 might receive multiple of these requests at the same time, for example, due to network congestion or speed. The sender UE 110 could then in some scenarios respond only to the latest request or the request with a highest priority or urgency.

[0222] In another scenario, the PI response frames 609 might arrive at the receiver UE 120 at the same time, in which case the receiver UE 120 can be configured to choose to process the latest response and ignore the other responses of same type or to process the process with the highest priority or urgency.

[0223] In some example, the sender UE 110 is configured to reject a PI request frame 605 sent from the receiver UE 120. For example, if the sender UE 110 cannot provide the requested input format, then the sender UE 110 can be configured to transmit a PI response frame 609 comprising a rejection response to the receiver UE 120.

[0224] The request PI frames 605 and feedback PI frames 607 can, in some embodiments, be identified with a “usage” field of the PI frame. For example, new “usage” field types of REQUEST and FEEDBACK could be allocated to indicate these types of PI frames, respectively. The response PI frames could similarly be identified with a RESPONSE “usage” field type.

[0225] In some embodiments, some of the response PI frames 609 could be transmitted as forward / indication PI frames 603. In these examples a received PI frame at the receiver UE 120 can comprise data or information which would indicate an ID for the frame. This ID can be used by the receiver UE 120 to link the received PI frame to a previously transmitted PI request frame 605. In such embodiments if there is no PI request frame 605 with the same ID, the received PI frame can be identified as a conventional forward / indication PI frame 603 (rather than a response PI frame 609).

[0226] In some embodiments, the reason for a PI frame request or response is also signaled. For example, the receiver UE 120 is configured to want to increase the overall quality of the playback and therefore request a higher bitrate from the sender UE 110. In another example, the sender UE 110 could add a reason to a rejected request response to inform the receiver UE 120 why the request was rejected. The reason could be signaled, for example, as an additional “reason” field in the PI frame, similar to the “usage” and “validity” fields identified in the example structure shown in Figure 4. In some embodiments the PI frame data comprises the reason information.

[0227] In some embodiments, a class for a PI frame is also signaled. The class could, for example, signal if the PI frame is related directly to, e.g., IVAS decoding or rendering or whether the PI frame is related to something else (e.g., application processing). For example, an orientation PI frame could be linked directly to IVAS rendering, in which case the PI frame data (i.e., orientation) would be directly forwarded to an IVAS Tenderer. For a label PI frame, for example, the labelling data is linked to application processing and could be forwarded to the application in the receiver instead of to the IVAS Tenderer. The class could be signaled, for example, as an additional “class” field in the PI frame, similar to the “usage” and “validity” fields identified in the example structure shown in Figure 4. In some embodiments, the PI frame may have an implicit validity based on the semantics specified regarding the validity. For example, this can be a validity until a new PI frame is received conveying the same signaling information. In embodiments, the feedback, request, response and indication PI frames can be transmitted by packetizing or including the processing information frames in an IVAS RTP payload. In some embodiments, the processing information frames can be packetized or included in an IVAS RTP header extension. In some further embodiments, the processing information frames can be packetized or included in RTCP packets or payloads.

[0228] Tables 1 , 2, and 3 show some example frame types for request, feedback and forward / indication PI frames, respectively. The names for the PI frame types are examples and may be something different in any further appliances.

[0229] In some embodiments, the different orientation information’s can be also informed as rotations. The sender UE 110 and receiver UE 120 notations follow the scenario presented in Figure 6.

[0230] In some embodiments, some of the feedback typed PI frames could also be requested from the receiver UE 120 by the sender UE 110.

[0231] Table 1 - Example request PI frame types.

[0232] The CAPTURE_DEVICE_ORIENTATION type can be used to request the orientation of the capturing device that the sender UE 110 is using. As a forward / indication or response frame from the sender UE 110 to the receiver UE 120, the PI frame data consists of the capture device orientation. The orientation can be applied to the spatial audio stream already at the sender UE side. In other words, the movement or orientation changes of the capturing device that may be undesired rotations for the captured audio scene are compensated for as part of the capture processing. In this case, the transmitted orientation indicates information about which orientation is applied to the audio stream. The receiver UE 120 can then in some cases undo the orientation compensation. In some embodiments, the orientation is not applied to the spatial audio stream at the sender UE 110, in which case the orientation can be applied at the receiver UE 120. In both cases, the information whether the orientation is applied to the audio stream or not should be transmitted to the receiver UE 120 somehow. The information can be part of the PI frame, for example, as an indication flag. In some embodiments the indication is signaled in the SSB part of the ToC byte associated with the IVAS frame that is linked to the orientation PI frame. One bit from the SSB bits could be used to indicate that orientation has been applied to the frame already at the sender UE 110 side.

[0233] The CAPTURE_DEVICE_MIC_CONSTELLATION type can be used to request the microphone array constellation of the capturing device that the sender 110 is using. As a forward / indication or response frame from the sender UE 110 to the receiver UE 120, the PI frame data consists of the microphone array constellation. The constellation could include, for example, the positions and orientations of the individual microphones in the recording array / phone.

[0234] The INPUT_FORMAT_CHANGE type can be used to request the sender UE 110 to switch the IVAS input format of the transmitted bitstream. The request does not require a response to the receiver UE 120 as the receiver UE 120 can observe the change of the input format from the received IVAS frames. Input format switching is covered in more detail later.

[0235] The CMR (CODEC_MODE_REQUEST) type can be used to request the sender UE 110 to change the bitrate of the transmitted bitstream. In case the bitstream contains EVS data, the CMR can also indicate codec mode change request (change between EVS Primary and EVS AMR-WB IO modes). CMR PI frames are explained in further detail later on. In some embodiments, the CMR and INPUT_FORMAT_CHANGE PI frames could be combined into a single request PI frame type.

[0236] The BINAURALIZATION_DATA type can be used to request the sender UE 110 to provide binauralization data for the receiver UE 120 that allows the receiver UE 120 to revert the binauralization of the transmitted binaural IVAS bitstream. The reversion process allows the receiver UE 120 to apply a new binauralization to the received signal, for example, with head orientation applied.

[0237] Table 2 - Example feedback PI frame types.

[0238] The PLAYBACK_DEVICE_ORIENTATION type can be used to inform the sender UE 110 which orientation the playback device of the receiver UE 120 has. The playback device could be, for example, a phone or a head-mounted display such as virtual reality goggles. The PLAYBACK_DEVICE_SPEAKER_CONSTELLATION type can be used to inform the sender UE 110 the loudspeaker constellation of the playback setup or device the receiver UE 120 is using. The playback setup / device could be, for example, a multi-loudspeaker setup or a phone. The constellation could include, for example, the positions and orientations of the loudspeakers in the playback setup / device.

[0239] OUTPUT_FORMAT_CHANGE type can be used to inform the sender UE 110 that the output format for the playback has been changed at the receiver UE 120. For example, if the receiver UE 120 is switching from using headphones to using a mono loudspeaker, the change can be informed with this PI frame type. Output format switching is covered in more detail later.

[0240] OUTPUT_FORMAT type can be used to inform the sender UE 110 which output format is used for the playback at the receiver UE 120.

[0241] HEAD_ORIENTATION type can be used to inform the sender UE 110 the head orientation of the listener at the receiver UE 120.

[0242] COMBINED_APPLIED_ORIENTATION type can be used to inform the sender UE 110 the total orientation that is applied to the processing at the receiver UE 120. The total orientation can constitute, for example, from head orientation of the listener, the scene orientation and from the capture device orientation.

[0243] REVERB_PARAMETERS type can be used to inform the sender UE 110 which reverb parameters the receiver UE 120 is using in their rendering process. The parameters can be used to optimize the mixing process in a server, for example, if the sender UE 110 is a mixing entity (e.g. an MCU).

[0244] Table 3 - Example indication / forward PI frame types.

[0245] The APPLIED_ORIENTATION type can be used to inform the receiver UE 120 which total orientation has been applied to the transmitted IVAS bitstream at the sender UE 110.

[0246] INPUT_FORMAT_CHANGE_NEXT type can be used to inform the receiver UE 120 that the input format is changed in the next IVAS frame that UE receiver 120 will receive. The change can be, for example, to mono or any other IVAS supported input format. Knowing beforehand that the input format will change can make the transition process more robust at the receiver UE 120. For example, the decoder for the upcoming input format can be initialized beforehand and the decoder state can be inherited from the current decoder. For example, the audio buffers of the second decoder can be filled with data from the first decoder to enable a smooth transition between the decoders, when the input format is switched.

[0247] A specific example where the application of the PI request frame 605 and PI response frame 609 exchange between the sender UE 110 and receiver UE 120 can be beneficial is switching from any IVAS stereo / spatial operation to EVS mono operation, where the following may occur.

[0248] Firstly, considering EVS. The speech coding core of EVS, based on the ACELP (algebraic code-excited linear prediction), handles packet loss at speech onsets by utilizing a so-called transition frame coding. The transition frame follows the onset, and its transmitted parameters make it possible to reconstruct a lost onset thus avoiding the propagation of the distortion over several speech frames. This mechanism significantly improves the EVS quality under packet loss scenarios. Now, if the onset is part of the IVAS stereo / spatial encoding, the codec switches to mono operation (using EVS) where the first mono frame would be a transition frame, and the first EVS frame is lost, the error will propagate for a long duration. When INPUT_FORMAT_CHANGE_NEXT is signaled, the EVS decoder (as part of IVAS) knows to initialize its state based on the IVAS stereo / spatial decoder properly. The onset is thus built based on the IVAS decoding, and the effect of the lost frame corresponds more closely to the regular EVS decoding despite the format change.

[0249] Furthermore INPUT_FORMAT_CHANGE_FUTURE type can be used in a manner similar to INPUT_FORMAT_CHANGE_NEXT, but the change of input format can occur after a specified number of frames or seconds or other suitable time format. The time of input format change can be indicated in the PI frame. By indicating the input format change well beforehand the receiver UE 120 receives more time to adapt to the upcoming change. With more time, the decoder of the upcoming input format can inherit more data from the first decoder which can result in a smoother transition between the decoders than with shorter transition period, e.g., as with INPUT_FORMAT_CHANGE_NEXT PI frames. In further examples, an indication can be provided directly to the user, e.g., via a user interface (III) such as smartphone display. For example, this indication can be useful for the user to be able to change a physical output device in order to have the optimal playback experience.

[0250] In addition to the timing information for the INPUT_FORMAT_CHANGE_FUTURE, the change could also be specified to occur at a specific type of time or detected type of audio signal. For example, the change could be requested to occur at a time of silence, non-speech, non-transient or other similar types. The time of the change would be unspecified, but changing the input format during a silence period, for example, can make the transition process more robust for the decoder. The decoder state does not necessarily need to be inherited during the transition if the input format change is occurring during a silence period. The change could, for example, be masked with DTX CNG (comfort noise) or PLC (packet loss concealment).

[0251] In some embodiments, the indication can be provided together with the IVAS frame for stereo format that the audio is already binauralized consequently the receiver UE is expected to provide the output directly to the headphones without any further processing. ACOUSTIC_ENVIRONMENT type can be used to convey acoustic environment data to the receiver. The data consists of room acoustics data such as, for example, absorption coefficients and reverberation time. The data is applied in the rendering process at the receiver.

[0252] In IVAS Compact format, only a single IVAS frame is transmitted in a RTP packet. That is, the packet does not comprise a ToC header. The PI frames, however, require a ToC byte to be recognized as PI frames by the receiver. In some embodiments PI frames can therefore be supported in a IVAS Compact format by integrating the PI frames into a RTP header extension.

[0253] In some embodiments PI frames can be supported in an IVAS Compact format by sending PI frames and their associated ToC bytes as separate packets from the IVAS frame packets. In these implementations the packets containing PI frames have a different payload size to the protected IVAS Compact format payload sizes. This could be implemented, for example, by zero padding the PI frame packets not to collide with the protected payload sizes. The zero padding could be achieved with individual bits, bytes or any other suitable means. The zero padding of the PI frame packets could be indicated with the padding (P) bit in the RTP header. The associated IVAS frame for the PI frame could furthermore be indicated by setting the timestamp of the PI frame packet to the same value as the timestamp of the IVAS frame packet. The same timestamp would indicate that the packets should be processed simultaneously by the receiver.

[0254] The PI frame design is such that each PI frame requires an associated ToC byte. However, if a RTP packet contains multiple PI frames, it could be beneficial to signal the presence of all the PI frames with a single compound header. This could be achieved, for example, by reserving one bit flag for each of the known PI frame types. The flag would indicate if the associated PI frame type is present in the current payload or not. For example, for three different PI frame types, a 3-bit header could be used to indicate the presence of each PI frame type.

[0255] Table 4 presented below shows an example of the compound PI frame header approach for three different PI frame types (HEAD_ORIENTATION, LABEL and INPUT_FORMAT_CHANGE).

[0256] Table 4 - Example 3-bit PI frame compound header.

[0257] The presence of the three different PI frames could be indicated in some embodiments by a 3-bit header, where each bit is associated with one of the available PI frame types. The first bit (from the left) is associated with INPUT_FORMAT_CHANGE, the second bit is associated with LABEL and the last bit is associated with HEAD_ORIENTATION. A header value of 001 , for example, indicates the presence of head orientation PI frame only. A value of 110 indicates the presence of both input format change and labelling PI frames in the payload, and so on. The PI frames could be attached to the payload after the 3-bit header. More PI frame types could be supported by adding more bits to the compound header.

[0258] Single PI frames could be supported together with the compound PI frames. For example, the first bit in the PI header could be used to indicate if the frame is a single or compound PI frame. In another embodiment, the single and compound PI frames could each have their own SSB and BLI combination in the IVAS ToC byte to indicate the type. For example, the single PI frame could have (SSB=011 , B Ll= 1110) and the compound PI frame could have (SSB=101 , SSB=1110).

[0259] In some embodiments the receiver UE 120 can be configured to request the sender UE 110 to switch the IVAS input format for the session. The sender UE 110 and receiver UE 120 notations follow the scenario presented in Figure 6.

[0260] For example, the requested format can be based on an initial list of formats offered or agreed in SDP negotiation. For example, if the receiver UE 120 is switching to use an external Tenderer that is specifically tuned for a specific IVAS input format (e.g., MASA), it is beneficial for the receiver UE 120 to receive that input format from the sender UE 110. In such a scenario, the receiver UE 120 can request the sender UE 110 to switch the IVAS input format accordingly.

[0261] The input format change request could be integrated to the IVAS ToC byte in a manner similar to the CMR, as explained in GB2314333.2. In summary these embodiments integrate the CMR into the IVAS ToC byte by reserving one SSB combination (e.g., 111 ) for signaling CMR and the BLI bits can be configured to indicate the requested bitrate. Similarly, one SSB combination (e.g., 110) could be reserved to signal an input format change request and the BLI bits could in some embodiments indicate the requested IVAS input format.

[0262] Table 5 shows example SSB and BLI allocations for input format change requests. Table 5 - Example SSB and BLI allocations for informing input format request. In other embodiments, the requested input formats could be structured differently to those as shown in Table 5. For example, in some embodiments the multichannel (MC) formats could be grouped together as a one type of requested format, and object formats (ISM, OMASA, OSBA) could be expanded to multiple input format requests to cover different number of objects in each request. The SBA input format request could also be expanded to cover the different SBA formats in IVAS (FOA, HOA2, HOA3).

[0263] In some embodiments the input format change request is supported in an IVAS session by employing PI frames for the request.

[0264] For example Figure 7 shows an example input format change request PI frame 701.

[0265] The input format change request PI frame 701 comprises a type field 703 set to INPUT_FORMAT_CHANGE.

[0266] Additionally the input format change request PI frame 701 comprises a usage field 705. The usage field is filled with a REQUEST value to inform the sender UE 110 that the PI frame should be treated as a request from the receiver UE 120 rather than information data.

[0267] The input format change request PI frame 701 further comprises a validity field 407 and size field 409 such as described previously.

[0268] The input format change request PI frame 701 additionally comprises a IVAS_input_format 711 value indicating which input format the receiver UE is requesting.

[0269] In some embodiments an input format change request can be transmitted to the sending UE 110 via a RTP header extension. For example, the input format change request PI frame could be integrated to the RTP header extension. In some embodiments, the change request could be transmitted via RTCP. In an IVAS session, most scenarios requesting an input format change occurs less frequently than, for example, requesting bitrate change from the sender UE 110. Consequently, the input format change request does not necessarily need to be transmitted at the same rate as the audio payload and could be handled, e.g., through RTCP.

[0270] In some embodiments the sender UE 110 can be informed that the receiver UE 120 has changed the output playback format. The sender UE 110 and receiver UE 120 can follow the scenario presented in Figure 6. For example, if the receiver UE 120 is switching from some IVAS supported output format to a mono output, it might be beneficial for the sender UE 110 to adjust the transmitted bitstream. For example, if the transmitted input format uses many channels, the sender UE 110 can be configured to reduce the number of transmitted channels without affecting the output quality of receiver UE 120 significantly because the output is employing a single channel playback. Consequently, the bitrate of the transmitted bitstream can be reduced.

[0271] In another example, the receiver UE 120 is switching from a binaural output to a loudspeaker output (or vice versa), which, depending on the bit rate, can trigger the sender UE 120 to modify encoder input format generation for a multi-channel input such that channel count is reduced, e.g., from 7.1.4 to 7.1 (or in the opposite case, adding channels to 7.1 input to provide a 7.1.4 input). This example corresponds to an input format modification technique disclosed in patent application 2310063.9 which optimizes perceptual quality of IVAS-encoded multichannel audio based on output format.

[0272] The output format change notification could, in some embodiments, be integrated to the IVAS ToC byte in a manner similar to CMR, as explained in GB2314333.2, and similarly as the input format change request described above. In such embodiments one SSB combination (e.g., 101 ) could be reserved to signal output format change and the BLI bits would indicate the output format.

[0273] Table 6 shows an example SSB and BLI allocations for output format change notifications.

[0274] Table 6 - Example SSB and BLI allocations for informing output format change.

[0275] In some embodiments, the output formats could be structured differently from the example shown in Table 6. For example, the multichannel (MC) or SBA formats (FOA, HOA2, HOA3) could be grouped together as a one type of output format.

[0276] In some embodiments output format change notification can be supported in an IVAS session by employing PI frames for the notification.

[0277] Figure 8 shows an example output format change PI frame.

[0278] The output format change request PI frame 801 comprises a type field 803 set to OUTPUT_FORMAT_CHANGE.

[0279] Additionally the output format change request PI frame 801 comprises a usage field 805. The usage field is filled with a FEEDBACK value to inform the sender UE 110 that the PI frame should be treated as feedback information from the receiver UE 120 and not as a request to change the output format of the sender UE 110.

[0280] The output format change request PI frame 801 further comprises a validity field 407 and size field 409 such as described previously.

[0281] The output format change request PI frame 801 additionally comprises a IVAS_output_format 811 value indicating which format the receiver UE is switching to.

[0282] In some embodiments the output format change can be informed to the other UE via the RTP header extension. For example, the output format change PI frame could be integrated to the RTP header extension. In some further embodiments, the output format change information could be transmitted via RTCP. In an IVAS session, in most scenarios an output format change occurs less frequently than, for example, requesting bitrate change from the sender UE 110. Consequently, the output format change does not necessarily need to be transmitted at the same rate as the audio payload and could be handled, e.g., through RTCP.

[0283] In some embodiments, for an IVAS session, the supported PI frame types could be negotiated during the SDP negotiation phase before the session is established. For example, the different PI frame types (e.g. as listed in Tables 1 , 2 and 3) could be offered in an SDP offer, and the SDP answer could pick only some of the offered PI frame types to be supported in the session. For example, if both parties are not using any head tracking, the HEAD_ORIENTATION PI frame type could be disabled for the session as it serves no purpose during the call. Additionally, the bi-directionality of the PI frames could in some embodiments be negotiated for. For example, the request, response and feedback PI frames could be disabled for the session if the session is unidirectional (i.e., the data is transmitted only from sender UE 110 to receiver UE 120).

[0284] Figure 9, for example, shows flow diagrams of the session negotiation offer creation and answer creation processes according to some embodiments.

[0285] With respect to the session negotiation creation process there is shown, by 901 , of the operation of obtaining one or more processing information (PI) frame types.

[0286] Following this, as shown by 902, is including the PI frame types in the session description file.

[0287] Then is shown, by 903, is including a parameter to enable bi-directional PI frames in the session description file.

[0288] This then results, as shown by 904, in generating session negotiation offer to a receiver user equipment.

[0289] With respect to the session negotiation answer process there is shown, by 911 , of the operation of receiving a session negotiation offer.

[0290] Following this, as shown by 912, is parsing one or more PI frame types in the received session negotiation offer.

[0291] Then is shown, by 913, obtaining one or more support for PI frame types in the session negotiation offer. After which is shown, by 914, obtaining support for bi-directional PI in the session negotiation offer.

[0292] Then is shown, by 915, including the list of supported PI frame types and support for bi-directional PI frames within a session negotiation answer session description file.

[0293] This then results, as shown by 916, in transmitting the session negotiation answer to the sender user equipment which delivered the session negotiation offer.

[0294] Figure 10 for example shows flow diagrams of the packetization and depacketization of the multiple IVAS streams according to some embodiments.

[0295] With respect to the packetization of the multiple IVAS streams there is shown, by 1001 , the operation of obtaining request or feedback processing information (PI) frame(s).

[0296] Following this, as shown by 1002, is obtaining IVAS frame(s).

[0297] Then is shown, by 1003, generating RTP payload header for each request or feedback PI frame.

[0298] After which is shown, by 1004, generating IVAS RTP payload header for each IVAS frame.

[0299] There follows, as shown by 1005, appending the above IVAS and PI frame(s) to the above RTP (IVAS and PI) payload headers.

[0300] Then is the operation of, as shown by 1006, including the resulting bitstream comprising the PI frame header(s), IVAS frame header(s), PI frame(s) and IVAS frame(s) in the RTP payload.

[0301] This then results in the operation, as shown by 1007, of transmitting RTP packet to the encoder / sender user equipment.

[0302] With respect to the depacketization of the multiple IVAS streams there is shown, by 1011 , of the operation of receiving a RTP packet.

[0303] Following this, as shown by 1012, is extracting RTP payload and determining presence of PI frame(s) in the RTP payload.

[0304] Then is shown, by 1013, determining or detecting feedback PI frame type and read the data.

[0305] After which is shown, by 1014, taking the feedback data into account in the encoding / capturing process. Alternatively, after extracting the RTP payload and determining presence of PI frame(s), there can be as shown by 1015, detect a request PI frame type and read the data.

[0306] Then follows, as shown by 1016, creating a response PI frame based on the request OR change the encoding processing based on the request.

[0307] Furthermore as shown by 1017, attaching the response PI frame and a related payload header to an IVAS RTP payload if a response PI frame was created in 1016.

[0308] This results in, as shown by 1018, transmitting RTP packet to the decoder / receiver user equipment

[0309] With respect to Figure 11 an example electronic device is shown. The device may be any suitable electronics device or apparatus. For example in some embodiments the device 1900 is a mobile device, user equipment, tablet computer, computer, audio playback apparatus, etc.

[0310] In some embodiments the device 1900 comprises at least one processor or central processing unit 1907. The processor 1907 can be configured to execute various program codes such as the methods such as described herein.

[0311] In some embodiments the device 1900 comprises a memory 1911. In some embodiments the at least one processor 1907 is coupled to the memory 1911. The memory 1911 can be any suitable storage means. In some embodiments the memory 1911 comprises a program code section for storing program codes implementable upon the processor 1907. Furthermore in some embodiments the memory 1911 can further comprise a stored data section for storing data, for example data that has been processed or to be processed in accordance with the embodiments as described herein. The implemented program code stored within the program code section and the data stored within the stored data section can be retrieved by the processor 1907 whenever needed via the memory-processor coupling.

[0312] In some embodiments the device 1900 comprises a user interface 1905. The user interface 1905 can be coupled in some embodiments to the processor 1907. In some embodiments the processor 1907 can control the operation of the user interface 1905 and receive inputs from the user interface 1905. In some embodiments the user interface 1905 can enable a user to input commands to the device 1900, for example via a keypad. In some embodiments the user interface 1905 can enable the user to obtain information from the device 1900. For example the user interface 1905 may comprise a display configured to display information from the device 1900 to the user. The user interface 1905 can in some embodiments comprise a touch screen or touch interface capable of both enabling information to be entered to the device 1900 and further displaying information to the user of the device 1900.

[0313] In some embodiments the device 1900 comprises an input / output port 1909. The input / output port 1909 in some embodiments comprises a transceiver. The transceiver in such embodiments can be coupled to the processor 1907 and configured to enable a communication with other apparatus or electronic devices, for example via a wireless communications network. The transceiver or any suitable transceiver or transmitter and / or receiver means can in some embodiments be configured to communicate with other electronic devices or apparatus via a wire or wired coupling.

[0314] The transceiver can communicate with further apparatus by any suitable known communications protocol. For example in some embodiments the transceiver can use a suitable universal mobile telecommunications system (UMTS) protocol, a wireless local area network (WLAN) protocol such as for example IEEE 802. X, a suitable short-range radio frequency communication protocol such as Bluetooth, or infrared data communication pathway (IRDA).

[0315] The transceiver input / output port 1909 may be configured to receive the signals and in some embodiments obtain the focus parameters as described herein.

[0316] In some embodiments the device 1900 may be employed to generate a suitable audio signal using the processor 1907 executing suitable code. The input / output port 1909 may be coupled to any suitable audio output for example to a multichannel speaker system and / or headphones (which may be a headtracked or a non-tracked headphones) or similar.

[0317] In general, the various embodiments of the invention may be implemented in hardware or special purpose circuits, software, logic or any combination thereof. For example, some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software which may be executed by a controller, microprocessor or other computing device, although the invention is not limited thereto. While various aspects of the invention may be illustrated and described as block diagrams, flow charts, or using some other pictorial representation, it is well understood that these blocks, apparatus, systems, techniques or methods described herein may be implemented in, as non-limiting examples, hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controller or other computing devices, or some combination thereof.

[0318] The embodiments of this invention may be implemented by computer software executable by a data processor of the mobile device, such as in the processor entity, or by hardware, or by a combination of software and hardware. Further in this regard it should be noted that any blocks of the logic flow as in the Figures may represent program steps, or interconnected logic circuits, blocks and functions, or a combination of program steps and logic circuits, blocks and functions. The software may be stored on such physical media as memory chips, or memory blocks implemented within the processor, magnetic media such as hard disk or floppy disks, and optical media such as for example DVD and the data variants thereof, CD.

[0319] The memory may be of any type suitable to the local technical environment and may be implemented using any suitable data storage technology, such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory and removable memory. The data processors may be of any type suitable to the local technical environment, and may include one or more of general-purpose computers, special purpose computers, microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASIC), gate level circuits and processors based on multi-core processor architecture, as non-limiting examples.

[0320] Embodiments of the inventions may be practiced in various components such as integrated circuit modules. The design of integrated circuits is by and large a highly automated process. Complex and powerful software tools are available for converting a logic level design into a semiconductor circuit design ready to be etched and formed on a semiconductor substrate. Programs, such as those provided by Synopsys, Inc. of Mountain View, California and Cadence Design, of San Jose, California automatically route conductors and locate components on a semiconductor chip using well established rules of design as well as libraries of pre-stored design modules. Once the design for a semiconductor circuit has been completed, the resultant design, in a standardized electronic format (e.g., Opus, GDSII, or the like) may be transmitted to a semiconductor fabrication facility or "fab" for fabrication.

[0321] The foregoing description has provided by way of exemplary and nonlimiting examples a full and informative description of the exemplary embodiment of this invention. However, various modifications and adaptations may become apparent to those skilled in the relevant arts in view of the foregoing description, when read in conjunction with the accompanying drawings and the appended claims. However, all such and similar modifications of the teachings of this invention will still fall within the scope of this invention as defined in the appended claims.

[0322] 3GPP 3rdGeneration Partnership Project

[0323] AMR-WB IO Adaptive Multi Rate Wideband Inter-Operable

[0324] BLI Bitrate Level Indication

[0325] CMR Codec mode request

[0326] CNG Comfort Noise Generator

[0327] DTX Discontinuous Transmission

[0328] EVS Enhanced Voice Services

[0329] FOA First-Order Ambisonics

[0330] FT Frame type (index)

[0331] HOA2 2ndorder Higher-Order Ambisonics

[0332] HOA3 3rdorder Higher-Order Ambisonics

[0333] ISM Independent Streams with Metadata

[0334] (i.e. , type of Object-Based Audio)

[0335] IVAS Immersive Voice and Audio Services kbps kilobits per second

[0336] MASA Metadata-Assisted Spatial Audio

[0337] MC Multichannel MCU Multipoint Conferencing Unit

[0338] OMASA Object-based audio with MASA (combined input format) OSBA Object-based audio with SBA (combined input format) PI Processing information (audio) PLC Packet Loss Concealment

[0339] RTCP Real-Time Transport Control Protocol

[0340] RTP Real-Time Transport Protocol

[0341] SBA Scene-Based Audio

[0342] SDP Session Description Protocol

[0343] SSB Supplemental Signaling Bits

[0344] ToC Table of Content

[0345] UE User equipment

Claims

CLAIMS:1 . An apparatus for controlling an immersive audio transmission comprising at least partially a bidirectional signalling, the apparatus comprising means configured to: obtain at least one immersive audio payload comprising at least one immersive audio coded bitstream; obtain at least one reverse direction processing information, the at least one reverse direction processing information for providing at least in part the bidirectional signalling; and packetize the at least one reverse direction processing information in a transport protocol payload.

2. The apparatus as claimed in claim 1 , wherein the at least one immersive audio payload further comprises at least one processing information frame for assisting the processing of the at least one immersive audio coded bitstream.

3. The apparatus as claimed in any of claims 1 or 2, wherein the at least one reverse direction processing information is associated with the at least one immersive audio payload.

4. The apparatus as claimed in any of claims 1 to 3, wherein the means configured to packetize the at least one reverse direction processing information in a transport protocol payload is configured to packetize the at least one reverse direction processing information in at least one of: a real-time transport protocol payload a real-time transport protocol header extension; and a real-time transport control protocol payload.

5. The apparatus as claimed in any of claims 1 to 4, wherein the means is further configured to transmit the packetized at least one reverse direction processing information to a further apparatus.

6. The apparatus as claimed in any of claims 1 to 5, wherein the means is further configured to at least one of: obtain at least one response processing information with an identifier header, the at least one response processing information being obtained in response to the at least one reverse direction processing information; and obtain the at least one response processing information within at least one of: a real-time transport protocol payload; a real-time transport protocol header extension; and a real-time transport control protocol payload.

7. The apparatus as claimed in any of claims 1 to 6, wherein the at least one reverse direction processing information comprises at least one of: an action request; a request to change an IVAS input format; a request to change a coded format; a request for an orientation of the further apparatus; a request for configuration information relating to a microphone array constellation of the further apparatus; a request to switch a bitrate from the further apparatus to the apparatus; a request for binauralization data that can be used to revert the binauralization at the apparatus; information relating to the apparatus, for assisting the at least one further apparatus in encoding the at least one immersive audio coded bitstream; information relating to a class of the reverse direction processing information; information relating to the apparatus orientation and / or position; information relating to a head rotation of a listener of the at least one immersive audio coded bitstream; information relating to a position of a listener of the at least one immersive audio coded bitstream; information relating to a loudspeaker constellation associated with the apparatus;information relating to a change in an output audio format; information relating to a combined orientation applied at the apparatus; and information related to reverberation parameters applied at the apparatus.

8. The apparatus as claimed in any of claims 1 to 7, wherein the at least one reverse direction processing information further comprises at least one of: a time value indicating a time of the reverse direction processing information, to cause the further apparatus to process the reverse direction processing information based on the time value; an urgency value, to cause the further apparatus to process the reverse direction processing information relative to a further reverse direction processing information based on the urgency value; and a priority value, to cause the further apparatus to process the reverse direction processing information relative to a further reverse direction processing information based on the priority value.

9. The apparatus as claimed in any of claims 1 to 8, wherein the means configured to packetize the at least one reverse direction processing information in a transport protocol payload is configured to perform one of: attach an identifier to the header of the transport protocol payload; attach at least one reverse direction processing information frame to the transport protocol payload; and attach at least one response PI frame and a related payload header to an IVAS real-time transport protocol payload.

10. The apparatus as claimed in any of claims 1 to 9, wherein the at least one reverse direction processing information comprises at least one of: information identifying an orientation of further apparatus or capturing apparatus; information identifying a microphone array constellation configuration of further apparatus or capturing apparatus; information identifying an orientation applied to the at least one immersive audio coded bitstream;information identifying a subsequent input format change for the at least one immersive audio coded bitstream; information identifying a subsequent coded format change for the at least one immersive audio coded bitstream; information identifying a specified number of frames or number of seconds until a change in an input format for the at least one immersive audio coded bitstream; information identifying binauralization data that can be used to revert the binauralization at the apparatus; and information about an acoustic environment.

11. The apparatus as claimed in any of claims 1 to 10, wherein the means is further configured to at least one of: negotiate with a further apparatus support for the reverse direction processing information; and negotiate with the further apparatus is configured to negotiate employing a session description file.

12. An apparatus for controlling bidirectional audio transmission comprising at least partially a bidirectional signalling, the apparatus comprising means configured to: transmit at least one immersive audio payload comprising at least one immersive audio coded bitstream; obtain packetized at least one reverse direction processing information, the packetized at least one reverse direction processing information for providing at least in part the bidirectional signalling; and process the packetized at least one reverse direction processing information.

13. The apparatus as claimed in claim 12, wherein the at least one immersive audio payload further comprises at least one processing information frame for assisting the processing of the at least one immersive audio coded bitstream.

14. The apparatus as claimed in any of claim 12 or 13, wherein the at least one reverse direction processing information is associated with the at least one immersive audio payload.

15. The apparatus as claimed in any of claims 12 to 14, wherein the means configured to obtain the packetized at least one reverse direction processing information from at least one of: a real-time transport protocol payload; a real-time transport protocol header extension; a real-time transport control protocol payload; and a further apparatus.

16. The apparatus as claimed in claim 15, wherein the means configured to obtain the packetized at least one reverse direction processing information is configured to one of: obtain the packetized at least one reverse direction processing information as part of a packet comprising at least one further at least one immersive audio coded bitstream; and obtain the packetized at least one reverse direction processing information separate to a further packet comprising at least one further at least one immersive audio coded bitstream.

17. The apparatus as claimed in any of claims 12 to 16, wherein the means is further configured to transmit at least one response processing information with an identifier header, the at least one response processing information being generated in response to the processing of the at least one reverse direction processing information.

18. The apparatus as claimed in claim 17, wherein the means configured to transmit the at least one response processing information is configured to transmit the at least one response processing information within at least one of: a real-time transport protocol payload; a real-time transport protocol header extension; anda real-time transport control protocol payload.

19. The apparatus as claimed in any of claims 12 to 18, wherein the reverse direction processing information comprises at least one of: an action request; a request to change an IVAS input format; a request to change a coded format; a request for an orientation of the apparatus; a request for configuration information relating to a microphone array constellation of the apparatus; a request to switch a bitrate from the apparatus to the further apparatus; a request for binauralization data that can be used to revert the binauralization at the further apparatus; information relating to the further apparatus, for assisting the apparatus in encoding the at least one immersive audio coded bitstream; information relating to the apparatus orientation and / or position; information relating to a class of the reverse direction processing information; information relating to a head rotation of a listener of the at least one immersive audio coded bitstream; information relating to a position of a listener of the at least one immersive audio coded bitstream; information relating to a loudspeaker constellation associated with the further apparatus; information relating to a change in an output audio format; information relating to a combined orientation applied at the further apparatus; and information related to reverberation parameters applied at the further apparatus.

20. The apparatus as claimed in any of claims 12 to 19, wherein the reverse direction processing information further comprises at least one of:a time value indicating a time of the reverse direction processing information, to cause the apparatus to process the reverse direction processing information based on the time value; an urgency value, to cause the apparatus to process the reverse direction processing information relative to a further reverse direction processing information based on the urgency value; and a priority value, to cause the apparatus to process the reverse direction processing information relative to a further reverse direction processing information based on the priority value.

21. The apparatus as claimed in any of claims 12 to 20, wherein the reverse direction processing information comprises at least one of: information identifying an orientation of the apparatus or a capturing apparatus; information identifying a microphone array constellation configuration of the apparatus or a capturing apparatus; information identifying an orientation applied to the at least one immersive audio coded bitstream; information identifying a subsequent changes input format for the at least one immersive audio coded bitstream; information identifying a subsequent coded format change for the at least one immersive audio coded bitstream; information identifying a specified number of frames or number of seconds until a change in an input format for the at least one immersive audio coded bitstream; information identifying binauralization data that can be used to revert the binauralization at the apparatus; and information about an acoustic environment.

22. The apparatus as claimed in any of claims 12 to 21 , wherein the means is further configured to at least one of:negotiate with a further apparatus support for the reverse direction processing information; and negotiate employing a session description file.

23. An apparatus for controlling an immersive audio transmission comprising at least partially a bidirectional signalling, the apparatus comprising at least one processor and at least one memory including a computer program code, the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus at least to: obtain at least one immersive audio payload comprising at least one immersive audio coded bitstream; obtain at least one reverse direction processing information, the at least one reverse direction processing information for providing at least in part the bidirectional signalling; and packetize the at least one reverse direction processing information in a transport protocol payload.

24. An apparatus for controlling an immersive audio transmission comprising at least partially a bidirectional signalling, the apparatus comprising at least one processor and at least one memory including a computer program code, the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus at least to: transmit at least one immersive audio payload comprising at least one immersive audio coded bitstream; obtain packetized at least one reverse direction processing information, the packetized at least one reverse direction processing information for providing at least in part the bidirectional signalling; and process the packetized at least one reverse direction processing information.

25. A method for an apparatus for controlling an immersive audio transmission comprising at least partially a bidirectional signalling, the method comprising:obtaining at least one immersive audio payload comprising at least one immersive audio coded bitstream; obtaining at least one reverse direction processing information, the at least one reverse direction processing information for providing at least in part the bidirectional signalling; and packetizing the at least one reverse direction processing information in a transport protocol payload.

26. A method for an apparatus for controlling bidirectional audio transmission comprising at least partially a bidirectional signalling, the method comprising: transmitting at least one immersive audio payload comprising at least one immersive audio coded bitstream; obtaining packetized at least one reverse direction processing information, the packetized at least one reverse direction processing information for providing at least in part the bidirectional signalling; and processing the packetized at least one reverse direction processing information.

Images