An apparatus and method for spatial audio editing

The apparatus and method for spatial audio editing and transmission in IVAS systems address the lack of efficient editing methods by precisely manipulating audio objects through various channels, enhancing the immersive audio experience.

GB2706014APending Publication Date: 2026-11-11NOKIA TECHNOLOGIES OY
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
GB2025005182
Authority / Receiving Office
GB · GB
Patent Type
Applications
Current Assignee / Owner
Filing Date
2025-04-07
Publication Date
2026-11-11

AI Technical Summary

Technical Problem

Existing immersive audio technologies lack efficient methods for spatial audio editing and transmission, particularly in immersive voice and audio services (IVAS), which are not adequately addressed by current 3GPP IVAS codecs and RTP protocols.

Method used

An apparatus and method for spatial audio editing and transmission that includes obtaining, modifying, encoding, and transmitting spatial audio signals based on spatial audio editing information received through various channels, such as IVAS RTP payloads, RTCP feedback messages, and IMS messages, allowing for precise editing of audio objects using parameters like gain, orientation, and position.

Benefits of technology

Enables precise editing and efficient transmission of spatial audio signals, enhancing the immersive audio experience by allowing for dynamic manipulation of audio objects, improving the quality of immersive audio services.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 00000000_0000_ABST
    Figure 00000000_0000_ABST
Patent Text Reader

Abstract

An Immersive Audio Voice Services (IVAS) transmission system obtains a spatial audio signal (of eg. a spatial ISM audio object with metadata in a transport signal) and spatial audio editing informatio
Need to check novelty before this filing date? Find Prior Art

Description

FIELD

[0001] The present application relates to a method, apparatus, system and computer program for object editing and handling object editing requests not exclusively within Immersive Voice and Audio Services (IVAS) applications. BACKGROUND

[0002] Spatial audio can be captured and represented in various ways. For mobile communications, e.g., in the scope of 3GPP services, the smartphone is an example capture device and form factor.

[0003] Additionally a spatial or immersive audio experience over headphones is possible by binaural reproduction (headphone playback) of a binaural recording or through a dedicated binauralization processing using a suitable rendering algorithm. As a result of binaural processing, a listener can thus experience an audio scene as if it surrounded them in real life.

[0004] 3GPP IVAS (Immersive Voice and Audio Services) is a new conversational audio codec standardized in 3GPP SA4 and part of Release 18. This is a versatile codec with many supported input formats and output formats. The main supported input formats for IVAS are stereo, multichannel (MC), object-based audio (ISM), scene-based audio (SBA), and Metadata-assisted spatial audio (MASA). In addition, the following combinations are supported: Objects with MASA (OMASA) and Objects with SBA (OSBA). IVAS furthermore includes the EVS codec for backwards compatible mono input operation. The IVAS output formats include mono, stereo, multi-channel (including custom loudspeaker layouts), FOA, HOA2, HOA3, and binaural. In addition, so-called pass-through operation is possible allowing, e.g., MASA output for MASA input. As a spatial audio codec supporting at least three degrees of rotation freedom (yaw, pitch, roll) for all spatial inputs, the IVAS codec is expected to be used in a variety of scenarios, all of which cannot be known beforehand.

[0005] The IVAS codec algorithm is described in 3GPP TS 26.253 (Codec for Immersive Voice and Audio Services; Detailed Algorithmic Description incl. RTP payload format and SDP parameter definitions). The IVAS codec floating-point C code is provided in 3GPP TS 26.258. IVAS codec fixed-point C code is currently being standardized for Release 19.

[0006] The IVAS codec bitstream contains information on the contents of the each bitstream frame. This information includes the aforementioned input format and additional information on the present format, i.e., sub format signaling, (e.g., order of Ambisonics, channel layout of MC, number of transport channels, etc.). This is in addition to the encoded audio signal and possible metadata. Furthermore, IVAS bitstream frames are designed to be standalone in such way that IVAS decoder can start decoding from any valid IVAS frame (containing full data of for the format specified in the bitstream) and produce good quality output.

[0007] RTP (Real Time Protocol) is designed to carry a multitude of multimedia formats, which permit the transport of new formats without revising the RTP standard. To this end, the information required by a specific application of the protocol is not included in the generic RTP header. For a class of applications (e.g., audio, video), an RTP profile may be defined. For a media format (e.g., a specific video coding format), an associated RTP payload format may be defined. Every instantiation of RTP in a particular application may require a profile and payload format specifications.

[0008] The profile defines the codecs used to encode the payload data and their mapping to payload format codes in the protocol field Payload Type (PT) of the RTP header.

[0009] For example, the RTP profile for audio and video conferences with minimal control is defined in RFC 3551. The profile defines a set of static payload type assignments, and a dynamic mechanism for mapping between a payload format, and a PT value using Session Description Protocol (SDP). The latter mechanism is used for newer video codecs such as RTP payload format for H.264 Video defined in RFC 6184 or RTP Payload Format for High Efficiency Video Coding (HEVC) defined in RFC 7798.

[0010] IVAS RTP payload format is currently being enhanced as part of IVAS_Codec_Ph2 work item in 3GPP SA4, and the latest state is described in TS 26.253 Annex A. Recent changes are also described in a CR document S4-241325.

[0011] An RTP session can be established for each multimedia stream. Audio and video streams may use separate RTP sessions, enabling a receiver to selectively receive components of a particular stream. The RTP specification recommends even port numbers for RTP, and the use of the next odd port number for the associated RTCP (RTP Control Protocol) session. A single port can be used for RTP and RTCP in applications that multiplex the protocols.

[0012] Each RTP stream consists of RTP packets, which in turn consist of RTP header and payload pairs. SUMMARY

[0013] According to a first aspect, there is provided an apparatus for immersive audio transmission, the apparatus comprising at least one processor and at least one memory storing instructions that, when executed by the at least one processor, cause the system at least to: obtain at least one spatial audio signal; obtain spatial audio editing information from a further apparatus; modify at least one spatial audio signal based on the spatial audio editing information; encode the modified at least one spatial audio signal; and transmit the encoded modified at least one spatial audio signal.

[0014] The apparatus caused to obtain spatial audio editing information from a further apparatus may be caused to receive the spatial audio editing information in one of: an IVAS RTP payload; an IVAS RTP packet; a reverse direction PI signalling field; a control channel RTCP feedback message; an RTP Header Extension data; a data channel IMS message; and a data channel non-IMS message.

[0015] The at least one spatial audio signal may comprise at least one audio object, the spatial audio editing information is at least one audio object editing information, and wherein the apparatus caused to modify at least one spatial audio signal based on the spatial audio editing information may be caused to modify the at least one audio object based on the at least one audio object editing information.

[0016] The apparatus caused to obtain at least one spatial audio signal may be caused to: obtain at least one input audio signal; and analyse the at least one input audio signal to determine the at least one spatial audio signal comprising the at least one audio object.

[0017] The at least one object editing information may comprise at least one of: gain editing information for modifying a gain associated with at least one identified audio object or all objects; orientation editing information for editing an orientation associated with at least one identified audio object or all the objects; yaw orientation editing information for editing a yaw orientation associated with at least one identified audio object or all the objects; pitch orientation editing information for editing a pitch orientation associated with at least one identified audio object or all the objects; roll orientation editing information for editing a roll orientation associated with at least one identified audio object or all the objects; positional editing information for editing a position associated with at least one identified audio object or all the objects; x-position editing information for editing a x-position associated with at least one identified audio object or all the objects; y-position editing information for editing a y-position associated with at least one identified audio object or all the objects; z-position editing information for editing a z-position associated with at least one identified audio object or all the objects; azimuth editing information for editing an azimuth position associated with at least one identified audio object or all the objects; elevation editing information for editing an elevation position associated with at least one identified audio object or all the objects; distance editing information for editing a distance associated with at least one identified audio object or all the objects; and directivity editing information for editing a directivity associated with at least one identified audio object or all the objects.

[0018] The apparatus caused to obtain at least one spatial audio signal may be further caused to obtain at least one audio object metadata, the at least one audio object metadata may comprise at least one spatial parameter.

[0019] The at least one spatial parameter may comprise at least one of: a direction and / or orientation to the at least one object; a position of the at least one object; a distance to the at least one object; an energy of the at least one object; an energy ratio of the at least one object; and a directivity of the at least one object.

[0020] The apparatus caused to modify at least one spatial audio signal based on the spatial audio editing information may be caused to modify the at least one spatial parameter based on the object editing information.

[0021] The apparatus caused to obtain at least one audio object metadata may be caused to: determine the at least one audio object metadata based on a sensor associated with a close microphone; determine the at least one audio object metadata based on an input associated with a captured mono audio from a close microphone; and determine the at least one audio object metadata based on an analysis of capture microphone array audio signals, the at least one spatial audio signal generated from the microphone array audio signals.

[0022] The apparatus caused to encode the modified at least one spatial audio signal may be further caused to: determine at least one transport audio signal; and encode the at least one transport audio signal.

[0023] According to a second aspect there is provided a method for an apparatus for immersive audio transmission, the method comprising at least: obtaining at least one spatial audio signal; obtaining spatial audio editing information from a further apparatus; modifying at least one spatial audio signal based on the spatial audio editing information; encoding the modified at least one spatial audio signal; and transmitting the encoded modified at least one spatial audio signal.

[0024] Obtaining spatial audio editing information from a further apparatus may comprise receiving the spatial audio editing information in one of: an IVAS RTP payload; an IVAS RTP packet; a reverse direction PI signalling field; a control channel RTCP feedback message; an RTP Header Extension data; a data channel IMS message; and a data channel non-IMS message.

[0025] The at least one spatial audio signal may comprise at least one audio object, the spatial audio editing information may be at least one audio object editing information, and wherein modifying at least one spatial audio signal based on the spatial audio editing information may comprise modifying the at least one audio object based on the at least one audio object editing information.

[0026] Obtaining at least one spatial audio signal may comprise: obtaining at least one input audio signal; and analysing the at least one input audio signal to determine the at least one spatial audio signal comprising the at least one audio object.

[0027] The at least one object editing information may comprise at least one of: gain editing information for modifying a gain associated with at least one identified audio object or all object; orientation editing information for editing an orientation associated with at least one identified audio object or all the objects; yaw orientation editing information for editing a yaw orientation associated with at least one identified audio object or all the objects; pitch orientation editing information for editing a pitch orientation associated with at least one identified audio object or all the objects; roll orientation editing information for editing a roll orientation associated with at least one identified audio object or all the objects; positional editing information for editing a position associated with at least one identified audio object or all the objects; x-position editing information for editing a x-position associated with at least one identified audio object or all the objects; y-position editing information for editing a y-position associated with at least one identified audio object or all the objects; z-position editing information for editing a z-position associated with at least one identified audio object or all the objects; azimuth editing information for editing an azimuth position associated with at least one identified audio object or all the objects; elevation editing information for editing an elevation position associated with at least one identified audio object or all the objects; distance editing information for editing a distance associated with at least one identified audio object or all the objects; and directivity editing information for editing a directivity associated with at least one identified audio object or all the objects.

[0028] Obtaining at least one spatial audio signal may comprise obtaining at least one audio object metadata, the at least one audio object metadata comprising at least one spatial parameter.

[0029] The at least one spatial parameter may comprise at least one of: a direction and / or orientation to the at least one object; a position of the at least one object; a distance to the at least one object; an energy of the at least one object; an energy ratio of the at least one object; and a directivity of the at least one object.

[0030] Modifying at least one spatial audio signal based on the spatial audio editing information may comprise modifying the at least one spatial parameter based on the object editing information.

[0031] Obtaining at least one audio object metadata may comprise: determining the at least one audio object metadata based on a sensor associated with a close microphone; determining the at least one audio object metadata based on an input associated with a captured mono audio from a close microphone; and determining the at least one audio object metadata based on an analysis of capture microphone array audio signals, the at least one spatial audio signal generated from the microphone array audio signals.

[0032] Encoding the modified at least one spatial audio signal may further comprise: determining at least one transport audio signal; and encoding the at least one transport audio signal.

[0033] According to a third aspect, there is provided an apparatus for immersive audio transmission, the apparatus comprising means configured to: obtain at least one spatial audio signal; obtain spatial audio editing information from a further apparatus; modify at least one spatial audio signal based on the spatial audio editing information; encode the modified at least one spatial audio signal; and transmit the encoded modified at least one spatial audio signal.

[0034] The means configured to obtain spatial audio editing information from a further apparatus may be configured to receive the spatial audio editing information in one of: an IVAS RTP payload; an IVAS RTP packet; a reverse direction PI signalling field; a control channel RTCP feedback message; an RTP Header Extension data; a data channel IMS message; and a data channel non-IMS message.

[0035] The at least one spatial audio signal may comprise at least one audio object, the spatial audio editing information is at least one audio object editing information, and wherein the apparatus caused to modify at least one spatial audio signal based on the spatial audio editing information may be caused to modify the at least one audio object based on the at least one audio object editing information.

[0036] The means configured to obtain at least one spatial audio signal may be configured to: obtain at least one input audio signal; and analyse the at least one input audio signal to determine the at least one spatial audio signal comprising the at least one audio object.

[0037] The at least one object editing information may comprise at least one of: gain editing information for modifying a gain associated with at least one identified audio object or all objects; orientation editing information for editing an orientation associated with at least one identified audio object or all the objects; yaw orientation editing information for editing a yaw orientation associated with at least one identified audio object or all the objects; pitch orientation editing information for editing a pitch orientation associated with at least one identified audio object or all the objects; roll orientation editing information for editing a roll orientation associated with at least one identified audio object or all the objects; positional editing information for editing a position associated with at least one identified audio object or all the objects; x-position editing information for editing a x-position associated with at least one identified audio object or all the objects; y-position editing information for editing a y-position associated with at least one identified audio object or all the objects; z-position editing information for editing a z-position associated with at least one identified audio object or all the objects; azimuth editing information for editing an azimuth position associated with at least one identified audio object or all the objects; elevation editing information for editing an elevation position associated with at least one identified audio object or all the objects; distance editing information for editing a distance associated with at least one identified audio object or all the objects; and directivity editing information for editing a directivity associated with at least one identified audio object or all the objects.

[0038] The means configured to obtain at least one spatial audio signal may be further configured to obtain at least one audio object metadata, the at least one audio object metadata may comprise at least one spatial parameter.

[0039] The at least one spatial parameter may comprise at least one of: a direction and / or orientation to the at least one object; a position of the at least one object; a distance to the at least one object; an energy of the at least one object; an energy ratio of the at least one object; and a directivity of the at least one object.

[0040] The means configured to modify at least one spatial audio signal based on the spatial audio editing information may be configured to modify the at least one spatial parameter based on the object editing information.

[0041] The means configured to obtain at least one audio object metadata may be configured to: determine the at least one audio object metadata based on a sensor associated with a close microphone; determine the at least one audio object metadata based on an input associated with a captured mono audio from a close microphone; and determine the at least one audio object metadata based on an analysis of capture microphone array audio signals, the at least one spatial audio signal generated from the microphone array audio signals.

[0042] The means configured to encode the modified at least one spatial audio signal may be further caused to: determine at least one transport audio signal; and encode the at least one transport audio signal.

[0043] According to a fourth aspect, there is an apparatus for immersive audio transmission, the apparatus comprising: obtaining circuitry configured to obtain at least one spatial audio signal; obtaining circuitry configured to obtain spatial audio editing information from a further apparatus; modifying circuitry configured to modify at least one spatial audio signal based on the spatial audio editing information; encoding circuitry configured to encode the modified at least one spatial audio signal; and transmitting circuitry configured to transmit the encoded modified at least one spatial audio signal.

[0044] According to a fifth aspect, there is an apparatus for immersive audio transmission comprising: means for obtaining at least one spatial audio signal; means for obtaining spatial audio editing information from a further apparatus; means for modifying at least one spatial audio signal based on the spatial audio editing information; means for encoding the modified at least one spatial audio signal; and means for transmitting the encoded modified at least one spatial audio signal.

[0045] According to a sixth aspect, there is provided a non-transitory computer readable medium comprising program instructions that, when executed by an apparatus for immersive audio transmission, cause the apparatus to perform at least the method according to any of the preceding aspects.

[0046] According to a seventh aspect there is provided a computer program comprising instructions [or a computer readable medium comprising instructions] for causing an apparatus for immersive audio transmission, caused to perform at least the following: obtain at least one spatial audio signal; obtain spatial audio editing information from a further apparatus; modify at least one spatial audio signal based on the spatial audio editing information; encode the modified at least one spatial audio signal; and transmit the encoded modified at least one spatial audio signal.

[0047] According to an eighth aspect there is provided an apparatus for immersive audio rendering, the apparatus comprising at least one processor and at least one memory storing instructions that, when executed by the at least one processor, cause the system at least to: obtain a spatial audio editing input, the spatial audio editing input associated with modifying at least one spatial audio signal; generate spatial audio editing information based on the spatial audio editing input; transmit the spatial audio editing information to a further apparatus; obtain, from the further apparatus, a bitstream comprising an encoded modified spatial audio signal, the encoded modified spatial audio signal based on the spatial audio editing information; decode the encoded modified spatial audio signal; and render at least one output spatial audio signal based on the decoded modified spatial audio signal.

[0048] The apparatus caused to transmit the spatial audio editing information parameter to the further apparatus may be caused to transmit the spatial audio editing information in one of: an IVAS RTP payload; an IVAS RTP packet; a reverse direction PI signalling field; a control channel RTCP feedback message; an RTP Header Extension data; a data channel IMS message; and a data channel non-IMS message.

[0049] The encoded modified at least one spatial audio signal may comprise at least one audio object, the spatial audio editing input comprises an audio object editing input, and the spatial audio editing information is at least one audio object editing information.

[0050] The object editing information parameter may comprise at least one of: gain editing information for modifying a gain associated with at least one identified audio object or all objects; orientation editing information for editing an orientation associated with at least one identified audio object or all objects; yaw orientation editing information for editing a yaw orientation associated with at least one identified audio object or all objects; pitch orientation editing information for editing a pitch orientation associated with at least one identified audio object or all objects; roll orientation editing information for editing a roll orientation associated with at least one identified audio object or all objects; positional editing information for editing a position associated with at least one identified audio object or all objects; x-position editing information for editing a x-position associated with at least one identified audio object or all objects; y-position editing information for editing a y-position associated with at least one identified audio object or all objects; z-position editing information for editing a z-position associated with at least one identified audio object or all objects; azimuth editing information for editing an azimuth position associated with at least one identified audio object or all objects; elevation editing information for editing an elevation position associated with at least one identified audio object or all objects; distance editing information for editing a distance associated with at least one identified audio object or all objects; and directivity editing information for editing a directivity associated with at least one identified audio object or all objects.

[0051] The at least one spatial audio signal comprises at least one spatial parameter which comprises at least one of: a direction and / or orientation to the at least one object; a position of the at least one object; a distance to the at least one object; an energy of the at least one object; an energy ratio of the at least one object; and a directivity of the at least one object.

[0052] The at least one encoded modified spatial audio signal may further comprise at least one encoded transport audio signal, and the apparatus caused to decode the encoded modified spatial audio signal may be caused to decode the at least one encoded transport audio signal, wherein the apparatus caused to render at least one output spatial audio signal based on the decoded modified at least one spatial audio signal may be caused to render the at least one output spatial audio signal further based on the decoded at least one transport audio signal.

[0053] The apparatus caused to generate spatial audio editing information based on the spatial audio editing input may be caused to generate the spatial audio editing information further based on a determination of a number of audio objects within the at least one encoded modified spatial audio signal.

[0054] The apparatus caused to generate spatial audio editing information based on the spatial audio editing input may be caused to generate the spatial audio editing information parameter further based on a comparison between the spatial audio editing input and a threshold parameter.

[0055] According to a ninth aspect there is provided a method for an apparatus for immersive audio rendering, the method comprising at least: obtaining a spatial audio editing input, the spatial audio editing input associated with modifying at least one spatial audio signal; generating spatial audio editing information based on the spatial audio editing input; transmitting the spatial audio editing information to a further apparatus; obtaining, from the further apparatus, a bitstream comprising an encoded modified spatial audio signal, the encoded modified spatial audio signal based on the spatial audio editing information; decoding the encoded modified spatial audio signal; and rendering at least one output spatial audio signal based on the decoded modified spatial audio signal.

[0056] Transmitting the spatial audio editing information parameter to the further apparatus may comprise transmitting the spatial audio editing information in one of: an IVAS RTP payload; an IVAS RTP packet; a reverse direction PI signalling field; a control channel RTCP feedback message; an RTP Header Extension data; a data channel IMS message; and a data channel non-IMS message.

[0057] The encoded modified at least one spatial audio signal may comprise at least one audio object, the spatial audio editing input comprises an audio object editing input, and the spatial audio editing information is at least one audio object editing information.

[0058] The object editing information parameter may comprise at least one of: gain editing information for modifying a gain associated with at least one identified audio object or all objects; orientation editing information for editing an orientation associated with at least one identified audio object or all objects; yaw orientation editing information for editing a yaw orientation associated with at least one identified audio object or all objects; pitch orientation editing information for editing a pitch orientation associated with at least one identified audio object oral! objects; roll orientation editing information for editing a roll orientation associated with at least one identified audio object or all objects; positional editing information for editing a position associated with at least one identified audio object or all objects; x-position editing information for editing a x-position associated with at least one identified audio object or all objects; y-position editing information for editing a y-position associated with at least one identified audio object or all objects; z-position editing information for editing a z-position associated with at least one identified audio object or all objects; azimuth editing information for editing an azimuth position associated with at least one identified audio object or all objects; elevation editing information for editing an elevation position associated with at least one identified audio object or all objects; distance editing information for editing a distance associated with at least one identified audio object or all objects; and directivity editing information for editing a directivity associated with at least one identified audio object or all objects.

[0059] The at least one spatial audio signal may comprise at least one spatial parameter which comprises at least one of: a direction and / or orientation to the at least one object; a position of the at least one object; a distance to the at least one object; an energy of the at least one object; an energy ratio of the at least one object; and a directivity of the at least one object.

[0060] The at least one encoded modified spatial audio signal may further comprise at least one encoded transport audio signal, and decoding the encoded modified spatial audio signal comprises decoding the at least one encoded transport audio signal, wherein rendering at least one output spatial audio signal based on the decoded modified at least one spatial audio signal comprises rendering the at least one output spatial audio signal further based on the decoded at least one transport audio signal.

[0061] Generating spatial audio editing information based on the spatial audio editing input may comprise generating the spatial audio editing information further based on a determination of a number of audio objects within the at least one encoded modified spatial audio signal.

[0062] Generating spatial audio editing information based on the spatial audio editing input may comprise generating the spatial audio editing information parameter further based on a comparison between the spatial audio editing input and a threshold parameter.

[0063] According to a tenth aspect, there is provided an apparatus for immersive audio rendering, the apparatus comprising means configured to at least: obtain a spatial audio editing input, the spatial audio editing input associated with modifying at least one spatial audio signal; generate spatial audio editing information based on the spatial audio editing input; transmit the spatial audio editing information to a further apparatus; obtain, from the further apparatus, a bitstream comprising an encoded modified spatial audio signal, the encoded modified spatial audio signal based on the spatial audio editing information; decode the encoded modified spatial audio signal; and render at least one output spatial audio signal based on the decoded modified spatial audio signal.

[0064] The means configured to transmit the spatial audio editing information parameter to the further apparatus may be configured to transmit the spatial audio editing information in one of: an IVAS RTP payload; an IVAS RTP packet; a reverse direction PI signalling field; a control channel RTCP feedback message; an RTP Header Extension data; a data channel IMS message; and a data channel non-IMS message.

[0065] The encoded modified at least one spatial audio signal may comprise at least one audio object, the spatial audio editing input comprises an audio object editing input, and the spatial audio editing information is at least one audio object editing information.

[0066] The object editing information parameter may comprise at least one of: gain editing information for modifying a gain associated with at least one identified audio object or all objects; orientation editing information for editing an orientation associated with at least one identified audio object or all objects; yaw orientation editing information for editing a yaw orientation associated with at least one identified audio object or all objects; pitch orientation editing information for editing a pitch orientation associated with at least one identified audio object or all objects; roll orientation editing information for editing a roll orientation associated with at least one identified audio object or all objects; positional editing information for editing a position associated with at least one identified audio object or all objects; x-position editing information for editing a x-position associated with at least one identified audio object or all objects; y-position editing information for editing a y-position associated with at least one identified audio object or all objects; z-position editing information for editing a z-position associated with at least one identified audio object or all objects; azimuth editing information for editing an azimuth position associated with at least one identified audio object or all objects; elevation editing information for editing an elevation position associated with at least one identified audio object or all objects; distance editing information for editing a distance associated with at least one identified audio object or all objects; and directivity editing information for editing a directivity associated with at least one identified audio object or all objects.

[0067] The at least one spatial audio signal comprises at least one spatial parameter which comprises at least one of: a direction and / or orientation to the at least one object; a position of the at least one object; a distance to the at least one object; an energy of the at least one object; an energy ratio of the at least one object; and a directivity of the at least one object.

[0068] The at least one encoded modified spatial audio signal may further comprise at least one encoded transport audio signal, and the means configured to decode the encoded modified spatial audio signal may be configured to decode the at least one encoded transport audio signal, wherein the means configured to render at least one output spatial audio signal based on the decoded modified at least one spatial audio signal may be configured to render the at least one output spatial audio signal further based on the decoded at least one transport audio signal.

[0069] The means configured to generate spatial audio editing information based on the spatial audio editing input may be configured to generate the spatial audio editing information further based on a determination of a number of audio objects within the at least one encoded modified spatial audio signal.

[0070] The means configured to generate spatial audio editing information based on the spatial audio editing input may be configured to generate the spatial audio editing information parameter further based on a comparison between the spatial audio editing input and a threshold parameter.

[0071] According to an eleventh aspect, there is an apparatus for immersive audio rendering comprising: obtaining circuitry configured to obtain a spatial audio editing input, the spatial audio editing input associated with modifying at least one spatial audio signal; generating circuitry configured to generate spatial audio editing information based on the spatial audio editing input; transmitting circuitry configured to transmit the spatial audio editing information to a further apparatus; obtaining circuitry configured to obtain, from the further apparatus, a bitstream comprising an encoded modified spatial audio signal, the encoded modified spatial audio signal based on the spatial audio editing information; decoding circuitry configured to decode the encoded modified spatial audio signal; and rendering circuitry configured to render at least one output spatial audio signal based on the decoded modified spatial audio signal.

[0072] According to a twelfth aspect, there is an apparatus for immersive audio rendering, the apparatus comprising: means for obtaining a spatial audio editing input, the spatial audio editing input associated with modifying at least one spatial audio signal; means for generating spatial audio editing information based on the spatial audio editing input; means for transmitting the spatial audio editing information to a further apparatus; means for obtaining, from the further apparatus, a bitstream comprising an encoded modified spatial audio signal, the encoded modified spatial audio signal based on the spatial audio editing information; means for decoding the encoded modified spatial audio signal; and means for rendering at least one output spatial audio signal based on the decoded modified spatial audio signal.

[0073] .According to a thirteenth aspect, there is provided a non-transitory computer readable medium comprising program instructions that, when executed by an apparatus, cause the apparatus to perform at least the method according to any of the preceding aspects.

[0074] According to a fourteenth aspect there is provided a computer program comprising instructions [or a computer readable medium comprising instructions] for causing an apparatus to for immersive audio rendering, caused to perform at least the following: obtain a spatial audio editing input, the spatial audio editing input associated with modifying at least one spatial audio signal; generate spatial audio editing information based on the spatial audio editing input; transmit the spatial audio editing information to a further apparatus; obtain, from the further apparatus, a bitstream comprising an encoded modified spatial audio signal, the encoded modified spatial audio signal based on the spatial audio editing information; decode the encoded modified spatial audio signal; and render at least one output spatial audio signal based on the decoded modified spatial audio signal.

[0075] In the above, many different embodiments have been described. It should be appreciated that further embodiments may be provided by the combination of any two or more of the embodiments described above. DESCRIPTION OF FIGURES

[0076] Embodiments will now be described, by way of example only, with reference to the accompanying Figures in which:

[0077] Fig. 1 shows schematically example systems within which embodiments may be implemented;

[0078] Fig.2 shows an example IVAS RTP packet format configuration according to some embodiments;

[0079] Fig.3 shows an example spatial audio or object editing process at the decoder according to some embodiments;

[0080] Fig.4 shows an example spatial audio or object editing process at the encoder according to some embodiments;

[0081] Fig.5 shows schematically an example apparatus for implementing some embodiments;

[0082] Fig.6 shows schematically an example encoder as shown in Fig.5 in further detail according to some embodiments;

[0083] Fig.7 shows a flow diagram showing the operation of the example encoder as shown in Fig.6 according to some embodiments;

[0084] Fig.8 shows schematically an example decoder and renderer as shown in Fig.5 in further detail according to some embodiments;

[0085] Fig.9 shows a flow diagram showing the operation of an example decoder and renderer as shown in Fig.5 in further detail according to some embodiments;

[0086] Figs. 10a to 10c shows example spectrograms of the edited object according to some embodiments; and

[0087] Fig. 11 shows an example ISM_GAIN PI data frame structure;

[0088] Fig. 12 shows an example PI orientation data frame according to some embodiments;

[0089] Fig. 13 shows an example PI position data frame according to some embodiments;

[0090] Fig.14 shows an example ISM_DISTANCE_ATTENUATION PI data frame;

[0091] Fig. 15 shows an example ISM_DIRECTIVITY PI data frame;

[0092] Figs. 16a to 16c show example structures for an OBJECT_EDITING PI data type;

[0093] Fig. 17 shows an example SWAPJSM PI frame structure;

[0094] Fig. 18 shows a flow diagram showing the operations of an example method implementing some embodiments; and

[0095] Fig. 19 shows apparatus suitable for implementing some embodiments. DETAILED DESCRIPTION

[0096] The concept, as discussed in further detail in the embodiments herein, is related to apparatus and methods for immersive audio transmission, at least to: obtain at least one spatial audio signal; obtain spatial audio editing information from a further apparatus; modify at least one spatial audio signal based on the spatial audio editing information; encode the modified at least one spatial audio signal; and transmit the encoded modified at least one spatial audio signal.

[0097] The spatial audio editing information can be received in one of: an IVAS RTP payload; an IVAS RTP packet; a reverse direction PI signalling field; a control channel RTCP feedback message; an RTP Header Extension data; a data channel IMS message; and a data channel non-IMS message.

[0098] The at least one spatial audio signal can as described herein comprise at least one audio object, the spatial audio editing information is at least one audio object editing information, and wherein the apparatus caused to modify at least one spatial audio signal based on the spatial audio editing information is caused to modify the at least one audio object based on the at least one audio object editing information.

[0099] The at least one spatial audio signal can be obtained for example by obtaining at least one input audio signal and analysing the at least one input audio signal to determine the at least one spatial audio signal comprising the at least one audio object.

[0100] The at least one object editing information can comprises at least one of: gain editing information for modifying a gain associated with at least one identified audio object or all objects; orientation editing information for editing an orientation associated with at least one identified audio object or all the objects; yaw orientation editing information for editing a yaw orientation associated with at least one identified audio object or all the objects; pitch orientation editing information for editing a pitch orientation associated with at least one identified audio object or all the objects; roll orientation editing information for editing a roll orientation associated with at least one identified audio object or all the objects; positional editing information for editing a position associated with at least one identified audio object or all the objects; x-position editing information for editing a x-position associated with at least one identified audio object or all the objects; y-position editing information for editing a y-position associated with at least one identified audio object or all the objects; z-position editing information for editing a z-position associated with at least one identified audio object or all the objects; azimuth editing information for editing an azimuth position associated with at least one identified audio object or all the objects; elevation editing information for editing an elevation position associated with at least one identified audio object or all the objects; distance editing information for editing a distance associated with at least one identified audio object or all the objects; and directivity editing information for editing a directivity associated with at least one identified audio object or all the objects.

[0101] In some embodiments there is obtained at least one audio object metadata, the at least one audio object metadata comprising at least one spatial parameter. The at least one spatial parameter can comprise at least one of: a direction and / or orientation to the at least one object; a position of the at least one object; a distance to the at least one object; an energy of the at least one object; an energy ratio of the at least one object; a directivity of the at least one object.

[0102] The modifying at least one spatial audio signal based on the spatial audio editing information can be modifying the at least one spatial parameter based on the object editing information.

[0103] Obtaining at least one audio object metadata can be determining the at least one audio object metadata based on a sensor associated with a close microphone or determining the at least one audio object metadata based on an input associated with a captured mono audio from a close microphone or determining the at least one audio object metadata based on an analysis of capture microphone array audio signals, the at least one spatial audio signal generated from the microphone array audio signals.

[0104] Encoding the modified at least one spatial audio signal can be further determining at least one transport audio signal and encoding the at least one transport audio signal.

[0105] Similarly the concept can be implemented in some embodiments by an apparatus for immersive audio rendering, the apparatus obtaining a spatial audio editing input, the spatial audio editing input associated with modifying at least one spatial audio signal; generating spatial audio editing information based on the spatial audio editing input; transmitting the spatial audio editing information to a further apparatus; obtaining, from the further apparatus, a bitstream comprising an encoded modified spatial audio signal, the encoded modified spatial audio signal based on the spatial audio editing information; decoding the encoded modified spatial audio signal; and rendering at least one output spatial audio signal based on the decoded modified spatial audio signal.

[0106] The encoded modified at least one spatial audio signal can comprise at least one audio object, the spatial audio editing input comprises an audio object editing input, and the spatial audio editing information is at least one audio object editing information.

[0107] Fig.1, for example, shows an example immersive call or teleconferencing system within which some embodiments can be implemented. In this example there is shown two sites or rooms, Room A 100 and Room B 102. Room A 100 comprises a ‘talker’ or user, Talker 1 103. Room B 102 comprises one ‘talker’ or user, Talker RX 141.

[0108] In the following example within room A 100 is a suitable audio communication system 110, such as a teleconference apparatus (or more generally telecommunications apparatus 110 which can operate as capture apparatus) configured to spatially capture and encode the audio environment and furthermore is configured to render a spatial audio signal to the room. The apparatus 110 can in some embodiments be implemented by a user equipment (UE), such as a mobile communication device, or a smartphone operating within a communications system, e.g., a cellular communications system or accessing any suitable access network. The user equipment in some embodiments comprises a smartphone. In some further or additional use cases the apparatus 110 can be implemented in a vehicle.

[0109] Within each of the other spaces, e.g. rooms, may be a suitable audio communication device, such as a teleconference apparatus (or more generally telecommunications apparatus such as apparatus 120 within room B, which in the following examples can be designated as rendering apparatus) configured to render a spatial audio signal to the room. In some embodiments the apparatus 120 within room B can furthermore be configured to capture and encode at least a mono audio and optionally configured to spatially capture and encode the audio environment. The apparatus 120 can in some embodiments be implemented by a user equipment (UE), such as a mobile communication device, or a smartphone operating within a communication system, e.g., a cellular communications system or accessing any suitable access network. The user equipment in some embodiments comprises a smartphone. In some further or additional use cases the apparatus 120 can be implemented in a vehicle.

[0110] In the following examples each room is provided with the means to spatially capture, encode spatial audio signals, receive spatial audio signals, and render these in a suitable way to a listener. It would be understood that there may be other embodiments where the system comprises some apparatus configured to only capture and encode audio signals (in other words the apparatus is a ‘transmit’ only or purely capture apparatus), and other apparatus configured to only receive and render audio signals (in other words the apparatus is a ‘receive’ only or purely rendering apparatus). In such embodiments the system within which embodiments may be implemented may comprise apparatus with varying abilities to capture / render audio signals.

[0111] The telecommunications apparatus (for each space, site, or room) 110, 120 in this example can be configured to call into a teleconference controlled by and implemented over a server 111.

[0112] In some embodiments the communications or teleconferencing system comprises a (peer-to-peer) communications system (rather than the server-based system shown in Fig.1) within which some embodiments can be implemented. Thus, for example, two or more UEs can be configured to interact directly with each other (for example, to implement an immersive audio phone call between users) without the need for a central server.

[0113] For example, the communications or teleconferencing, e.g., the immersive audio phone call, between users can be an IMS (IP (Internet Protocol) Multimedia Subsystem) call over 4G (Generation) (LTE (Long Term Evolution)), 5G (NR (New Radio)), or 6G link or any other suitable link, for example, WLAN (Wireless Local Area Network).

[0114] The telecommunications apparatus or UEs 110, 120 can be configured to spatially capture and encode the audio environment and furthermore can be configured to render a spatial audio signal to the space. In this example only the communications path from the Room A 100 to the Room B 102 is shown for simplicity but a duplex or multipoint communication system comprising multiple signalling paths can be implemented using the methods as described herein without significant inventive input.

[0115] The teleconference apparatus (for each site or room) 110, 120 is further configured to communicate with each other to implement a teleconference function.

[0116] In some embodiments the user equipment 110 comprises a microphone array 115.

[0117] Furthermore Fig.1 shows a capture (front end) processor 117 configured to receive the audio signals 118 from the microphone array 115 and perform processing on these audio signals 118 to generate a suitable input format audio signal 114, for example, an object format audio signal, MASA spatial audio format or an Ambisonics (SBA) spatial audio format, for the (IVAS) encoder 101. In some embodiments the capture (front end) processor 117 is configured by a controller 119 via control 116.

[0118] Additionally, as shown in Fig.1 the UE 110 can comprise a controller 119 configured to control the capture (front end) processor 117 and in some embodiments the (IVAS) encoder 101. In some further embodiments the controller 119 can furthermore control the operation of the microphone array 115, for example, where at least some of the capture processor 117 functionality is incorporated or integrated into the microphone array (for example, selection or activation of microphones to provide audio signals, detection of blocked microphones or microphone ports, or configuration of integrated analogue-to-digital conversion and / or frequency domain conversion).

[0119] As shown in Fig. 1, the apparatus, UEs 110, 120 and server 111 can comprise suitable encoder and decoder functionality. For example, the apparatus 110 is shown comprising the (IVAS) encoder 101, the server 111 is shown comprising an (IVAS) decoder and encoder 121, and the apparatus 120 is shown comprising an (IVAS) decoder 131. In such a manner the audio signals representing the user or talker 1 103 can be encoded by the encoder 101 which generates a bitstream 106 to be passed to the server 111. The server 111 can then decode, (optionally then mix, e.g., with other objects and otherwise process the audio signals) and encode them to generate the bitstream 108 to be passed to the apparatus 120. The apparatus 120 can then decode the audio signals and present them to the user or listener RX’ 141.

[0120] Although this example shows a teleconference application the encoder / decoder functionality can be applied to the (real-time) transmission or streaming of any suitable media. Thus, for example, the microphone array 115 is supplemented by at least one close microphone (for example, lavalier microphone or similar located on a talker in the audio scene) or replaced by one or more other microphone types.

[0121] The IVAS decoder / renderer for each of the UE 120, such as the teleconference apparatus 102 can be furthermore configured to handle multiple input streams that may each originate from a different encoder.

[0122] Fig. 1 furthermore shows the room B apparatus or UE2 comprising a suitable (IVAS) renderer 135 configured to receive the decoded spatial audio signals 134 output from the (IVAS) decoder 131 and generate or render a suitable output audio signal, for example, a binaural audio signal output, to be provided to suitable output apparatus, which in Fig.1 is shown as headphones (or more generally a headset comprising suitable audio transducers) 143 worn by listener 141. In some embodiments the listener position and orientation can be tracked, for example, by suitable headset sensors and this information passed back to the (IVAS) renderer 135 (or UE2 120 more generally) which uses this information in generating or rendering the output audio signals. Although this example shows the listener using headphones, any suitable output or output device can be employed in other embodiments, for example UE2 can be configured to generate a multichannel audio signal format (for example 7.1 multichannel audio signals) to be output to a multi-speaker arrangement in the room. A suitable output device can be, e.g., AR (augmented reality) orVR (virtual reality) headset. In another use case, the suitable output device can be a vehicle having the multi-speaker arrangement with an optional display.

[0123] Furthermore, the UE 120 can comprise a controller 133 configured to control the operations of the (IVAS) decoder 131 and (IVAS) renderer 135. In some embodiments the controller 133 in the renderer apparatus 120 is configured to communicate with or negotiate with a controller 119 within the capture apparatus 110, in this example UE1. In some embodiments the controllers 133 and 119 can communicate directly with each other (or as shown in Fig. 1 via a server 111 or central controller) as shown by bitstreams 124. In some embodiments this communication can be incorporated within the bitstreams 106 and 108.

[0124] In some embodiments the communication or negotiation between UEs can be implemented by use of RTP (Real-time transport protocol). RTP is designed to carry a multitude of multimedia formats, which permit the transport of new formats without revising the RTP standard. To this end, the information required by a specific application of the protocol is not included in the generic RTP header. For a class of applications (e.g., audio, video), an RTP profile may be defined. For a media format (e.g., a specific video coding format), an associated RTP payload format may be defined. Every instantiation of RTP in a particular application may require a profile and payload format specifications.

[0125] The profile defines the codecs used to encode the payload data and their mapping to payload format codes in the protocol field Payload Type (PT) of the RTP header.

[0126] For example, the RTP profile for audio and video conferences with minimal control is defined in RFC 3551. The profile defines a set of static payload type assignments, and a dynamic mechanism for mapping between a payload format, and a PT value using Session Description Protocol (SDP). The latter mechanism is used for newer video codecs such as RTP payload format for H.264 Video defined in RFC 6184 or RTP Payload Format for High Efficiency Video Coding (HEVC) defined in RFC 7798.

[0127] IVAS RTP payload format, originally provided as part of the IVAS Release 18 specifications, is currently being enhanced with the latest state described in TS 26.253 Annex A.

[0128] For a class of applications (e.g., audio, video, control), an RTP profile may be defined. For a media format (e.g., a specific video coding format), an associated RTP payload format may be defined. Every instantiation of RTP in a particular application may therefore require a profile and payload format specifications.

[0129] An RTP session can be established for each multimedia stream. Audio, control, and video streams may be implemented which use separate RTP sessions, enabling a receiver to selectively receive components of a particular stream. The RTP specification can furthermore be configured to recommend port numbers for RTP, and furthermore to recommend the use of the next odd port number for the associated RTCP session. A single port can be used for RTP and RTCP in applications that multiplex the protocols.

[0130] Each RTP stream can comprise RTP packets, and the RTP packet in turn can comprise a RTP header and payload pair.

[0131] Fig.2 shows an example IVAS RTP packet structure (following TS 26.253 Annex A) 201. The RTP Header (and possible RTP Header Extension) 203 follows a known RTP design. The structure 201 further comprises a IVAS payload 211. The IVAS payload 211 comprises a payload header 205 section, a (IVAS) frame data 207 section and an optional PI (processing information) data 209 section.

[0132] The payload header 205 section can comprise different types of header bytes, for example: ToC (Table of Content) and E-bytes (Extra bytes, or further bytes which can be used to define or signal aspects of the payload).

[0133] The ToC bytes can be employed to describe the content of the frame data section (by indicating the size / bitrate for the data frames). The ToC bytes can also differentiate IVAS frames from EVS frames, in situations where the IVAS is operating in mono EVS mode.

[0134] The E-bytes can signal additional information, such as Codec Mode Requests (CMR) which indicate a request to change or signal at least one codec configuration parameter or rendering configuration parameter. The E-bytes may also be used to explicitly indicate the presence of the PI data section at the end of the payload.

[0135] The frame data 207 section can include the IVAS data frames (and EVS data frames in a case of employing IVAS in mono mode). The data frames represent the encoded IVAS bitstreams. The bitstream includes the encoded IVAS audio data with possible additional metadata. The bitstream can also include initialization data or information, for example information about the input / coded format and sub-format of the encoded data (e.g., multichannel 5.1 format). Any EVS frames do not include such format data.

[0136] The PI data section can include processing information data and related headers. PI data can be used to transmit any (non-audio) data that can be used to assist the rendering or processing of the audio data, such as, for example, scene and device orientation data. Furthermore, in future updates to the IVAS RTP payload description, the PI data may also be used to request something from the other session participant, such as, for example, to mute an incoming stream or increase noise suppression. Feedback data can also be transmitted, for example, the head orientation of a listener.

[0137] The IVAS RTP specification (TS 26.253 Annex A) currently also provides a method for providing device orientation, scene orientation, and acoustic environment information as part of the PI (Processing Information) frames.

[0138] An object editing interface has furthermore been incorporated to the IVAS codec. The object editing interface enables modifying the audio rendering at the media receiver (i.e., in the IVAS decoder). The enabled features include changing the direction and the gain of each object, for the ISM (objects), OMASA (objects and MASA), and OSBA (objects and SBA) formats.

[0139] However, object editing in the decoder is not supported with the OMASA and OSBA formats at the lowest bitrates, since the object audio signals have been mixed together with the other signals, without any information transmitted about the contributions of the different objects. Thus, it is not possible to edit the objects in any way in these cases (i.e., OMASA and OSBA at lowest bitrates).

[0140] Moreover, at mid bitrates with OMASA format and low bitrates with ISM format, information about the contributions of the different objects is transmitted (using ISM ratios or power ratios), but the audio signals are still mixed together to a stereo downmix. As a result, the objects can be edited at the decoder (e.g., using the methods presented in UK patent applications GB2104309.6 and GB2404398.6), but the quality is compromised, as the processing is implemented parametrically to the common stereo downmix (i.e., the different objects are not individually available). The parametric object editing does not produce as good a quality output as would be expected by editing individual object signals. For example, “musical noise” artefacts may be produced, and the editing may not affect perception as much or in the manner as desired (e.g., the object may not move as much as desired, but instead, only slightly move from the original non-edited position). In addition, the output rendering including object editing is computationally more expensive, e.g., in OMASA, than the output rendering without the object editing.

[0141] Thus, the current object editing methods which are performed at the media receiver are problematic in that they do not allow editing for all bitrates (only the mid and higher bitrates) and furthermore provide a sub-optimal quality output for editing at the mid bitrates.

[0142] As discussed in the concept above there can be implemented a configurable rendering of a spatial audio stream or spatial audio signal (and which in some embodiments comprises at least one audio object), in an immersive audio transmission system containing at least a media sender and a media receiver (e.g., two user equipment devices such as shown in Fig.1 where UE1 110 is the media sender and UE2 120 is the media receiver.

[0143] In summary the embodiments enable the media receiver UE2 120 to perform object editing using the original audio signals before the encoding (in other words enabling editing at the media sender UE1 110) in order to aim to improve the quality of the output after editing (or actually enable object editing for low bit rate systems).

[0144] These embodiments, as discussed herein, can achieve this by determining object editing metadata (or more generally spatial audio editing information) based on a number of objects and / or based on an object editing input (or the spatial audio editing input), the metadata being generated in the media receiver UE2 120.

[0145] The determined spatial audio editing information or object editing metadata can then be transmitted to the media sender UE1 110, for example, using a predetermined RTP signaling.

[0146] Then an obtained spatial audio signal or stream in the media sender UE1 110 can be edited based on the received determined information or object editing metadata.

[0147] The edited spatial audio stream can then be encoded and transmitted to the media receiver UE2 120. Wherein the media receiver UE2 120 can be configured to decode and render a suitable spatial audio (e.g., binaural audio) using the received encoded spatial audio stream.

[0148] As discussed later the spatial audio editing information or object editing feedback is delivered from the media receiver to the media sender via reverse direction PI signaling, when the session is using bidirectional signaling or can be delivered from the media receiver to the media sender over a control channel as an RTCP feedback message.

[0149] In some embodiments this information or metadata is delivered or passed from the media receiver to the media sender over a data channel (IMS or non-IMS).

[0150] As described earlier with respect to Fig. 1, the immersive audio system can be a bidirectional conversational session.

[0151] In some embodiments, the transport protocol may be a QUIC protocol or any other suitable transport protocol.

[0152] In some embodiment the, UE2 120 is configured to compare edits requested by the user with threshold values and only transmit the spatial audio editing information if the edit request exceeds the threshold value.

[0153] Two example object editing scenarios are presented below with respect to Fig.3 and Fig.4. These examples focus on the ISM format (i.e., the audio objects), but similar processes can be followed for other audio formats such as OMASA and the OSBA.

[0154] Fig.3, for example, presents an example scenario where the object editing is performed at the receiver side 310 during the decoding and rendering process which is the current approach applied by the IVAS codec.

[0155] In the scenario, UE1 110 (operating as the sender / encoder entity 300) receives or otherwise obtains an audio input 301 and is configured to generate a spatial audio signal (for example IVAS RTP stream with ISM audio objects) and transmits this to UE2 120 as shown by 311.

[0156] The UE2 120 also receives an object editing input (or more generally in this and other examples described herein a spatial audio editing input), for example, from the user of UE2 120. Then, using the received IVAS RTP stream (comprising the spatial audio signal) and the object editing input, the UE2 120 renders an output spatial audio (e.g., binaural audio) with the objects edited according to the object editing input as shown by 313. As a result, an output spatial audio with modified audio objects is provided as output as shown by 315.

[0157] Fig.4 shows presents an example scenario where the object editing is performed at the sender / encoder 300 side according to embodiments described herein.

[0158] In a manner similar to Fig.3, UE1 110 (operating as the sender / encoder entity 300) receives or otherwise obtains an audio input 301 and is configured to generate a spatial audio signal (for example, IVAS RTP stream with ISM audio objects) and transmits this to UE2 120 as shown by 311. The UE2 120 also receives an object editing input 303 (or more generally in this and other examples described herein a spatial audio editing input), for example, from the user of UE2 120.

[0159] However, in this example UE2 120 is configured as shown by 413 to generate spatial audio editing information (based on the spatial audio editing input) or object editing metadata (e.g., modified azimuth and elevation angles) for the received audio objects.

[0160] The UE2 120 can then transmit the spatial audio editing information or object editing metadata to UE1 as shown by 415. In the example shown in Fig.4 the object editing data is transmitted as part of an IVAS RTP payload.

[0161] UE1 110 receives the spatial audio editing information or object editing metadata from UE2 120 and uses the information or metadata to modify the spatial audio signals (for example, the audio objects). For example, change the object directions in the object metadata before the encoding process as shown by 417. Then, the modified spatial audio signals (modified audio objects) are encoded, and the resulting modified ISM bitstream is transmitted to UE2 120 as shown by 419, which the UE2 120 decodes and renders into output spatial audio signals which are then output as shown by 421.

[0162] An example system for implementing the apparatus according to the scenario described in Fig.4 is shown in Fig.5.

[0163] In this example implementation the sender / encoder side 300 (for example, UE1 110) has an input which is defined audio objects 500. Each of the audio objects contain an audio signal s(t, 0 (where t is the time-domain sample index and i is the object index) and some object metadata. In this example embodiment, the object metadata comprises an azimuth angle 0(n, i) and an elevation angle $(n, i) for each object (where n is the frame index).

[0164] In addition, the input to the system on the receiver / decoder &renderer side 310 (for example, the UE2 120) is the object editing input. The object editing input 510 is an example of the spatial audio editing information and comprises, e.g., user input for how to edit the objects.

[0165] The object editing metadata determiner 513 at the receiver / decoder &renderer side 310 determines the object editing metadata 512 (an example of spatial audio editing information) based on the object editing input 510 (and optionally the number of objects 514 value received from the decoder &renderer 511).

[0166] For example, the object editing metadata 512 comprises a flag ((n, i) for each object i if object editing is desired for that object or not, and the desired object direction 0edit(n, i), $edit(n, 0 if the editing was desired for that object. The determined object editing metadata 512 is transmitted to the sender / encoder side 300.

[0167] The object modifier 501 at the sender / encoder side 300 (for example, UE1 110) receives the audio objects 500 and the object editing metadata 512 as an input. The object modifier 501 is configured to modify the audio objects 500 based on the object editing metadata 512, and, as a result, produces modified audio objects 502 (an example of modified spatial audio signals) as an output (smod(tj), 0mod(n, i), 0)-

[0168] The modified audio objects 502 are passed to an encoder 503 at the sender / encoder side 300 (for example, UE1 110). The encoder 503 can, for example, be an IVAS encoder. The encoder 503 encodes the modified audio objects 502 and generates a bitstream 504. The encoder 503 is described in further detail with respect to Fig.6. The generated bitstream 504 is transmitted to the receiver / decoder &renderer 310 side.

[0169] The decoder &renderer 311 receives the bitstream 504 as an input and renders spatial audio output (with edited objects) 516 (an example output spatial audio signal) as an output. In addition, the decoder &renderer 511 decodes from the bitstream 504 the number of objects 514 information, which can be passed to the object editing metadata determiner 513. The decoder &renderer 511 in some embodiments can, for example, be an IVAS decoder. The spatial audio output (with edited objects) 516 can, e.g., be binaural audio signals.

[0170] In these output signals, the objects have been positioned based on the object editing input 510, e.g., based on the user control.

[0171] In some embodiments the input to the object modifier 501 is the spatial audio signal or audio objects 500 (audio signals and directions 0(n, 0, 0(n, 0) and object editing metadata 512 (editing enabled flag ((n, 0 and edited directions ^editC^'O, 0edit(A 0)-

[0172] In this example, only the object directions are edited. In some other embodiments, also other features can be edited, such as object levels.

[0173] As only the object directions are edited in this embodiment, the audio signals can be passed to the output unmodified Smod(t>0 = s(t,l)

[0174] Then, the object directions are edited for the objects for which the editing is enabled, and the rest of the object directions are passed unmodified

[0175] if ((n, 0 == true-.

[0176] 0mod fn, i) = 0edit fn, i)

[0177] tpedit^V-r 0

[0178] else:

[0179] 0mod(n,i) = 0(n,i)

[0180] (fmod (n, Q = (p (n, 0

[0181]

[0182] The modified audio objects 502 smod(t, 0, 0mod(n, if (pmod(n, 0 are then output from the block.

[0183] In some alternative embodiments, object levels could be edited or modified instead of or in addition to the direction editing. In this case, the audio signals are processed using the desired gains, e.g., by Smod^O = g(n,i)s(t,i)

[0184] where g(n, 0 is the desired gain for the object i for frame n. In some embodiments, the gains can be interpolated between frames to avoid artefacts due to the abrupt changes in the gains.

[0185] The encoder 503 can be implemented using any suitable means. For example, the encoder can be an IVAS encoder and can be implemented as described in 3GPP TS 26.253. The IVAS encoder is described herein as an example, and it is briefly described below, when operating in the parametric ISM mode. For more detailed description, please see 3GPP TS 26.253.

[0186] Fig.6 shows an example encoder 503 in further detail according to some embodiments. The input audio objects 502 are forwarded to parametric metadata analyzer 601. The parametric metadata analyzer 601 is configured to determine some suitable parametric metadata 602 based on the object audio signals smod(t, i) and the directions 6mod(n, i), As an example, the parametric metadata 602 comprises object directions, power ratios, and object indices.

[0187] The parametric metadata 602 can then be forwarded to the parametric metadata encoder 605, which applies suitable encoding to the parametric metadata 602. As a result, encoded parametric metadata 606 is produced (containing, e.g., encoded object directions, encoded power ratios, and encoded object indices).

[0188] The audio objects 502 furthermore can be forwarded to the transport / downmix generator 603, which determines a stereo downmix sdm(t,j) (where j is the downmix channel index) or suitable transport audio signals 604 from the object audio signals, based on the object directions 0mod(n,Q, (pmod(n,i)- For example, downmix gains 9dm(.nJ>0 can be determined based on the cardioid directional patterns pointing to the ±90 degrees, or alternatively amplitude panning gains could be used. The stereo downmix signals are determined by applying the determined gains to the object audio signals and summing the gained signals together Sdm&> 9dm(h,j, 0 ^mod^' 0 i

[0189] The determined transport / downmix signals 604 are forwarded to the audio encoder 607, which applies suitable encoding to the audio signals. For example, the IVAS core coder can be used, or alternatively some other coder, such as EVS or AAC. The resulting encoded downmix signals 608 are output from the audio encoder 607.

[0190] The multiplexer or MUX block 609 receives the encoded parametric metadata 606 and the encoded downmix signals 608 as an input, and multiplexes them into a bitstream 504, which is output from the encoder 503.

[0191] With respect to Fig.7 is shown a flow diagram of an example operation of the encoder as shown in Fig.6.

[0192] Thus as shown in Fig.7 by 701 is the operation of the obtaining the audio objects.

[0193] Then as shown in Fig.7 by 703 is the operation of analysing the obtained audio objects to generate parametric metadata.

[0194] Furthermore as shown in Fig.7 by 707 is the operation of encoding parametric metadata.

[0195] In parallel as shown in Fig.7 by 705 is the operation of generating transport / downmix audio signals from the audio objects.

[0196] Then is shown in Fig.7 by 709 the operation of encoding transport / downmix audio signals.

[0197] Furthermore is the operation as shown in Fig.7 by 711 of multiplexing encoded parametric metadata and encoded transport / downmix audio signals to generate bitstream.

[0198] Finally is the operation of outputting the bitstream as shown in Fig.7 by 713.

[0199] The decoder can be implemented using any suitable means. For example, it can be an IVAS decoder, and it can be implemented as described in 3GPP TS 26.253. The IVAS decoder is used as an example, and it is briefly described below, when operating in the Parametric ISM mode, similarly as was described above for the encoder. For more detailed description, please see 3GPP TS 26.253.

[0200] The decoder and renderer 511 as shown in Fig.5 is shown in further detail in Fig.8. The bitstream 504 can be forwarded to a demultiplexer or “DEMUX” block 801, which demultiplexes the bitstream 504 into encoded parametric metadata 802 and encoded downmix / transport audio signals 804.

[0201] The encoded parametric metadata 802 is forwarded to a parametric metadata decoder 803, which decodes the encoded parametric metadata 802 into decoded parametric metadata 806. In addition, the parametric metadata decoder 803 can be configured to determine the number of objects 514 value, which is output from the decoder 511 to the object editing metadata determiner.

[0202] The encoded downmix signals 804 are forwarded to audio decoder 805, which decodes the encoded downmix / transport audio signals 804 into decoded downmix / transport audio signals 808.

[0203] The decoded parametric metadata 806 and the decoded downmix / transport audio signals 808 can then be forwarded to a renderer 807, which renders the spatial audio output (with edited objects) 516 in a suitable format (e.g., binaural audio signals). The rendering can, e.g., be performed using the methods described in PCT application WO2019086757A1. The objects are positioned based on the object editing input in the spatial audio signals.

[0204] With respect to Fig.9 is shown a flow diagram of an example operation of the decoder as shown in Fig.8.

[0205] Thus as shown in Fig.9 by 901 is the operation of the obtaining or receiving the bitstreams.

[0206] Then as shown in Fig.9 by 903 is the operation of demultiplexing the bitstream into encoded parametric metadata and transport / downmix audio signals.

[0207] Furthermore as shown in Fig.9 by 905 is the operation of decoding parametric metadata to generate decoded parametric metadata and optionally determining the number of objects.

[0208] Then as shown in Fig.9 by 907 is the optional operation of outputting the number of objects.

[0209] Then is shown in Fig.9 by 909 is the operation of decoding the encoded downmix / transport audio signals to generate decoded downmix / transport audio signals encoding transport / downmix audio signals.

[0210] Furthermore as shown in Fig.9 by 911 is the operation of rendering spatial output with edited objects from decoded parametric metadata and decoded downmix / transport audio signals.

[0211] Then is the operation as shown in Fig.9 by 913 is the operation of outputting the spatial audio output with edited objects.

[0212] In some embodiments the object editing metadata determiner 513 is configured to receive from the decoder the information regarding the number of ISM instances in the decoded IVAS frame (in other words the number of objects). Unless otherwise indicated, the object editing metadata determiner 513 assumes the ISM instances to retain the same order. By default, the ISM instances are rendered by the media receiver device according to the ISM extended metadata in the bitstream or based on the ISM specific metadata in the associated PI frame transmitted in the RTP payload. In an implementation embodiment, the media receiver playback interface can indicate the metadata for one or more of the ISM instances, with the possibility to interactively modify each of the ISM positions. The rendering modifications being indicated by the end user at the playback device interface are captured by this module. In scenarios where the modifications are performed locally, these modifications modify the rendering parameters for the renderer. However, for scenarios such as very low bitrates where local modifications may not be possible due to the encoding approach or may result in artefacts which adversely affect the subjective quality. Consequently, in such scenarios, modification request is generated by this module and delivered to the media sender.

[0213] The metadata representing the rendering modification requests can be delivered via extending the Processing Information (PI) frames in the reverse direction (i.e. from the media receiver to the media sender). In other embodiments, this request can be delivered as RTCP feedback message. In other implementations, this modification request message can be delivered via the data channel. The format of the message can be JSON, binary, XML or any other suitable representation format. Furthermore, HTTP based requests can also be a method to request changes in the rendering parameters for the received ISM instances.

[0214] As shown in Fig.5, the bitstream 504 can be transmitted in an RTP payload, e.g., in an IVAS RTP payload in case of IVAS is used. The IVAS RTP payload structure is presented in Fig.2. The bitstream 504, in other words, the encoded IVAS frames would be carried in the frame data section 207 of the IVAS RTP payload 211. The ISM extended metadata can be carried as part of the IVAS frames 207 or the metadata can be carried in the PI data section 209. On a system level, the sender / encoder 300 could include an RTP packetizer block, which would take in the encoded IVAS bitstream and packetize it into IVAS RTP payload. Additionally, the RTP packetizer could take in the ISM extended metadata and include the metadata in the PI data section of the IVAS RTP payload.

[0215] The decoder receives the IVAS frame from the media after it has been depacketized by the RTP payload depacketizer at the media receiver. The IVAS frame may optionally carry the ISM extended metadata which is used for rendering the ISM instances in the bitstream. There are three possible scenarios:

[0216] Firstly, ISM extended metadata is carried in the IVAS frame. In this case, the extended metadata values are used to render the ISM instances;

[0217] Secondly ISM extended metadata is carried only in the (forward direction) PI data which accompanies the IVAS frame in the RTP payload. In this case, the extended metadata values from the PI data are used to render the ISM instances;

[0218] Thirdly ISM extended metadata is carried in both the (forward direction) PI data in the RTP payload and also in the ISM extended metadata in the IVAS frame. In this case, based on the usage policy of the media receiver, either the extended metadata from the IVAS frame or the PI data is used for rendering the ISM instances.

[0219] Furthermore, the extended metadata values used for rendering can also be used to visualize for enabling interactive modifications by the end user.

[0220] In the object editing metadata in IVAS RTP payload section methods are described how to transmit the object editing metadata in the reverse direction PI data.

[0221] In an example scenario, UE2 120 receives an OMASA stream with 4 objects. In the OMASA stream, all 4 objects are positioned at the most right (-90 degrees in azimuth). To improve intelligibility / separation between different objects, UE2 120 wants to edit certain object’s position such, that its direction is at most left (90 degrees in azimuth). Furthermore, as the UE2 120 renders the OMASA stream to stereo output, the output signal should comprise one channel where is only the edited object signal, and one channel where the other objects are maintained.

[0222] Editing of the target object in aforesaid example scenario is illustrated in the following Fig. 10a, 10b and 10c. In this example a spectrogram of the original target object is visualized in Fig. 10a. With ideal editing functionality, the one output channel should contain only signal which resembles accurately this reference signal. For visualization, a short segment of a sentence is plotted. This segment comprises a couple of words. From the spectrogram different phonemes and speaker’s intonation can be visualized by the changes of the frequencies over time. As this reference signal comprises only clean speech, the shape of spectral content is clearly visible.

[0223] Fig. 10b shows a spectrogram of an edited object according to the embodiments as discussed herein (i.e., the editing is performed at the encoder). The spectrogram of the left channel of the decoded OMASA stream rendered to the stereo output is presented. Due to the original object positions and edited target position, the visualized channel should contain only signal corresponding to the target object. Thus, the spectral content over time is similar to that shown in Fig. 10a.

[0224] Fig. 10c shows a spectrogram of an edited object according to the current state of the art (i.e., the editing is performed at the decoder). The spectrogram of the left channel of the decoded OMASA stream rendered to stereo output, when the target object is edited at the decoder, is visualized. Fig. 10c shows that while the spectrogram somewhat resembles the original object spectrogram, significant amount of an additional noise is propagated to the visualized output channel.

[0225] In the presented example, the audio stream is encoded at such bitrate that the target object is coded as a parametric representation. Due to this, when performing the editing at the decoder with the state-of-the-art editing functionality, other objects in the parametric representation are leaked to the wrong output channel along the edited, target object. This leakage of unintended objects’ spectral content can be seen as a noise in Fig. 10c, where the editing is performed at the decoder. While the presented spectrogram in Fig. 10b is not exactly the same as the original spectrogram in Fig. 10a (due to the coding of the audio signals), the spectral similarity between these two signals (original object signal, object signal edited at the encoder) is significantly higher, compared to the spectral similarity between the signals illustrated in Fig. 10c and Fig. 10a.

[0226] Furthermore, the signal where the editing is performed according to these embodiments (as shown in Fig. 10b) sounds clear and doesn’t comprise artefacts. Target object is clearly distinguishable from the other objects. In addition, the target object signal is accurately positioned at the intended direction, and the directionality is well maintained.

[0227] However, when listening at the signal for which the editing is performed at the decoder (illustrated in Fig. 10c), it sounds like the target object is mixed with the other objects. Object signal is not clearly distinguishable from other objects, and the actual object direction after the editing doesn’t resemble accurately the intended target direction. Furthermore, the object is not as directional, i.e., object direction is wide and not clear for the listener, as it should be.

[0228] In some embodiments, as discussed earlier, the object editing metadata can be transmitted from a media receiver UE2 to a media sender UE1 utilizing the reverse direction PI data signaling in IVAS RTP payload.

[0229] Figs. 11 to 14 present some of the ISM related forward direction PI data frames. These PI data frames are transmitted in the forward direction (i.e., from the media sender to the media receiver) and the metadata can be used in the rendering of the associated ISM objects.

[0230] Fig. 11 shows an ISM_GAIN PI data frame 1100 which can be used to transmit gain data 1101 as 7-bit values. The possible gain values vary in the range of 0... -96, -Inf, +1, +2 and +3 in decibels. -Inf gain value can be interpreted as muting the ISM object.

[0231] Fig. 12 presents a general orientation PI frame structure 1200 for transmitting quaternion data with 2 bytes reserved for each component. ISM_ORIENTATION PI type 1200 follows similar structure and can be used to transmit orientation information for ISMs in terms of W 1201, X 1203, Y 1205, Z 1207.

[0232] Fig. 13 presents a general position PI frame structure for transmitting position data as Cartesian coordinates (x 1301, y 1303, z 1305) with 2 bytes reserved for each component. ISM_POSITION PI type 1300 follows similar structure and can be used to transmit position information for ISMs.

[0233] Fig. 14 presents ISM_DISTANCE_ATTENUATION PI data frame 1400 which can be used to transmit distance attenuation metadata for ISMs as reference distance (6 bits) 1401, maximum distance (6 bits) 1403 and a roll-off factor (6 bits) 1405.

[0234] Fig. 15 presents ISM_D!RECTIVITY PI data frame 1500 which can be used to transmit directivity metadata for ISMs as inner angle (5 bits) 1501, outer angle (5 bits) 1503 and outer attenuation (5 bits) 1505.

[0235] In some embodiments, the object editing metadata can be transmitted via reverse direction PI signaling from the media receiver to the media sender. Similar ISM metadata PI data structures as presented above and in S4-250203 can be used for the reverse direction signaling. The indication between forward and reverse direction signaling (i.e., requests) can be indicated, for example, via the PI types. For example, ISM_POSITION PI type could indicate position data in forward direction and ISM_POSITION_REVERSE or ISM_POSITION_REQUEST or ISM_POSITION_EDIT or similar PI type could indicate that the transmitted position data should be interpreted as a reverse direction signaling (i.e., as an object editing request). The addition of _REVERSE or -REQUEST or _EDIT or any similar postfix could be applied to all ISM PI types that are relevant for object editing.

[0236] In another embodiment, the ISM specific PI header could be modified to indicate if the associated PI type should be interpreted as forward or reverse direction signalling. For example, a 1 -bit flag could be added to the header to indicate if the PI data is forward direction (1) or reverse direction (0). In this case the pre-existing proposed ISM PI types (such as ISM-POSITION) could also be used for reverse direction signalling and transmitting the object editing metadata.

[0237] In another embodiment, a separate PI type of OBJECT-EDITING or OBJECT-EDIT or ISM-EDIT or similar could be introduced to transmit the object editing metadata in the reverse direction. The PI type could include all the object editing metadata for the ISMs. Figs. 16a to 16c presents example data structures for the OBJECT_EDITING P! type for two ISMs. The preceding object editing metadata indicator flags 1601 indicate if a certain metadata is present (1) or not (0) in the metadata section. For example, the following flags could be indicated: Reposition data, G=gain data, O=orientation data, D=directivity data and A=distance attenuation data. In some other embodiments, more metadata flags could be introduced. If a flag is present and set to 1, the associated metadata is present in the PI frame. For example, if PGODA combination equals to 11000, the PI frame would contain position and gain metadata for editing.

[0238] In option 1 1601, as shown in Fig. 16a, the full metadata 1603, 1605 for each ISM is included in the packet before including the metadata for the next ISM. For example, the position and gain values for ISM1 would be included in the PI data frame first, and then the position and gain values for ISM2, if PGODA=11000 and the number of ISMs to be edited is two.

[0239] In option 2 1610, as shown in Fig. 16b, the metadata values are grouped together based on metadata types. For example, if PGODA=11110 and there are two ISMs to be edited, the PI frame first contains the position metadata values for ISMs 1 and 2, then the gain values, then the orientation value and then the directivity values.

[0240] In option 3 1620, as shown in Fig. 16c, the object editing metadata for each ISM are separated with individual indicator flags. Additionally, an identifier (ID) is added after the metadata flags to identify the object. This approach allows sending different metadata for each ISM, e.g., only position edit for ISM1 and only gain edit for ISM2. In some embodiments, the indicator flags (and ID) and the metadata for each ISM could be included in the same PI frame. In some other embodiments, the indicator flags (and ID) and the metadata for each ISM could be in separate PI frames.

[0241] In some embodiments the identifier flags presented in option 3. could also be used in approaches such as options 1. and 2. to identify explicitly the objects to be edited.

[0242] In embodiments, the object editing metadata could include gain editing data and could be transmitted in a PI data. The gain edit values could follow the gain values presented in S4-250203 but could also include different values. A request for -Inf gain edit (or similar) can be interpreted as a request to mute the audio object.

[0243] In embodiments, the object editing metadata could include orientation editing data and could be transmitted in a PI data, i.e., as quaternion values. The orientation values could also be transmitted as yaw, pitch and roll values, for example, with 2 bytes reserved for each component. The edit metadata could include all the orientations or just some components (e.g., only yaw) or only some combinations of the components (e.g., yaw and pitch only).

[0244] In embodiments, the object editing metadata could include positional data in Cartesian coordinates (x, y, z) and could be transmitted in a PI data. The positional edit data could also be transmitted as azimuth and elevation values with 2 bytes reserved for each value, for example.

[0245] In embodiments, the object editing metadata could include distance attenuation metadata for objects and could be transmitted in a PI data.

[0246] In embodiments, the object editing metadata could include directivity metadata for objects and could be transmitted in a PI data.

[0247] For the editing metadata values, it is beneficial for the media receiver to transmit absolute values instead of differential values. This is because the media sender might have updated the values before the editing information reaches the sender. For example, at time instance T=0 the media sender UE A sends ISM bitstream with ISM1 at position (azi1, elevl). At T=1, the media receiver UE B receives the bitstream, creates object editing data which edits the ISM1 position to (azi1+5, elevl) and transmits the editing data to UE A. At T=2, UE A receives the object editing data from UE B. At T=1 UE A has also updated the position of ISM1 to (azi1-5, elevl). If the object editing data from UE B would be a differential value, i.e., “+5 to azimuth”, the edited value would result in azi1-5+5=azi1. This would result in erroneous edit, as UE B wanted to edit the object to azi1+5. With absolute editing values transmitted, this erroneous behavior can be avoided.

[0248] In some other embodiments, transmitting differential edit values can be implemented. This could be beneficial in cases where the edit values are transmitted in dynamically adjusted transport payload sizes. For example, sending an edit value of 90 degrees azimuth may require more bits to be transmitted than sending a differential edit value of +5 degrees azimuth if the application dynamically adjusts how many bits are used for the transmitted edit values. E.g., instead of reserving 2 bytes for an azimuth edit value, the application could reserve the minimum number of bits required (for example, 4 bits for a +5 degree differential value).

[0249] In some cases, the media receiver UE B may want to swap or switch audio objects between each other. In an example scenario, an audio scene includes 4 ISM objects (ISM1, ISM2, ISM3, ISM4) and the user of UE B wants to swap the objects ISM1 and ISM2 between each other. Metadata-wise the swapping can be understood as changing positions of the objects. In some cases, the other metadata values (such as gain and directivity) can be left unchanged when performing the swapping. In some other cases, also other or all of the metadata values may be adjusted.

[0250] The swapping of the metadata between ISM1 and ISM2 can be performed either in the media receiver UE2 side or in the media sender UE1 side. For example, the UE2 may interpret the swapping edit information before sending the object edit information to UE1. That is, UE2 sends metadata edits to UE1 for both ISM1 and ISM2 where the metadata edit requests indicate the original metadata values of ISM2 for ISM1 and vice versa. The metadata edit process in UE1 remains the same as compared to normal metadata edit requests.

[0251] In another approach, UE2 can send an explicit request to UE1 which indicates that the metadata values for ISM1 and ISM2 should be swapped. The request can be included to the above-described OBJECT_EDITING PI type or a new PI type of SWAPJSM or SWITCH_ISM or similar could be used for the request. When receiving the swapping request from UE2, the UE1 swaps the indicated metadata values for the indicated ISMs (e.g., ISM1 and ISM2).

[0252] Fig. 17 presents an example SWAP_ISM PI data frame structure 1700. The PGODA flags 1601 (as described in the above sections) indicate the metadata values to be swapped. The ID fields 1711 1713 indicate the ISMs that are affected by the swap. The S-flag 1701 indicates if the following data should be interpreted as swapping data. The S-flag 1701 could be included in the OBJECT_EDITING PI type, so that the same PI type could be used for both normal object editing requests and swapping requests. If the swapping is requested with an explicit SWAP_ISM PI data type, the S-flag can be removed as the PI type already indicates a swapping request.

[0253] In another embodiment, the SWAPJSM PI data type could only indicate the IDs of the swapped ISMs. In this case UE A could swap all the metadata between the indicated ISMs. In another approach, the switched metadata values for the ISMs could be some pre-determined set (e.g., always position only). The pre-determined set of swapping metadata could always be the same, or the set could be negotiated for each session separately, e.g., through SDP negotiation.

[0254] In some cases, it may be beneficial to apply the same object edit to multiple objects. For example, the media receiver UE2 may want to edit the gain commonly for all objects. In this case the metadata edit requests could be grouped together to indicate a common edit. For example, a 1 -bit GR-flag (or similar) could be added to the object editing indicator flags in the OBJECT_EDITING PI data. A value of GR=1 would indicate that the requested metadata edits would be applied to all ISMs, a value of GR=0 would indicate a normal object editing request.

[0255] In another embodiment, a separate OBJECT_GROUP_EDITING or similar P! type could be used for common metadata edit requests for all the objects. In this case the PI data structure could follow the OBJECT_EDITING structure, but the PI data frame would only include single metadata values for each metadata type and the values would be applicable to all ISMs.

[0256] In another embodiment, in addition to adding a GR-flag to the indicator flags, the indication could include ID fields that identify the commonly edited objects. This would allow editing, e.g., objects ISM1 and ISM2 together but leave ISM3 and ISM4 out of the edits if those ISMs would not be indicated in the ID fields.

[0257] The benefit of grouping metadata edits together is that only a single metadata edit value (e.g., a single gain value) is needed to be transmitted. In comparison, without the grouping, requesting gain edits for 4 objects would require 4 gain edit requests with identical values. With the grouping, only a single gain edit request is needed.

[0258] The support for media sender side object editing can be explicitly negotiated for each session. For example, an SDP parameter of object-editing (or similar) with permissive values of 1 (supported) and 0 (not supported) could be introduced.

[0259] Additionally, the support for the different editable metadata types could be negotiated. For example, the supported metadata types for editing could be listed in the object-editing parameter (e.g., object-editing=[gain, orientation]). The listed metadata values would then be allowed to be edited during the session. An empty list could indicate the lack of support for object editing during the session.

[0260] Alternatively, the editable metadata types for the session could be negotiated with separate parameters. E.g., types like gain-edit and orientation-edit with permissive values of 1 (supported) and 0 (not supported) could be introduced to explicitly indicate which metadata types would be supported for the session.

[0261] The supported PI types could be negotiated with the pi-types parameter, which lists all the supported PI types for the session. OBJECT_EDITING and SWAP_ISM (and all other PI types presented in this invention) could be listed in the pi-types parameter to indicate support for them in the session.

[0262] Fig. 18 presents example method steps for an embodiment of the invention. First, the media sender UE A transmits IVAS payload with audio objects to the media receiver UE B as shown by 1801. The receiver UE B then obtains or generates object edit metadata as shown by 1803 and transmits the data to UE A (e.g., via IVAS RTP payload). The UE A receives as shown by 1805 the object editing metadata and edits the audio objects based on the received metadata before the encoding process as shown by 1807. Then UE A encodes as shown by 1809 the edited objects and transmits the encoded stream to UE B. After receiving the edited encoded object stream as shown by 1811, UE B decodes and renders the stream to the user. The user experiences that the objects have been edited.

[0263] With respect to Fig. 19 an example electronic device which may be used as the apparatus or any of the functional blocks described herein is shown. The device may be any suitable electronics device or apparatus. For example, in some embodiments the device 1900 is a mobile device, user equipment, tablet computer, computer, audio playback apparatus, car or other vehicle, loT device, etc.

[0264] In some embodiments the device 1900 comprises at least one processor or central processing unit 1907. The processor 1907 can be configured to execute various program codes such as the methods such as described herein.

[0265] In some embodiments the device 1900 comprises a memory 1911. In some embodiments the at least one processor 1907 is coupled to the memory 1911. The memory 1911 can be any suitable storage means. In some embodiments the memory 1911 comprises a program code section for storing program codes implementable upon the processor 1907. Furthermore, in some embodiments the memory 1911 can further comprise a stored data section for storing data, for example data that has been processed or to be processed in accordance with the embodiments as described herein. The implemented program code stored within the program code section and the data stored within the stored data section can be retrieved by the processor 1907 whenever needed via the memory-processor coupling.

[0266] In some embodiments the device 1900 comprises a user interface 1905. The user interface 1905 can be coupled in some embodiments to the processor 1907. In some embodiments the processor 1907 can control the operation of the user interface 1905 and receive inputs from the user interface 1905. In some embodiments the user interface 1905 can enable a user to input commands to the device 1900, for example, via a keypad. In some embodiments the user interface 1905 can enable the user to obtain information from the device 1900. For example, the user interface 1905 may comprise a display configured to display information from the device 1900 to the user. The user interface 1905 can in some embodiments comprise a touch screen or touch interface capable of both enabling information to be entered to the device 1900 and further displaying information to the user of the device 1900.

[0267] In some embodiments the device 1900 comprises an input / output port 1909. The input / output port 1909 in some embodiments comprises a digital to analogue converter for receiving or outputting audio signals.

[0268] Additionally in some embodiments the device 1900 input / output port 1909 comprises a transceiver. The input / output port 1909 in such embodiments can be coupled to the processor 1907.

[0269] The transceiver in some embodiments can be configured to enable a communication with other apparatus or electronic devices, for example, via a wireless communications network. The transceiver or any suitable transceiver or transmitter and / or receiver means can in some embodiments be configured to communicate with other electronic devices or apparatus via a wire or wired coupling.

[0270] The transceiver can communicate with further apparatus by any suitable known communications protocol.

[0271] It should be understood that the apparatuses may comprise or be coupled to other units or modules used in or for transmission and / or reception. Although the apparatuses have been described as one entity, different modules and memory may be implemented in one or more physical or logical entities.

[0272] It is noted that whilst some embodiments have been described in relation to 5G networks, similar principles can be applied in relation to other networks and communication systems. Therefore, although certain embodiments were described above by way of example with reference to certain example architectures for wireless networks, technologies and standards, embodiments may be applied to any other suitable forms of communication systems than those illustrated and described herein.

[0273] It is also noted herein that while the above describes example embodiments, there are several variations and modifications which may be made to the disclosed solution without departing from the scope of the present invention.

[0274] As used herein, “at least one of the following: ” and “at least one of ” and similar wording, where the list of two or more elements are joined by “and” or “or”, mean at least any one of the elements, or at least any two or more of the elements, or at least all the elements.

[0275] In general, the various embodiments may be implemented in hardware or special purpose circuitry, software, logic or any combination thereof. Some aspects of the disclosure may be implemented in hardware, while other aspects may be implemented in firmware or software which may be executed by a controller, microprocessor or other computing device, although the disclosure is not limited thereto. While various aspects of the disclosure may be illustrated and described as block diagrams, flow charts, or using some other pictorial representation, it is well understood that these blocks, apparatus, systems, techniques or methods described herein may be implemented in, as non-limiting examples, hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controller or other computing devices, or some combination thereof.

[0276] As used in this application, the term “circuitry” may refer to one or more or all of the following: (a) hardware-only circuit implementations (such as implementations in only analog and / or digital circuitry) and (b) combinations of hardware circuits and software, such as (as applicable): (c) a combination of analog and / or digital hardware circuit(s) with software / firmware and (i) any portions of hardware processor(s) with software (including digital signal processor(s)), software, and memory(ies) that work together to cause an apparatus, such as a mobile phone or server, to perform various functions); and (ii) hardware circuit(s) and or processor(s), such as a microprocessor(s) or a portion of a microprocessor(s), that requires software (e.g., firmware) for operation, but the software may not be present when it is not needed for operation.

[0277] This definition of circuitry applies to all uses of this term in this application, including in any claims. As a further example, as used in this application, the term circuitry also covers an implementation of merely a hardware circuit or processor (or multiple processors) or portion of a hardware circuit or processor and its (or their) accompanying software and / or firmware. The term circuitry also covers, for example and if applicable to the particular claim element, a baseband integrated circuit or processor integrated circuit for a mobile device or a similar integrated circuit in server, a cellular network device, or other computing or network device.

[0278] The embodiments of this disclosure may be implemented by computer software executable by a data processor of the mobile device, such as in the processor entity, or by hardware, or by a combination of software and hardware. Computer software or program, also called program product, including software routines, applets and / or macros, may be stored in any apparatus-readable data storage medium and they comprise program instructions to perform particular tasks. A computer program product may comprise one or more computer-executable components which, when the program is run, are configured to carry out embodiments. The one or more computer-executable components may be at least one software code or portions of it.

[0279] Further in this regard it should be noted that any blocks of the logic flow as in the Figures may represent program steps, or interconnected logic circuits, blocks and functions, or a combination of program steps and logic circuits, blocks and functions. The software may be stored on such physical media as memory chips, or memory blocks implemented within the processor, magnetic media such as hard disk or floppy disks, and optical media such as DVD and the data variants thereof, CD. The physical media is a non-transitory media.

[0280] The term “non-transitory,” as used herein, is a limitation of the medium itself (i.e., tangible, not a signal) as opposed to a limitation on data storage persistency (e.g., RAM vs. ROM).

[0281] The memory may be of any type suitable to the local technical environment and may be implemented using any suitable data storage technology, such as semiconductor based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory and removable memory. The data processors may be of any type suitable to the local technical environment, and may comprise one or more of general purpose computers, special purpose computers, microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASIC), FPGA, gate level circuits and processors based on multi core processor architecture, as non-limiting examples.

[0282] Embodiments of the disclosure may be practiced in various components such as integrated circuit modules. The design of integrated circuits is by and large a highly automated process. Complex and powerful software tools are available for converting a logic level design into a semiconductor circuit design ready to be etched and formed on a semiconductor substrate.

[0283] The scope of protection sought for various embodiments of the disclosure is set out by the independent claims. The embodiments and features, if any, described in 5 this specification that do not fall under the scope of the independent claims are to be interpreted as examples useful for understanding various embodiments of the disclosure.

[0284] The foregoing description has provided by way of non-limiting examples a full and informative description of the exemplary embodiment of this disclosure. However, 10 various modifications and adaptations may become apparent to those skilled in the relevant arts in view of the foregoing description, when read in conjunction with the accompanying drawings and the appended claims. However, all such and similar modifications of the teachings of this disclosure will still fall within the scope of this invention as defined in the appended claims. Indeed, there is a further embodiment 15 comprising a combination of one or more embodiments with any of the other embodiments previously discussed.

Claims

1. An apparatus for immersive audio rendering, the apparatus comprising at least one processor and at least one memory storing instructions that, when executed by the at least one processor, cause the system at least to:obtain a spatial audio editing input, the spatial audio editing input associated with modifying at least one spatial audio signal;generate spatial audio editing information based on the spatial audio editing input;transmit the spatial audio editing information to a further apparatus;obtain, from the further apparatus, a bitstream comprising an encoded modified spatial audio signal, the encoded modified spatial audio signal based on the spatial audio editing information;decode the encoded modified spatial audio signal; andrender at least one output spatial audio signal based on the decoded modified spatial audio signal.

2. The apparatus as claimed in claim 1, caused to transmit the spatial audio editing information parameter to the further apparatus is caused to transmit the spatial audio editing information in one of:an IVAS RTP payload;an IVAS RTP packet;a reverse direction PI signalling field;a control channel RTCP feedback message;an RTP Header Extension data;a data channel IMS message; anda data channel non-IMS message.

3. The apparatus as claimed in any of claims 1 or 2, wherein the encoded modified at least one spatial audio signal comprises at least one audio object, the spatial audio editing input comprises an audio object editing input, and the spatial audio editing information is at least one audio object editing information.

4. The apparatus as claimed in claim 3, wherein the object editing information parameter comprises at least one of:gain editing information for modifying a gain associated with at least one identified audio object or all objects;orientation editing information for editing an orientation associated with at least one identified audio object or all objects;yaw orientation editing information for editing a yaw orientation associated with at least one identified audio object or all objects;pitch orientation editing information for editing a pitch orientation associated with at least one identified audio object or all objects;roll orientation editing information for editing a roll orientation associated with at least one identified audio object or all objects;positional editing information for editing a position associated with at least one identified audio object or all objects;oneonex-position editing information for editing identified audio object or all objects;y-position editing information for editing identified audio object or all objects;z-position editing information for editingx-positiony-positionz-positionassociated withassociated withassociated withatatatleastleastleastoneidentified audio object or all objects;azimuth editing information for editing an azimuth position associated with atleast one identified audio object or all objects;elevation editing information for editing an elevation position associated with at least one identified audio object or all objects;distance editing information for editing a distance associated with at least one identified audio object or all objects; anddirectivity editing information for editing a directivity associated with at least one identified audio object or all objects.

5. The apparatus as claimed in any of claims 1 to 4, wherein the at least one spatial audio signal comprises at least one spatial parameter which comprises at least one of:a direction and / or orientation to the at least one object;a position of the at least one object;a distance to the at least one object;an energy of the at least one object;an energy ratio of the at least one object; and a directivity of the at least one object.

6. The apparatus as claimed in any of claims 1 to 5, wherein the at least one encoded modified spatial audio signal further comprises at least one encoded transport audio signal, and the apparatus caused to decode the encoded modified spatial audio signal is caused to decode the at least one encoded transport audio signal, wherein the apparatus caused to render at least one output spatial audio signal based on the decoded modified at least one spatial audio signal is caused to render the at least one output spatial audio signal further based on the decoded at least one transport audio signal.

7. The apparatus as claimed in claim 3 or any of claims dependent on claim 3, caused to generate spatial audio editing information based on the spatial audio editing input is caused to generate the spatial audio editing information further based on a determination of a number of audio objects within the at least one encoded modified spatial audio signal.

8. The apparatus as claimed in claim 3 or any of the claims dependent on claim 3, caused to generate spatial audio editing information based on the spatial audio editing input is caused to generate the spatial audio editing information parameter further based on a comparison between the spatial audio editing input and a threshold parameter.

9. A method for an apparatus for immersive audio rendering, the method comprising at least:obtaining a spatial audio editing input, the spatial audio editing input associated with modifying at least one spatial audio signal;generating spatial audio editing information based on the spatial audio editing input;transmitting the spatial audio editing information to a further apparatus,;obtaining, from the further apparatus, a bitstream comprising an encoded modified spatial audio signal, the encoded modified spatial audio signal based on the spatial audio editing information;decoding the encoded modified spatial audio signal; andrendering at least one output spatial audio signal based on the decoded modified spatial audio signal.

10. The method as claimed in claim 9, wherein transmitting the spatial audio editing information parameter to the further apparatus comprises transmitting the spatial audio editing information in one of:an IVAS RTP payload;an IVAS RTP packet;a reverse direction PI signalling field;a control channel RTCP feedback message;an RTP Header Extension data;a data channel IMS message; anda data channel non-IMS message.

11. The method as claimed in any of claims 9 or 10, wherein the encoded modified at least one spatial audio signal comprises at least one audio object, the spatial audio editing input comprises an audio object editing input, and the spatial audio editing information is at least one audio object editing information.

12. The method as claimed in claim 11, wherein the object editing information parameter comprises at least one of:gain editing information for modifying a gain associated with at least one identified audio object or all objects;orientation editing information for editing an orientation associated with at least one identified audio object or all objects;yaw orientation editing information for editing a yaw orientation associated with at least one identified audio object or all objects;pitch orientation editing information for editing a pitch orientation associated with at least one identified audio object or all objects;roll orientation editing information for editing a roll orientation associated with at least one identified audio object or all objects;positional editing information for editing a position associated with at least oneidentified audio object or all objects;x-position editing information for editinga x-position associated with at leastone identified audio object or all objects;y-position editing information for editinga y-position associated with at leastone identified audio object or all objects;z-position editing information for editinga z-position associated with at leastone identified audio object or all objects;azimuth editing information for editing an azimuth position associated with atleast one identified audio object or all objects;elevation editing information for editing an elevation position associated with at least one identified audio object or all objects;distance editing information for editing a distance associated with at least one identified audio object or all objects; anddirectivity editing information for editing a directivity associated with at least one identified audio object or all objects.

13. The method as claimed in any of claims 9 to 12, wherein the at least one spatial audio signal comprises at least one spatial parameter which comprises at least one of:a direction and / or orientation to the at least one object;a position of the at least one object;a distance to the at least one object;an energy of the at least one object;an energy ratio of the at least one object; and a directivity of the at least one object.

14. The method as claimed in any of claims 9 to 13, wherein the at least one encoded modified spatial audio signal further comprises at least one encoded transport audio signal, and decoding the encoded modified spatial audio signal comprises decoding the at least one encoded transport audio signal, wherein rendering at least one output spatial audio signal based on the decoded modified atleast one spatial audio signal comprises rendering the at least one output spatial audio signal further based on the decoded at least one transport audio signal.

15. The method as claimed in claim 11 or any of claims dependent on claim 11, 5 wherein generating spatial audio editing information based on the spatial audio editing input comprises generating the spatial audio editing information further based on a determination of a number of audio objects within the at least one encoded modified spatial audio signal.io 16. The method as claimed in claim 11 or any of the claims dependent on claim 11, wherein generating spatial audio editing information based on the spatial audio editing input comprises generating the spatial audio editing information parameter further based on a comparison between the spatial audio editing input and a threshold parameter.15

Citation Information

Patent Citations

  • ViewWO2020102153A1onEspacenetopensinnewtab

  • ViewGB2639905AonEspacenetopensinnewtab