Split binaural rendering

HK40137435APending Publication Date: 2026-09-11DOLBY LABORATORIES LICENSING CORP +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
HK62026125674
Authority / Receiving Office
HK · HK
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-02-28
Filing Date
2026-07-03
Publication Date
2026-09-11

Smart Images

  • Figure 00000000_0000_ABST
    Figure 00000000_0000_ABST
Patent Text Reader

Abstract

The present disclosure describes techniques for low-complexity split rendering based on a limited number of pre-renditions, thereby significantly reducing the required number of computations on the pre-rendering side, as well as the amount of transmitted metadata. The present disclosure further describes low complexity solutions for post-renderer corrections around one, two or three rotation axes, e.g., for deviations of yaw, pitch and roll.
Need to check novelty before this filing date? Find Prior Art

Description

W O 2 0 2 4 / 1 8 2 4 5 7 A 1 (12) INTERNATIONAL APPLICATION PUBLISHED UNDER THE PATENT COOPERATION TREATY (PCT) (19) World Intellectual Property Organization International Bureau (43) International Publication Date 06 September 2024 (06.09.2024) (51) International Patent Classification: H04S 7 / 00 (2006.01) WIPO PCT (10) International Publication Number WO 2024 / 182457 A1 (21) International Application Number: PCT / US2024 / 017570 (22) International Filing Date: 27 February 2024 (27.02.2024) (25) Filing Language: English (26) Publication Language: 28 February 2023 (28.02.2023) US 27 February 2024 (27.02.2024) (30) Priority Data: 63 / 448.830 63 / 558,596 (71) (72) (74) (81) English US Applicants: DOLBY LABORATORIES LICENSING CORPORATION [US / US]; 1275 Market Street, San Fran- cisco, California 94103 (US). DOLBY INTERNATION- AL AB [SE / IE]; 77 Sir John Rogerson's Quay, Block C, Grand Canal Docklands, Dublin, D02 VK60 (IE). Inventors: BRUHN, Stefan; c / o Dolby Laboratories, Inc.. 1275 Market Street, San Francisco, California 94103 (US). TYAGI, Rishabh; c / o Dolby Laboratories, Inc., 1275 Mar- ket Street, San Francisco, California 94103 (US). Agent: HERTZBERG, Brett A. et al.; c / o DOLBY LABO- RATORIES, INC., Intellectual Property Group, 1275 Mar- ket Street, San Francisco, California 94103 (US). Designated States (unless otherwise indicated, for every kind of national protection available): AE, AG, AL, AM, AO, AT, AU, AZ, BA, BB, BG, BH, BN, BR, BW, BY, BZ, CA, CH, CL, CN, CO, CR, CU, CV, CZ, DE, DJ, DK, DM, (84) DO, DZ, EC, EE, EG, ES, FI, GB, GD, GE, GH, GM, GT, HN, HR, HU, ID, IL, IN, IQ, IR, IS, IT, JM, JO, JP, KE, KG, KH, KN, KP, KR, KW, KZ, LA, LC, LK, LR, LS, LU, LY, MA, MD, MG, MK, MN, MU, MW, MX, MY, MZ, NA, NG, NI, NO, NZ, OM, PA, PE, PG, PH, PL, PT, QA, RO, RS, RU, RW, SA, SC, SD, SE, SG, SK, SL, ST, SV, SY, TH, TJ, TM, TN, TR, TT, TZ, UA, UG, US, UZ, VC, VN, WS, ZA, ZM, ZW. Designated States (unless otherwise indicated, for every kind of regional protection available): ARIPO (BW, CV, GH, GM, KE, LR, LS, MW, MZ, NA, RW, SC, SD, SL, ST, SZ, TZ, UG, ZM, ZW), Eurasian (AM, AZ, BY, KG, KZ, RU, TJ, TM), European (AL, AT, BE, BG, CH, CY, CZ, DE, DK, EE, ES, FI, FR, GB, GR, HR, HU, IE, IS, IT, LT, LU. LV. MC. ME, MK, MT, NL, NO, PL, PT. RO. RS, SE, SI, SK, SM, TR), OAPI (BF, BJ, CF, CG, CI, CM, GA, GN, GQ, GW, KM, ML, MR, NE, SN, TD, TG). Published: - with international search report (Art. 21(3)) - before the expiration of the time limit for amending the claims and to be republished in the event of receipt of amendments (Rule 48.2(h)) (54) Title: SPLIT BINAURAL RENDERING S21 OBTAIN AUDIO CONTENT S22 OBTAIN REFERENCE POSE RENDER Binrof -S23 (57) Abstract: The present disclosure describes techniques for low-complexity split rendering based on a limited number of pre-renditions, thereby significantly reducing the required number of computations on the pre-rendering side, as well as the amount of transmitted metadata. The present disclosure further describes low complexity so- lutions for post-renderer corrections around one, two or three rotation axes, e.g., for deviations of yaw, pitch and roll. PRE-RENDERING -S24 COMPUTE M&H -S25 ENCODE AND OUTPUT BITSTREAM -S26 FIG. 5 WO 2024 / 182457 SPLIT BINAURAL RENDERING PCT / US2024 / 017570 5 10 15 20 25 CROSS-REFERENCE TO RELATED APPLICATIONS

[001] This application claims priority to U.S. Provisional Application No. 63 / 448,830 filed February 28, 2023 and U.S. Provisional Application No. 63 / 558,596 filed February 27, 2024, the contents of which are herein incorporated by reference in their entirety. TECHNICAL FIELD OF THE INVENTION

[002] The present invention relates generally to audio processing, and more specifically to audio rendering (e.g., binaural rendering) performed on two separate devices ("split rendering"). BACKGROUND

[003] Unless otherwise indicated herein, the materials described in this section are not prior art to the claims in this application and are not admitted as prior art by inclusion in this section.

[004] Immersive audio is an essential media component of extended reality (XR) applications, which includes augmented reality (AR), mixed reality (MR) and virtual reality (VR). To enhance the user experience, immersive audio may support adjusting the presented immersive audio / visual scene in response to motion of the user. For example, it may be desirable to track a user's head position and head movement during audio rendering and to adjust the audio accordingly. Thus, an immersive audio experience may process head movements using models with three degrees of freedom (3DoF) or six degrees of freedom (6DoF).

[005] Various immersive audio services, e.g., immersive voice and audio services (IVAS), may be used to render high quality audio renditions at the XR device that include awareness of pose information, which may include metadata for head positions with relative or absolute movements of the user. However, making such adjustments according to pose information may require significant computational processing capabilities to achieve a high- quality immersive audio experience.

[006] The computational complexity requirements for immersive audio may be problematic for small form factor devices such as AR glasses. To make them as practical and user-friendly as possible, such AR glasses may avoid using powerful processors and heavy batteries, which may otherwise result in bulky, more expensive, and heavy weight user-worn 30 devices that consume more power and generate a significant amount of heat. Consequently, to -1- WO 2024 / 182457 PCT / US2024 / 017570 5 10 enable reasonable form factor low power operation with low latency, such AR devices tend to have processors with reduced complexity and constrained numerical operations.

[007] The present disclosure recognizes the above noted problems and explores potential solutions. One potential solution is to reduce audio rendering requirements at the end-device (e.g., the AR device operated by the user) with a split-rendering topology that leverages processing from some other entity of the mobile / wireless network (e.g., a network based device) to which the end-device is connected or tethered (e.g., via a network or cloud-based connection). For example, a powerful network entity such as mobile user equipment (e.g., UE, a device used by an end-user, a portable multi-function device, a gaming console, a cloud-based resource, etc.) may be connected to the end-device to assist in split-rendering of immersive audio. Pose information based on the user movement may be gathered at the end-device and transmitted to the network entity. The end-device may then receive the already rendered audio from the network entity; where the high complexity calculations such as processing 3DoF / 6DoF pose information (e.g., head-tracking metadata) may be performed by the rendering entity (e.g., 15 network entity). One problem with the described split-rendering topology is the latency for transmissions between end-device and network entity may be on the order of 100ms; which means the network entity may be relying on outdated pose / head-tracking information. Because of this delay, the rendered audio from the network entity may not match the current head 20 25 30 pose / head position of the user at the end-device. If the motion-to-sound latency is too large, the end user will experience a perceivable loss of quality in the immersive experience.

[008] U.S. Prov. Appl. No. US 63 / 340,181 discloses a novel approach to interactive headtracking. The described approach generates multiple binaural representations (pre- renditions) corresponding to various head poses at the main device or pre-renderer and computes metadata which can be used along with a reference binaural representation to reconstruct binaural output corresponding to any given pose at the post-renderer. The reference binaural representation and the metadata are sent to a post-rendering device. Based on the reference binaural representation and metadata, and on a difference between a reference pose and a detected current head pose of the user, the post-renderer determines binaural audio corresponding to the current head pose.

[009] U.S. Prov. Appl. No. US 63 / 386,465 describes a system relying on a binaural rendering for a reference pose P'obtained upstream from the post-renderer device, and a number of N pre-renditions for 'probing' poses P1-N which are close to reference pose P'. SUMMARY -2- WO 2024 / 182457 PCT / US2024 / 017570 5 10 15 20

[010] In some applications, complexity constraints at the pre-renderer device and metadata bit rate limitations on the transmission interface between the pre- and post-renderer devices may limit the number of pre-renditions (or binaural representations) to be computed at the pre-renderer device and also may limit the amount of metadata to be transmitted to the post renderer for pose correction. Hence, it may be desired to carefully choose the pre-rendition head poses at the pre-renderer such that pose correction to any head pose at the post-renderer can be achieved with a limited number of pre-renditions at the pre-render and a limited amount of metadata transmission.

[011] One cause of numerical complexity at the pre-renderer device is the required number of pre-renditions. A typical case contemplated for split renderer metadata enabling yaw correction may employ one reference binaural representation and two binaural pre-renditions (N = 2), where the two probing poses are P' + X and P' - X' and wherein X and X' are angular deviations from P' around a rotational axis.

[012] It is an object of the present invention to address the described issues, and to enable efficient split rendering with a reduced number of pre-renditions, and consequently a smaller amount of associated metadata.

[013] The present disclosure describes a low-complexity split rendering technique based on a limited number of pre-renditions, thereby significantly reducing the required number of computations on the pre-rendering side, as well as reducing the amount of transmitted metadata. The present disclosure further describes low complexity solutions for post-renderer corrections around one, two or three rotation axes, e.g., for deviations of yaw, pitch and roll.

[014] This and other objects are achieved by various aspects of the present invention including those defined by the independent claims.

[015] According to a first aspect, this and other objects are achieved by a method of 25 rendering audio (in a main device) to enable split rendering techniques with pose correction around multiple rotational axes (in a lightweight device), the method comprising obtaining an immersive audio content, obtaining a reference pose, rendering the immersive audio content into a first number of binaural pre-renditions, wherein the binaural pre-renditions correspond to a set 30 of probing poses, wherein the set of probing poses include poses equal to the reference pose (P') and / or poses that deviate from the reference pose by rotation around at least one of the rotational axes, calculating a second number of approximate binaural representations based on the binaural pre-renditions, wherein the approximate binaural representations correspond to a set of virtual probing poses, wherein the virtual probing poses deviate from the probing poses by rotation around at least one of the rotational axes, determining a reference binaural representation based -3- WO 2024 / 182457 PCT / US2024 / 017570 on one or more of the binaural pre-renditions and the approximate binaural representations, computing reconstruction metadata enabling reconstruction of the binaural pre-renditions and the approximate binaural representations from the reference binaural representation, encoding the reconstruction metadata and the reference binaural representation in an output bitstream, and 5 outputting the output bitstream. 10 15

[016] According to a second aspect, this and other objects are achieved by a method of rendering audio (in a main device) to enable split rendering with pose correction around yaw axis and pitch axis (in a lightweight device), the method comprising obtaining an immersive audio content, obtaining a reference pose, rendering the immersive audio content into a reference binaural representation corresponding to a reference pose, rendering the immersive audio content into one or more binaural pre-renditions, wherein the binaural pre-renditions correspond to one or more probing poses deviating from the reference pose by rotation deviate from the reference pose by rotation around both yaw and pitch axes, computing, for each probing pose, yaw metadata representing a deviation around the yaw axis, and pitch metadata representing deviation around the pitch axis, encoding the reference binaural representation, the yaw metadata and the pitch metadata in an output bitstream, and outputting the output bitstream

[017] According to a third aspect, this and other objects are achieved by a method of rendering audio in a main device to enable split rendering with pose correction around at least one rotational axis (in a lightweight processing device), the method comprising obtaining an 20 immersive audio content, receiving head pose information associated with a user of a lightweight processing device, determining, based on the head pose information, a reference pose, rendering the immersive audio content into a reference binaural representation corresponding to a reference pose, rendering the immersive audio content into one or more binaural pre-renditions, the binaural pre-renditions corresponding to one or more probing poses deviating from the reference 25 30 pose around the rotational axis, computing reconstruction metadata which enables reconstruction of the binaural pre-renditions from the reference binaural representation, the reconstruction metadata including, for each time-frequency tile, a transformation matrix, computing enhanced metadata by multiplying, each transformation matrix for a specific probing pose with an additional gain matrix (G), the additional gain matrix having the form: =91ps 01 Gps =0 gr.ps] 9ips = sqrt(psps Cyp), ( grps = sgrt(xppRy.ps.ps (2,2) where Ryps,ps represents a 2x2 covariance matrix of the binaural pre-rendition for the specific probing pose Ps, and Rps,ps represents a 2x2 covariance matrix of a reconstructed binaural pre- -4- WO 2024 / 182457 PCT / US2024 / 017570 5 10 rendition for the specific probing pose Ps, encoding the reference binaural representation and the enhanced metadata in an output bitstream, and outputting the output bitstream

[018] According to a fourth aspect, this and other objects are achieved by a method of rendering audio (in a main device) to enable split rendering with pose correction around multiple rotational axes (in a lightweight device), the method comprising obtaining an immersive audio content, receiving head pose information associated with a user of a lightweight processing device, determining, based on the head pose information, a reference pose and at least one of a head pose rotation axis and a head pose rate of rotation, rendering the immersive audio content into a reference binaural representation corresponding to the reference pose, rendering the immersive audio content into a set of binaural pre-renditions, wherein the binaural pre-renditions correspond to a set of probing poses rotated with respect to the reference pose, wherein the probing poses are selected based on the head pose information, computing reconstruction metadata to enable reconstruction of the binaural pre-renditions from the reference binaural representation, encoding the reference binaural representation and the reconstruction metadata in 15 an output bitstream, and outputting the output bitstream.

[019] According to a fifth aspect, this and other objects are achieved by a method of audio processing with pose correction around multiple rotational axes, the method comprising receiving a bitstream from a main device, decoding the bitstream to obtain a reference binaural representation and first reconstruction metadata associated with a set of probing poses 20 representing deviation from a reference pose by rotation around the multiple rotational axes, detecting a current head-pose, for each of the rotational axes, selecting a probing pose closest to the detected pose along the rotation axis, determining axis-specific reconstruction metadata based on the first metadata associated with the selected probing pose and a difference between the reference pose and the current head pose along the rotational axis, and determining a binaural 25 output corresponding to the current head pose based on the reference binaural representation and the axis-specific reconstruction metadata for each rotational axis.

[020] According to a sixth aspect, various objects are achieved by a main processing device, comprising a decoder configured to decode a first bitstream to obtain decoded immersive audio content, a renderer configured to obtain a reference pose, render the immersive audio 30 content into a reference binaural representation based on the reference pose, and render the immersive audio content into a number of binaural pre-renditions, wherein the binaural pre- renditions correspond to a set of probing poses associated with a reference pose, wherein the set of probing poses include poses that deviate from the reference pose by rotation about at least one of the rotational axes, a metadata generator configured to compute reconstruction metadata to -5- WO 2024 / 182457 PCT / US2024 / 017570 5 10 enable reconstruction of the binaural pre-renditions from the reference binaural representation, an encoder configured to encode the reference binaural representation and the reconstruction metadata into an output bitstream, and an interface configured to output the output bitstream.

[021] According to a seventh aspect, various objects are achieved by a lightweight processing device comprising a decoder configured to decode a bitstream to obtain a reference binaural representation and first reconstruction metadata associated with a set of probing poses representing deviation from a reference pose by rotation around multiple rotational axes, a head- tracker configured to detect a current head-pose, a binaural reconstruction block configured to, for each of the rotational axes, select a probing pose closest to the detected pose along the rotation axis, and determine axis-specific reconstruction metadata based on the first metadata associated with the selected probing pose and a difference between the reference pose and the current head pose along the rotational axis, and determine a binaural output corresponding to the current head pose based on the reference binaural representation, and the axis-specific reconstruction metadata. 15

[022] BRIEF DESCRIPTION OF THE DRAWINGS The present invention will be described in more detail with reference to the appended drawings.

[023] 20 FIG. 1 shows a user with smartphone and a set of headphones.

[024] FIG. 2 is a diagram illustrating a head pose and rotation around the three axis yaw, pitch and roll.

[025] FIG. 3 is a schematic block diagram showing split rendering in a main processing device and a lightweight processing device.

[026] FIG. 4 is a flow chart illustrating processing in a main processing device, in 25 accordance with embodiments of a first aspect of the invention. 30

[027] FIG. 5 is a flow chart illustrating processing in a main processing device, in accordance with embodiments of a second aspect of the invention.

[028] FIG. 6 is a flow chart illustrating processing in a lightweight processing device, in accordance with embodiments of a further aspect of the invention.

[029] FIG. 7 is a flow chart illustrating processing in a main processing device, in accordance with embodiments of a yet further aspect of the invention.

[030] FIG. 8 is a flow chart illustrating processing in a main processing device, in accordance with embodiments of a still further aspect of the invention. -6- WO 2024 / 182457 PCT / US2024 / 017570

[031] FIG. 9 illustrates a schematic block diagram of an example device or architecture that may be used to implement embodiments of the invention. 5

[032] DETAILED DESCRIPTION In the following detailed description, reference is made to the accompanied drawings, which form a part hereof, and which is shown by way of illustration, specific example configurations of which the concepts can be practiced. These configurations are described in sufficient detail to enable those skilled in the art to practice the techniques disclosed herein, and it is to be understood that other configurations can be utilized, and other changes may be made, 10 without departing from the spirit or scope of the presented concepts. The following detailed description is, therefore, not to be taken in a limiting sense, and the scope of the presented concepts is defined only by the appended claims.

[033] Embodiments of the invention disclosed herein assume compatibility and consistency with usage of an immersive audio codec such as IVAS in an XR application. In 15 particular, the inventive concepts described in detail below are applicable to systems, devices, architectures, methods, and techniques where main decoding and pre-rendering are performed by a main device (UE) with high resources such a powerful computational processing (or processor) resources with significant power or battery capabilities (e.g., an edge or other network node / server of an 5G system, a high performance mobile device, etc.) and final decoding and 20 post-rendering are performed by a different device with lower resources relative to the main device (e.g., a lightweight device, a wearable device, AR glasses, head-mounted display, heads- up-display, etc.). 25

[034] FIG. 1 shows schematically a user having a smartphone 2 and wearing a headset 3. The smartphone could in the context of the present invention serve as the main, pre-rendering device, while the headset could serve as the lightweight, user-held, post-rendering device. It is the post rendering device that has the most recent information about the users head pose Р.

[035] With reference to FIG: 2, a user head pose P is in the present context defined by three degrees of freedom, namely rotation a around the yaw axis 101, rotation ẞ around the pitch axis 102, and rotation y around the roll axis 103. Of course, the head pose will also be associated 30 with a position in the room, defined by three additional degrees of freedom, spatial coordinates x, y, z. However, spatial translation will not be relevant for the purposes of the present disclosure.

[036] FIG. 3 shows an example of some of the functional blocks that may be implemented in the main device 2 and lightweight device 3 in an example split rendering system. -7- WO 2024 / 182457 PCT / US2024 / 017570

[037] Here, the main device, or pre-rendering device, 2 includes a decoder 11, a binaural renderer 12, a metadata generator 13, a first encoder 14, a second encoder 15, and a multiplexer 16. The main device 2 may also include a pose decoder 17.

[038] The decoder 11, e.g., an IVAS decoder, is configured to receive and decode a 5 bitstream b1, and decode an immersive audio content A. The binaural renderer 12 is configured to receive (or obtain) the immersive audio content A and a reference pose P', and responsively provide a reference binaural representation (rendition) Binref, associated with the reference pose P'. The reference pose may be an assumed head pose, or be determined based on head pose information received from the lightweight device 3 via pose decoder 17. The binaural renderer 10 12 is further configured to responsively render N (N>0) binaural representations (pre-renditions) Binn corresponding to a set of probing poses Pn associated with the reference pose Р'.

[039] The metadata generator 13 is configured to receive the reference binaural 15 representation Binref and pre-renditions Binn, and responsively generate reconstruction metadata M to enable reconstruction of the pre-renditions from the reference binaural representation. Optionally, the reference pose P' may be received by metadata generator 13, and responsively encoded in the reconstruction metadata M.

[040] Pose decoder 17 is an optional block that is not required for all implementations. When pose decoder 17 is present, pose decoder 17 is configured to receive head pose information from the lightweight device 3 via bitstream bp, and responsively generate the 20 reference pose P'. 25 30

[041] The first encoder 14 is configured to receive the reference binaural representation Binref, and responsively encode the reference binaural representation Binref as encoded bitstream b11. The second encoder 15 is configured to receive reconstruction metadata M, and responsively encode the reconstruction metadata (and optionally pose information) as encoded bitstream b12. The multiplexer 16 is configured to receive the encoded bitstreams b11 and b12 from the outputs of the two encoders 14 and 15, and responsively combine the encoded bitstreams b11 and b12 into a bitstream b2. The main device may also include an interface to output the bitstream b2, whereby the bitstream may be subsequently transmitted or otherwise made available to another device that is external to the main device 2, here the lightweight device 3.

[042] In some embodiments, the encoder 15 is further configured to encode pose information into bitstream b12, where the encoded pose information is indicative of the reference pose P' and / or the probing poses Pn. -8- WO 2024 / 182457 PCT / US2024 / 017570 5

[043] The lightweight device, or post-renderer device, 3 here includes a demultiplexer 21, a first decoder 22, a second decoder 23, a binaural reconstruction block 24 and a head-tracker 25. Optionally, the lightweight device 3 also includes a pose information encoder 26.

[044] The demultiplexer 21 is configured to receive bitstream b2 from the main device 2 and responsively separate the received bitstream b2 into two encoded bitstreams b21 and b22. The decoder 22 is configured to receive encoded bitstream b21, and responsively decode bitstream b21 into a reference binaural signal Binref. The decoder 23 is configured to receive encoded bitstream b22, and responsively decode bitstream b22 into metadata M' (and, if present, information about the reference pose P' and / or the probing poses Pn). The binaural reconstruction block 24 is 10 configured to receive a current user head pose P detected by the head tracker 25, and responsively determine a binaural output based on the reference binaural signal Binref, and the metadata M', and the current head pose Pin relation to the reference pose P'.

[045] As noted, the reference pose P' and / or the probing poses Pn may be included in the bitstream b22 received from the main device 2. This is especially useful when the reference pose 15 is based on pose information received by the main device 2 from the lightweight device 3. However, in some implementations, the reference pose P'is an assumed pose and thus the lightweight device is already aware of pose information P'. For example, the reference pose may be a "straight ahead" pose, e.g., a pose looking straight at a display device.

[046] In a similar manner, information about the probing poses may be received in the 20 bitstream, but may alternatively be predefined and known by the lightweight device. For example, the probing poses may be pre-defined deviations from the reference pose. 25

[047] The encoder 26 is an optional block that is not required in all implementations. If encoder 26 is present, encoder 26 is configured to receive pose information P from the head- tracker 25, and responsively encode the pose information in a bitstream bp, which is sent to the main device 2.

[048] In an example implementation, a heavy weight device 2 uses a pose P' to generate a reference binaural signal Binp, and metadata (MD) such that the light weight post renderer can do the pose correction from P' to the actual pose P and generate Bin, from Binp, using metadata M, wherein Bin, has all the spatial cues as per Pose P. In some implementations, pose 30 P' at the pre-renderer is an assumed pose without any information from light weight device. In some other implementations, pose P' at the pre-renderer is received from light weight device through a back channel. Computing metadata corresponding to various probing poses such that the lightweight post renderer can do the pose correction from P' to the actual pose P can require multiple binaural renditions at the pre-renderer, also referred to as pre-renditions in this -9- WO 2024 / 182457 PCT / US2024 / 017570 5 10 15 document, and these pre-renditions may be complexity intensive. Moreover, computing metadata corresponding to multiple probing poses can increase the metadata bitrate significantly. Hence, it is desired to carefully select the probing pose points for pre-renditions such that the total number of pre-renditions can be limited. Moreover, it is also desired to reduce the amount of metadata corresponding to these probing pose points while preserving the overall perceptual quality in the estimated binaural signal Bin corresponding to actual pose P at the post renderer. Following embodiments provide example implementations of such low complexity low metadata rate implementations. 1-AXIS SPLIT RENDERING METADATA CALCULATION

[049] An example case is described where split rendered metadata is calculated with single-axis (e.g., yaw) corrections. This example may employ two pre-renditions (N = 2), where the two probing poses are P' + X and P' – X' and wherein X and X' are deviations from the reference pose P' around the axis 101. More specifically, X and X' may be (yaw) angles a and -a. Note that the deviation from the reference pose is not necessarily equal (±a) but is assumed here for simplifying the description.

[050] One potential way to save pre-renderer complexity is to skip one rendition. This is a workable solution as long as the trend in time of the yaw angle is known. In that case, the probing rendition may be done for just either +a or -a towards which the pose is expected to evolve. However, in general, such a trend may be unknown. For example, the trend is unknown 20 in cases where the current pose is static, since it is unknown whether the user will next turn the head to the right or the left. For this example, if the probing rendition is done in the wrong direction, the adjustments done by the post-renderer would have to rely on extrapolated rather than interpolated metadata, which may reduce the quality of the post-rendered output signal.

[051] A first solution to reducing the number of renditions to two is to give up pre- 25 rendering to the reference pose P'. Instead, pre-renditions are generated for poses P' - a and P'+ a and one of these binaural renditions, e.g., for pose P' - a is transmitted to the post- renderer device. In case, pose P at the post-renderer is static and thus identical to P', the post renderer will thus have to do a correction by +a. The advantage with this over the solution from 30 an approach with three renditions is that the complexity for one rendition is saved and that only a single set of correction metadata needs to be transmitted. One potential disadvantage of the solution is that a correction by the post-renderer will virtually always be needed even if the pose is static. A further potential disadvantage is the bias of the solution with potentially less accurate post-renderer output signal for pose P' + a compared to the (perfect) post-renderer output signal - 10 - WO 2024 / 182457 PCT / US2024 / 017570 5 for pose P' – a. This bias may be overcome by transmitting one of the 1 linaurall channels (e.g. left channel) of the rendition for pose P' - a and one of the binaural channels (e.g. right channel) of the rendition for pose P' + a. The post rendering for pose P will consequently involve adjusting the left channel using renderer metadata relative to the rendition of that channel for pose P' - a and adjusting the right channel using renderer metadata relative to the rendition for pose P' + a.

[052] Another solution for that problem is in the pre-renderer to firstly generate an approximation of the binaural rendition for the reference pose P' through interpolation between the available renditions for poses P' - a and P' + a. This may simply involve averaging the two 10 available binaural renditions to generate a reference binaural rendition for virtual reference pose P'. Secondly, split renderer metadata can be calculated as described in U.S. 63 / 340,181 based on the binaural renditions for P', P' - a and P' + a, whereby it is notable that the fact that the reference rendition is obtained through interpolation creates metadata symmetries which alleviates the need to calculate and transmit metadata associated with one of the probing 15 positions. Thirdly, the reference rendition along with the metadata are transmitted to the post renderer device where operations can take place as described in U.S. 63 / 340,181, hereby incorporated by reference.

[053] In the following, the above-described solutions are extended generating split renderer metadata for post-renderer corrections for pose deviations around 2 and 3 axes, e.g., for 20 corrections of yaw and pitch deviations or for corrections of yaw, pitch and roll deviations. 2-AXES SPLIT RENDERING METADATA CALCULATION

[054] Another example case is described where split rendered metadata is calculated with two-axis (e.g., yaw and pitch) correction. For this example, four probing poses may be considered with a reference pose, where techniques suggested by U.S. 63 / 340,181 may be 25 carried out. Two probing poses may be employed to probe first axis (e.g., yaw axis) deviations from the reference pose, e.g., by varying the pose relative to the reference pose by deviations of ±a about the first axis while keeping a second axis (e.g., pitch) unchanged. Two other probing poses may be employed to probe pitch deviations from the reference pose, e.g., by varying the pose relative to the reference pose by pitch deviations of ±ß while keeping the yaw unchanged. 30 The total number of renditions is five for this example.

[055] In the following description, techniques are described for calculating split renderer metadata for post-renderer corrections about a number of axes. Although the description may refer to yaw, pitch, and / or roll corrections, the same techniques may be applied for -11- WO 2024 / 182457 PCT / US2024 / 017570 5 10 computationally efficient calculation of post-renderer metadata around any axis or set of axes. Thus, the terms yaw and pitch and / or roll in the following description could thus be replaced by any other appropriate axis.

[056] FIG. 4 is a flow chart illustrating processing in the main device 2 in accordance with embodiments of a first aspect of the invention, relating to a method of rendering audio in the main device 2 to enable split rendering with pose correction around multiple rotational axes.

[057] The flow chart may be broken into various blocks or partitions, such as blocks S11 - S17. Processing for the various blocks of FIG. 4, which may be described as operations, processes, methods, steps, acts or functions, may commence at block S11.

[058] In step S11(obtain audio content), a first bitstream is received and decoded (e.g., by decoder 11) to obtain an immersive audio content A. Step S11 may be followed by step S12.

[059] In step S12 (obtain reference pose), a reference pose P' is obtained. The reference pose may be an assumed pose (e.g. straight ahead) or may be based on pose information received from the lightweight device 3. Step S12 may be followed by step S13. 15

[060] In step S13 (pre-rendering), a first number of binaural pre-renditions are rendered (e.g., by renderer 12), wherein the binaural pre-renditions correspond to a set of probing poses Pn, including poses deviating from the reference pose P'by rotation around at least one of the rotational axes. Optionally, the set of probing poses also includes the reference pose. Step S13 may be followed by step S14. 20 25 30

[061] In step S14 (calculate approximate representations), a second number of approximate binaural representations Bin'm are calculated based on the binaural pre-renditions Binn, wherein the approximate binaural representations Bin'm correspond to a set of virtual probing poses Pm, each virtual probing pose (Pm) deviating from the probing poses P, by rotation around at least one of the rotational axes. Step S14 may be followed by step S15.

[062] At step S15 (determine Binref), a reference binaural representation, Binref, is determined. As discussed in more detail in the following, the reference binaural representation Binref may be equal to one of the binaural pre-renditions Bin, or one of the approximate binaural representations Bin'm. The reference binaural representation Binref may correspond to the reference pose P'and may then be a pre-rendition corresponding to the reference pose. A reference binaural representation Binref corresponding to the reference pose P'may also be obtained by linearly combining several pre-renditions. Step S15 may be followed by step S16.

[063] In step S16 (generate M), reconstruction metadata M, which enables reconstruction of the binaural pre-renditions Bin, and the approximate binaural representations Bin'm from the reference binaural representation Binref, is computed. - 12 - WO 2024 / 182457 PCT / US2024 / 017570 5

[064] Steps S14 - S16 may all be performed by metadata generator 13 in figure 3. If the reference binaural representation is rendered, such rendering may be performed by renderer 12, and the reference binaural representation will be one of the binaural pre-renditions. Step S16 may be followed by step S17.

[065] In step S17 (encode and output bitstream), the reference binaural representation Binref and the reconstruction metadata M are encoded (e.g., by encoders 14, 15 or a single encoder) in an output bitstream (b2), which is subsequently outputted on an appropriate communication channel. The reconstruction metadata may be quantized and encoded based on symmetries in reconstruction metadata. For example, the reconstruction metadata may be 10 encoded using differential coding between metadata relating to different (symmetrical) probing poses.

[066] The step of computing reconstruction metadata (step S16) may include computing axis-specific metadata for each rotational axis. In that case, the method comprises, for each rotational axis, selecting a first set of representations from the binaural pre-renditions Binn and 15 the approximate binaural representations Bin'm, this first set of representations corresponding to probing poses deviating from each other by rotation around the axis (e.g. the yaw or pitch axis), and computing axis-specific reconstruction metadata M, H enabling reconstruction of at least one representations in the set from another representation in the set, the axis-specific 20 25 30 reconstruction metadata M, H representing deviation around that particular axis.

[067] According to an example solution, the pre-renderer may render binaural presentations for three probing poses P1, P₂ and P3. 1. In a first step, pre-rendering takes place to obtain pre-renditions Binı Bin2 and Bin3 for the three probing poses P1, P₂ and P3 with P₁ = P' + (-а, -B) P2 = P' + (-a, +B) P3 = P'+ (+a, 0) Herein, a denotes a probing angle for deviations around the yaw axis, herein referred to as yaw deviations, and ẞ a corresponding probing angle for deviations around the pitch axis, herein referred to as pitch deviations. 2. In a second step, an approximate binaural presentation Bin', for a virtual probing pose P4 = P'+ (-a, 0) is calculated, for instance by interpolating the renditions for P₁ and P2. - 13- WO 2024 / 182457 PCT / US2024 / 017570 5 3. In a third step, pitch correction metadata H can be calculated based on pre-renditions Binı and Bin2 and approximate rendition Bin', using a technique described in U.S. 63 / 340,181 and with Bin' as reference representation. 4. In a final step, yaw correction metadata M can be calculated based the pre-rendition for probing pose P3 and the approximate rendition for pose P4, using a technique described in U.S. 63 / 340,181 and again with Bin' as reference representation.

[068] It is notable that a sequence of operation steps can be performed for obtaining yaw correction metadata prior to calculating pitch correction metadata. In that case the probing poses P₁, P2 and P3 and the virtual probing pose P4 are given as 10 P₁ = P'+ (-a,-B) 15 20 P2 = P'+ (+a,-B) P3 = P' + (0, +B) P4 = P'+ (0, -B).

[069] The yaw and pitch metadata M, H and a binaural reference representation are encoded and transmitted to the lightweight device 3, which enables the lightweight device to perform pose correction. In the above example, approximate rendition Bin' was used as reference for the calculation of both yaw and pitch metadata, and it will be appropriate to encode and transmit this representation.

[070] However, in principle, it is not necessary to use the same reference presentation for both sets of metadata, and in fact the binaural reference rendition to be transmitted to the post- renderer (lightweight device 3) may be any of those available for the probing poses P1, P2 and P3 and the virtual probing pose P4.

[071] It is also possible to choose a different binaural reference rendition, e.g., associated with virtual reference pose P'. This reference rendition may be obtained through low-complex 25 post-renderer operations based on any (or a combination) of the available pre-renderings for the probing positions and using a technique described in U.S. 63 / 340,181. It is also possible to obtain the reference rendition directly based on linear or triangular interpolation. Let b1, b2 and b3 denote the binaural pre-renditions for probing poses P1, P₂ and P3, an interpolated reference rendition bp for virtual pose P' can be obtained by the following weighted averaging: 30 1 1 bp =(b₁+ b2) +b3 -14- WO 2024 / 182457 PCT / US2024 / 017570 10

[072] As described above in the example with yaw-only correction, the fact that the reference rendition and the approximate rendition for the virtual probing pose P4 are obtained through interpolation may cause metadata symmetries which may alleviate the need to calculate and transmit metadata associated with some of the probing positions. 5 3-AXES SPLIT RENDERER METADATA CALCULATION

[073] Now, considering the 3 axes case with yaw, pitch and roll correction, the obvious solution would be to consider 6 probing poses and the reference pose and to carry out techniques suggested by U.S. 63 / 340,181. Two additional probing poses compared to the obvious solution of the 2-axes case would be to probe roll deviations from the reference pose, e.g., by varying the pose relative to the reference pose by roll deviations of ±y while keeping pitch and yaw unchanged. This would require a total of 7 pre-renditions.

[074] In the following solutions are described for calculating split renderer metadata for post-renderer corrections around three axes. The description assumes yaw, pitch and roll corrections, but the same principles could be applied for computationally efficient calculation of 15 post-renderer metadata around any three (orthogonal) axes. The used entities yaw, pitch and roll in the following description could thus be replaced by any three rotation axes out of yaw, pitch, and roll. 20 25

[075] According to an example solution, the pre-renderer may render binaural presentations for only four probing poses P1, P2, P3 and P4. 1. In a first step, pre-rendering takes place to obtain pre-renditions Binı Bin2 and Bin3 for the three probing poses P1, P₂ and P3 with P₁ = P' + (-а,-B,-y) P2 = P'+ (-a,-B, +y) P3 = P' + (-a, +B, 0) Herein, a denotes a probing angle for yaw deviations, ẞ a corresponding probing angle for pitch deviations and y a probing angle for roll deviations. 2. In a second step, an approximate rendition Bin' for a virtual probing pose P5 = P'+ (-a, -B, 0) is calculated, for instance by interpolating the renditions for P₁ and P2. -15- WO 2024 / 182457 PCT / US2024 / 017570 5 10 15 3. In a third step, roll correction metadata can be calculated based on pre-renditions Bin and Bin2 and approximate rendition Bin', using a technique described in U.S. 63 / 340,181, and using Bin' for pose Pg as reference presentation. 4. In a fourth step, pitch correction metadata can be calculated based the pre-rendition Bin3 for pose P3 and the approximate rendition Bin' for virtual probing pose P using a technique described in U.S. 63 / 340,181 and again using Bin'; for pose Pg as reference presentation. 5. In a fifth step, an approximate rendition Bin'2 for a virtual probing pose P6 = P' + (-a, 0,0) is calculated, for instance by interpolating (e.g., averaging) the renditions for P3 and Ps or by carrying out low-complexity post-renderer operations of US 63 / 386,465 using the pitch correction metadata from step 4 and one or both pre-renditions for probing poses P3 or / and Pg. 6. In a sixth step, pre-rendering takes place to obtain a fourth pre-rendition Bin4 for probing pose P with 20 25 P4 = P'+ (+a, 0, 0) 7. In a final step, yaw correction metadata can be calculated based on the pre-rendition Bin4 for probing pose P4 and the approximate rendition Bin'2 for pose P, using a technique described in U.S. 63 / 340,181 and using either one of the renditions as reference presentation.

[076] As described above, it is possible to do the operation steps to obtain yaw, pitch and roll correction metadata in different orders. The order may also be adapted based on properties of the immersive audio signal. Some of the steps and correction metadata calculations may even be omitted based on such immersive audio signal properties. For instance, for an audio signal with dominant sound arriving from left or right relative to reference pose P', pitch pose correction can be approximated with a table that contains gain parameters corresponding to various pitch angles. Such a table can be computed once during initialization time and both pre-renderer and post renderer can have prior knowledge about these tables. In these cases, pitch correction metadata is not necessary in the bitstream, and the above steps could be adapted to calculate yaw 30 and roll correction metadata only. Another example is the case with an audio signal with ar relative to reference pose P'. In these cases, rolldominant sound arriving from front or re - 16- WO 2024 / 182457 PCT / US2024 / 017570 5 correction metadata is not necessary in the bitstream, and roll pose correction can be approximated with a table that contains gain parameters corresponding to various roll angles.

[077] As discussed above, the binaural reference rendition (representation) to be transmitted to the post-renderer may be any of the pre-renditions for the exercised probing poses or any of the approximated pre-renditions at the virtual probing poses. In addition, any other approximated binaural pre-rendition can be used as reference rendition based on the available pre-renditions. A reference rendition may be obtained through low-complex post-renderer operations based on any (or a combination) of the available pre-renderings for the probing positions and using a technique described in U.S. 63 / 340,181. An approximation of the pre- 10 rendition for the reference pose can be obtained through interpolation between the available pre- renditions. 15 20 25 30

[078] According to one example, the approximated rendition for reference pose P' can be obtained from the available pre-renditions for probing poses P1 through P4. Let b1 through b4 denote the binaural pre-renditions for probing poses P₁ through P4, an interpolated reference rendition bp for virtual pose P' can be obtained by the following weighted averaging: 1 1 1 bpr = (b₁ + b2) + b3 +b4.

[079] Notwithstanding the above, if complexity constraints allow, it may be preferable to additionally generate a pre-rendition for the reference pose P' and to transmit this signal as reference rendition to the post-renderer. The availability of a pre-rendition for the reference pose P' may also be used to enhance the yaw metadata calculation in step 8 above. ITERATIVE SPLIT RENDERER METADATA ENHANCEMENT

[080] The above examples of complexity-reduced split renderer metadata calculation for post-renderer corrections have a certain bias. For instance, in the 3-axes case, roll correction metadata is calculated for yaw and pitch angle deviations from the reference pose P' of -a and -ẞ. This makes the obtained roll correction metadata less precise for the more likely case that the yaw and pitch angles of the pose corresponds to those of the reference pose P'. It would thus be more correct to calculate the roll correction metadata for yaw and pitch angle deviations equal to 0. Likewise, the pitch correction metadata is biased since it is calculated for a yaw deviation angle of -a rather than 0. The bias in the metadata calculations may in turn cause inaccuracies in the renditions obtained by the post-renderer using that biased metadata.

[081] In the following an iterative enhancement technique is described that can mitigate the described bias and the resulting post-renderer inaccuracies. It is assumed that a binaural - 17- WO 2024 / 182457 PCT / US2024 / 017570 5 rendition for the reference pose P' is available. Reference is made to the above procedural description of the 3 axes case with yaw, pitch and roll correction.

[082] In a first enhancement step, roll correction metadata is enhanced using the reference pose P' and the virtual probing poses

[083] P, = P'+ (0,0,-y), as P8 = P' + (0,0, +y). Part of this procedure is the calculation of approximate renditions for these virtual probing poses, for instance by carrying out low-complexity post-renderer operations of 10 US 63 / 386,465 or U.S. 63 / 340,181 using the previously calculated pitch and yaw correction metadata from the steps above and the pre-renditions for probing poses P1 and P₂. With these approximate renditions and the rendition for the reference pose P', roll correction metadata is re- calculated, e.g., using a technique described in U.S. 63 / 340,181. 15 20 25 30

[084] In a corresponding second enhancement step, pitch correction metadata is enhanced using the reference pose P' and the virtual probing poses

[085] P, =P'+ (0, -B,0), P10 = P'+ (0, +B, 0). Part of this procedure is the calculation of approximate renditions for these virtual probing poses, for instance by carrying out low-complexity post-renderer operations of US 63 / 386,465 or U.S. 63 / 340,181 using the previously calculated roll and yaw correction metadata and / or the previously performed pre-renditions. For instance, an approximate rendition for pose P can be calculated by interpolating (e.g., averaging) between the pre-renditions for poses P1 and P₂ followed by adjusting that rendition with regards to changing the yaw deviation angle from -a to 0 applying the post-renderer technique using the previously calculated yaw correction metadata. The interpolating operations between the pre-renditions for poses P1 and P₂ may also involve applying post-renderer techniques using the previously enhanced roll correction metadata. An approximate rendition for pose P1o can be calculated using the pre- rendition for probing pose P3 applying post-rendering techniques using the previously calculated yaw correction metadata.

[086] In a corresponding third enhancement step, yaw correction metadata is enhanced using the pre-renditions for reference pose P' and probing pose P4 and an approximate rendition for virtual probing pose P6. This enhancement step may involve applying post-renderer - 18- WO 2024 / 182457 PCT / US2024 / 017570 5 10 operations using the previously enhanced roll and pitch metadata and the available pre-renditions for probing poses P1, P₂, and / or Рз.

[087] Each of the above-described metadata enhancement steps relies on previously calculated metadata. It is thus possible to achieve even more enhancements by carrying out multiple iterations. LOW COMPLEXITY, LOW SIDE INFORMATION (METADATA) PRE-RENDITIONS

[088] FOR YAW AND PITCH CORRECTION FIG. 5 is a flow chart illustrating processing in the main device 2 in accordance with embodiments of a second aspect of the invention, relating to a method of rendering audio to facilitate split rendering with pose correction around yaw axis and pitch axis.

[089] The flow chart may be broken into various blocks or partitions, such as blocks S21 - S26. Processing for the various blocks of FIG. 5, which may be described as operations, processes, methods, steps, acts or functions, may commence at block S21.

[090] In step S21 (obtain audio content), a first bitstream is received and decoded (e.g., 15 by decoder 11) to obtain an immersive audio content A and in step S22 (obtain reference pose) a reference pose P'is obtained. The reference pose may be an assumed pose (e.g. straight ahead) or may be based on pose information received from the lightweight device 3. Step S21 may be followed by step S22.

[091] At step S23 (render Binref), the immersive audio content A is rendered (e.g., by 20 renderer 12) into a reference binaural representation, Binref, corresponding to a reference pose P'. Step S23 may be followed by step S24. 25 30

[092] At step S24 (pre-rendering), the immersive audio content A is rendered (e.g., by renderer 12) into two binaural pre-renditions Binn, wherein the binaural pre-renditions correspond to two probing poses P, deviating from the reference pose by rotation around both yaw and pitch axes. In other words, each probing pose deviates form the reference pose by rotation around a probing axis 104 (see figure 2) with the same origin as the yaw and pitch axes, and extending between the yaw axis 101 and the pitch axis 102. The deviation in yaw and pitch may be equal, in which case the probing axis 104 extends symmetrically between the yaw and pitch axis (i.e., 45 degrees from each axis). Step S24 may be followed by step S25.

[093] In step S25 (compute M and H), yaw metadata M representing a deviation around the yaw axis, and pitch metadata H representing deviation around the pitch axis are computed (e.g. by metadata generator 13) for each probing pose Pn. Step S25 may be followed by step S26. - 19 - WO 2024 / 182457 PCT / US2024 / 017570 5

[094] In step S26 (encode and output bitstream), the reference binaural representation Binref and the yaw metadata M and the pitch metadata H are encoded (e.g., by encoders 14, 15) in an output bitstream b2, which is subsequently outputted on an appropriate communication channel. The yaw and pitch metadata may be quantized and encoded based on symmetries in reconstruction metadata. For example, the metadata may be encoded using differential coding between metadata relating to different (symmetrical) probing poses.

[095] As discussed in more detail below, the step of computing yaw metadata and pitch metadata involves computing complete reconstruction metadata M to enable reconstruction of the binaural pre-renditions Bin, from the reference binaural representation Binref, and then 10 computing the yaw and pitch metadata based on the complete reconstruction metadata. Such complete reconstruction metadata may include, for each time-frequency tile, a complex or real 2x2 transformation matrix M. 15

[096] The yaw metadata may include, for each time-frequency tile, a complex or real 2x2 yaw correction matrix M. The pitch metadata may include, for each time-frequency tile, a real 2x2 diagonal pitch correction matrix H.

[097] In an example implementation, the light-weight post renderer device 3 sends the reference head pose P' to the heavy weight pre renderer device 2 through a back channel. The heavy weight device uses the P' pose to generate a reference binaural signal Binp, and metadata such that the post renderer can do the pose correction from P' to the actual pose P and generate 20 Binp from Bin, using the metadata, wherein Bin, has all the spatial cues as per Pose P. The 25 deviation between P' and P depends on the motion-to-sound latency as described in this document.

[098] Let the 3DOF pose angles along yaw, pitch and roll axes in pose P'be aref, Bref, Yref respectively and the deviations in the angles along yaw, pitch and roll axes between P and P' be aa, Ba Y a. It has been observed that it is more perceptually important to correct aa (or deviation along yaw axis) than Ba and Ya (Pitch and roll deviations). Furthermore, deviations along pitch axis are more relevant to user applications than deviations along roll axis.

[099] In an example implementation, a low complexity and low metadata rate solution is 30 proposed to correct a and a wherein, the heavy weight device uses the reference pose P' to generate a reference binaural signal Bin,and two additional renditions are computed as per the probing head poses shown below P₁ = P' + (a, ẞ, 0), - 20 - 摘要 本公开描述了用于基于有限数量的预渲染的低复杂度的分离渲染的技术,从而 显著地减少了预渲染侧上需要的计算数量以及传输的元数据的量。本公开进一 步描述了针对围绕一个、两个或三个旋转轴的后渲染器校正(例如,针对偏航、 俯仰和翻滚的偏离)的低复杂度解决方案。

Claims

CLAIMS 1. A method of rendering audio to enable split rendering with pose correction around multiple rotational axes, the method comprising: obtaining an immersive audio content (A); obtaining a reference pose (P'); rendering the immersive audio content (A) into a first number of binaural pre-renditions (Binn), wherein the binaural pre-renditions (Binn) correspond to a set of probing poses (Pn) associated with the reference pose (P'), wherein the set of probing poses (Pn) include poses equal to the reference pose (P’) and / or poses that deviate from the reference pose (P') by rotation about at least one of the rotational axes; calculating a second number of approximate binaural representations (Bin'm) based on the binaural pre-renditions (Binn), wherein the approximate binaural representations (Bin'm) correspond to a set of virtual probing poses (Pm), wherein the virtual probing poses (Pm) deviate from the probing poses (Pn) by rotation around at least one of the rotational axes; determining a reference binaural representation (Binref) based on one or more of the binaural pre-renditions (Binn) and the approximate binaural representations (Bin'm); computing reconstruction metadata (M) to enable reconstruction of the binaural pre- renditions (Binn) and the approximate binaural representations (Bin'm) from the reference binaural representation (Binref); encoding the reconstruction metadata (M) and the reference binaural representation (Binref) in an output bitstream (b2); and outputting the output bitstream (b2).

2. The method according to claim 1, wherein the approximate binaural representations (Bin'm) are calculated by interpolating the binaural pre-renditions (Binn).

3. The method according to claim 1 or 2, wherein at least one of the approximate binaural representations (Bin'm) is computed by modifying the reconstruction metadata, and applying the modified reconstruction metadata to one of the binaural pre-renditions.

4. The method according to any one of the preceding claims, wherein the reference binaural representation (Binref) is selected from the binaural pre-renditions (Binn) and the approximate binaural representations (Bin'm).

5. The method according to any one of claims 1-3, wherein the reference binaural representation (Binref) corresponds to the reference pose (P') and is obtained by linearly combining the binaural pre-renditions.

6. The method according to any one of claims 1-3, wherein the reference binaural representation (Binref) corresponds to the reference pose (P') and is obtained by binaural rendering of the immersive audio content (A).

7. The method according to any one of the preceding claims, wherein the step of computing reconstruction metadata includes: for each rotational axis: selecting a first set of representations from the binaural pre-renditions (Binn) and the approximate binaural representations (Bin'm), the first set of representations corresponding to probing poses deviating from each other by rotation around the rotational axis, and computing axis-specific reconstruction metadata (M, H) which enables reconstruction of at least one representations in the set from another representation in the set, wherein the axis- specific reconstruction metadata (M, H) represents a deviation around the rotational axis.

8. The method according to claim 1, wherein at least one of the axis-specific reconstruction metadata (M, H) is computed before all approximate binaural representations (Bin'm) are calculated, and wherein at least one of the approximate binaural representations (Bin'm) is calculated by applying the axis-specific reconstruction metadata to one of the binaural pre- renditions.

9. The method according to any one of the preceding claims, wherein the first number is D+1 and the second number is D-1, where D is the number of rotational axes.

10. The method according to claim 9, wherein D=2, and wherein: a first probing pose (P1) deviates from the reference pose (P') by a first angle (-α) around a first axis and by a second angle (-β) around a second axis, a second probing pose (P2) deviates from the reference pose (P') by the first angle (-α) around the first axis and by a third angle (+β) around the second axis, a third probing pose (P3) deviates from the reference pose (P') by a fourth angle (+α) around the first axis and is equal to the reference pose (P') around the second axis, a first virtual probing pose (P4) deviates from the reference pose (P') by the first able (-α) around the first axis and is equal to the reference pose (P') around the second axis.

11. The method according to claim 10, wherein the fourth value (+α) is the negative of the first value (-α), and wherein the third value (+β) is the negative of the second value (-β).

12. The method according to claim 9, wherein D=3, and wherein: a first probing pose (P1) deviates from the reference pose (P') by a first angle (-α) around a first axis, by a second angle (-β) around a second axis, and by a third angle (-γ) around a third axis, a second probing pose (P2) deviates from the reference pose (P') by the first angle (-α) around the first axis, by the second angle (-β) around the second axis, and by a fourth angle (+γ) around the third axis, a third probing pose (P3) deviates from the reference pose (P') by the first angle (-α) around the first axis, by a fifth angle (+β) around the second axis, and is equal to the reference pose (P') around the third axis, a fourth probing pose (P4) deviates from the reference pose (P') by the sixth angle (+α) around the first axis, and is equal to the reference pose (P') around the second and third degrees of freedom, a first virtual probing pose (P5) deviates from the reference pose (P') by the first angle (-α) around the first axis, by the second angle (-β) around the second axis, and is equal to the reference pose (P') around the third axis, and a second virtual probing pose (P6) deviates from the reference pose (P') by the first angle (-α) around the first axis, and is equal to the reference pose (P') around the second and third degrees of freedom.

13. The method according to claim 12, wherein the sixth angle (+α) is the negative of the first angle (-α), wherein the fifth angle (+β) is the negative of the second angle (-β), and wherein the fourth angle (+γ) is the negative of the third angle (-γ).

14. The method according to one of claims 10-13, wherein the first axis is the yaw axis, the second axis is the pitch axis and the third axis is the roll axis.

15. The method according to one of the preceding claims, wherein the reference pose (P') is based on user head pose information obtained from a user-held device.

16. The method according to claim 15, wherein pose information indicative of the reference pose (P') is encoded and included in the output bitstream (b2).

17. The method according to one of the preceding claims, wherein pose information indicative of the probing poses (Pn) and virtual probing poses (Pm) is encoded and included in the output bitstream (b2).

18. The method according to any one of the preceding claims, wherein the reconstruction metadata (M) includes, for each time-frequency tile, a two-by-two matrix.

19. The method according to claim 18, further comprising quantizing and encoding the reconstruction metadata (M) based on symmetries in reconstruction metadata.

20. The method according to claim 18, further comprising encoding the reconstruction metadata (M) using differential coding between metadata relating to different probing poses.

21. An apparatus comprising a processor and a memory coupled to the processor, wherein the processor is adapted to cause the apparatus to carry out the method according to any one of claims 1-20.

22. A computer-readable storage medium storing a program comprising instructions that, when executed by a processor, cause the processor to carry out the method according to any one of claims 1-20.

23. A method of rendering audio to enable split rendering with pose correction around yaw axis and pitch axis, the method comprising: obtaining an immersive audio content (A); obtaining a reference pose (P'); rendering the immersive audio content (A) into a reference binaural representation (Binref) corresponding to a reference pose (P'); rendering the immersive audio content (A) into one or more binaural pre-renditions (Binn), wherein the binaural pre-renditions correspond to one or more probing poses (Pn) which deviate from the reference pose by rotation around both yaw and pitch axes; computing, for each probing pose, yaw metadata (M) representing a deviation around the yaw axis, and pitch metadata (H) representing deviation around the pitch axis; encoding the reference binaural representation (Binref), the yaw metadata (M) and the pitch metadata (H) in an output bitstream (b2); and outputting the output bitstream (b2).

24. The method according to claim 23, wherein the pitch metadata (H) is computed based on an energy difference between the reference binaural representation and the binaural pre- renditions.

25. The method according to claim 23 or 24, wherein the step of computing yaw metadata and pitch metadata involves computing complete reconstruction metadata (?@) which enables reconstruction of the binaural pre-renditions (Binn) from the reference binaural representation (Binref), and then computing the yaw metadata and pitch metadata based on the complete reconstruction metadata.

26. The method according to claim 25, wherein the complete reconstruction metadata includes, for each time-frequency tile, a 2x2 transformation matrix (?@).

27. The method according to claim 25, wherein the yaw metadata includes, for each time- frequency tile, a 2x2 yaw correction matrix (M).

28. The method according to claim 27, wherein the yaw correction matrix (M) is obtained by multiplying the transformation matrix (?@) with a 2x2 diagonal gain matrix (G).

39. The method according to claim 28, wherein the gain matrix (G) for a specific probing pose Pshas the form: ^ =^^,^^ 0^ , ^ = ^^^,^2,^2(^,^ ^^,^2,^2(#,#^^^ ^^,^^^^^( , ^^,^^= ^^^^( ,$%,^2,^2represents a covariance matrix of reference binaural presentation '()^^, and $%&,^^,^^represents a 2x2 covariance matrix of a reconstructed binaural pre-rendition for the specific probing pose Ps.

30. The method according to any one of claims 23 - 29, wherein the pitch metadata includes, for each time-frequency tile, a diagonal real 2x2 pitch correction matrix (H).

31. The method according to claim 30, wherein the elements of the pitch correction matrix O = ^ℎ^ 0for a pose P1are obtained as:^^^^$%,^^,^^(1,1^^^^$%,^^,^^(2,2$%,^2,^2(where $%,^2,^2is the covariance matrix of reference binaural presentation '()^^, and $%,^^,^^is the covariance matrix of a binaural pre-rendition for pose P1.

32. The method according to any one of claims 23 - 31, wherein the probing poses (Pn) deviate from the reference pose (P') by equal rotation about both the yaw axis and the pitch axis.

33. The method according to any one of claims 23 - 32, wherein the reference pose (P') is based on user head pose information obtained from a user-held device.

34. The method according to claim 33, wherein pose information indicative of the reference pose (P') is encoded and included in the output bitstream (b2).

35. The method according to any one of claims 23 - 34, wherein pose information indicative of the probing poses (Pn) is encoded and included in the output bitstream (b2).

36. The method according to any one of claims 23 - 35, wherein the yaw metadata (M) includes, for each time-frequency tile, a two-by-two matrix.

37. The method according to any one of claims 23 - 36, wherein the pitch metadata (H) includes, for each time-frequency tile, a two-by-two diagonal matrix.

38. The method according to claim 36 or 37, further comprising quantizing and encoding the yaw and / or pitch metadata (M, H) based on symmetries in reconstruction metadata.

39. The method according to any one of claims 36 - 40, further comprising encoding the yaw and / or pitch metadata (M, H) using differential coding between metadata relating to different probing poses.

40. An apparatus comprising a processor and a memory coupled to the processor, wherein the processor is adapted to cause the apparatus to carry out the method according to any one of claims 23 - 39.

41. A computer-readable storage medium storing a program comprising instructions that, when executed by a processor, cause the processor to carry out the method according to any one of claims 23-39.

42. A main processing device (2), comprising: a decoder (11) configured to decode a first bitstream (b1) to obtain decoded immersiveaudio content (A); a renderer (12) configured to: obtain a reference pose (P’); render the immersive audio content (A) into a reference binaural representation (Binref) based on the reference pose P'; and render the immersive audio content (A) into a number of binaural pre-renditions (Binn), wherein the binaural pre-renditions (Binn) correspond to a set of probing poses (Pn) associated with a reference pose (P'), wherein the set of probing poses (Pn) include poses that deviate from the reference pose (P') by rotation about at least one of the rotational axes; a metadata generator (13) configured to compute reconstruction metadata (M) to enable reconstruction of the binaural pre-renditions (Binn) from the reference binaural representation (Binref); an encoder (14.15) configured to encode the reference binaural representation (Binref) and the reconstruction metadata (M) into an output bitstream (b2); and an interface (16) configured to output the output bitstream (b2).

43. A method of rendering audio in a main device (2) to enable split rendering with pose correction around at least one rotational axis, the method comprising: obtaining an immersive audio content (A); receiving head pose information (P) associated with a user of a lightweight processing device; determining, based on the head pose information, a reference pose (P’); rendering the immersive audio content (A) into a reference binaural representation (Binref) corresponding to a reference pose; rendering the immersive audio content (A) into one or more binaural pre-renditions (Binn), the binaural pre-renditions corresponding to one or more probing poses (Pn) deviating from the reference pose about the rotational axis; computing reconstruction metadata which enables reconstruction of the binaural pre- renditions (Binn) from the reference binaural representation (Binref), the reconstruction metadata including, for each time-frequency tile, a transformation matrix (?@); computing enhanced metadata (M) by multiplying, each transformation matrix (?@) for a specific probing pose (Ps) with an additional gain matrix (G), the additional gain matrix having the form:^ = ^^^,^^ 0^, ^ = ^^^^(^^,^^,^^(^,^, ^ = ^^^^^,^^,^^(#,#^^ 0 ^^,^^ ^,^^ ^^!,^^,^^(^,^ ^,^^^(^^!,^^,^^(#,#, where:the probing pose Ps, and $%&,^^,^^represents a 2x2 covariance matrix of a reconstructed binaural pre-rendition for the specific probing pose (Ps); encoding the reference binaural representation (Binref) and the enhanced metadata (M) in an output bitstream (b2); and outputting the output bitstream (b2).

44. The method according to claim 43, wherein the immersive audio content is rendered into at least two pre-renditions corresponding to at least two probing poses.

45. The method according to claim 43 or 44, wherein the rotational axis is the yaw axis and / or the roll axis.

46. The method according to claim 43, wherein the at least one rotational axis include a yaw axis and a roll axis, wherein the set of probing poses includes yaw probing poses deviating from the reference pose by rotation only about the yaw axis, and roll probing poses deviating from the reference pose only by rotation only about the roll axis, and wherein the reconstruction metadata includes: yaw reconstruction metadata which enables reconstruction of binaural pre- renditions (Binn) corresponding to the yaw probing poses from the reference binaural (Binref), roll reconstruction metadata which enables reconstruction of binaural pre- renditions (Binn) corresponding to the roll probing poses from the reference binaural representation (Binref), wherein the enhanced metadata is computed based on the yaw reconstruction metadata and / or roll reconstruction metadata.

47. The method according to claim 46, wherein the at least one rotational axis further include a pitch axis, wherein the set of probing poses includes pitch probing poses deviating from the reference pose only by rotation around the pitch axis, and further including:computing pitch reconstruction metadata which enables reconstruction of binaural pre- renditions (Binn) corresponding to the pitch probing poses from the reference binaural representation (Binref), wherein the pitch reconstruction metadata includes, for each time- frequency tile, a diagonal real 2x2 pitch correction matrix (H).

48. The method according to claim 47, wherein the elements of the pitch correction matrix O1F= ^ℎ0^ 0ℎ ^ for a pose P1 ^ are obtained as: ℎ= ^^^^($%,^^,^^(1,1, ℎ = ^$%,^^,^^(2,2^ $%,^2,^2(1,1 ^ ^^^($%,^2,^2(2,2where $,^2is the '()^^, and $the covariance matrix of a binaural pre-rendition for pose P1.

49. The method according to claim 47, wherein the elements of the pitch correction matrix O1F= qℎ 00ℎr for a pose P1are obtained as: '()^^, and $the covariance matrix of a binaural pre-rendition for pose P1.

50. The method according to any one of claims 43 - 49, wherein pose information indicative of the reference pose (P') is encoded and included in the output bitstream (b2).

51. The method according to any one of claims 43 - 50, wherein pose information indicative of the probing poses (Pn) is encoded and included in the output bitstream (b2).

52. The method according to any one of claims 43 - 51, further comprising quantizing and encoding the enhanced metadata based on symmetries in reconstruction metadata.

53. The method according to any one of claims 43 - 52, further comprising encoding the enhanced metadata using differential coding between metadata relating to different probing poses.

54. An apparatus comprising a processor and a memory coupled to the processor, wherein the processor is adapted to cause the apparatus to carry out the method according to any one of claims 43 - 53.

55. A computer-readable storage medium storing a program comprising instructions that, when executed by a processor, cause the processor to carry out the method according to any one of claims 43 - 53.

56. A method of rendering audio to enable split rendering with pose correction around multiple rotational axes, the method comprising: obtaining an immersive audio content (A); receiving head pose information (P, ΩP, ωP) associated with a user of a lightweight processing device; determining, based on the head pose information, a reference pose (P') and at least one of a head pose rotation axis (ΩP) and a head pose rate of rotation (ωP); rendering the immersive audio content (A) into a reference binaural representation (Binref) corresponding to the reference pose (P'); rendering the immersive audio content (A) into a set of binaural pre-renditions (Binn), wherein the binaural pre-renditions correspond to a set of probing poses (Pn) rotated with respect to the reference pose, wherein the probing poses are selected based on the head pose information (P, ΩP, ωP); computing reconstruction metadata (M) to enable reconstruction of the binaural pre- renditions (Binn) from the reference binaural representation (Binref); encoding the reference binaural representation (Binref) and the reconstruction metadata (M) in an output bitstream (b2); and outputting the output bitstream (b2).

57. The method according to claim 56, wherein a head pose rotation axis (ΩP) is determined and wherein the probing poses (Pn) deviate from the reference pose (P') by rotation around the head pose rotation axis (ΩP).

58. The method according to claim 57, wherein the probing poses (Pn) are symmetrically distributed around the reference pose (P').

59. The method according to any one of claims 56 - 58, wherein a head pose rate of rotation (ωP) is determined, and wherein, when the head pose rate of rotation (ωP) is below a threshold value, the probing poses (Pn) deviate from the reference pose by less than a first angle (ωlower), wherein, when the head pose rate of rotation is above the threshold value, the probingposes (Pn) deviate from the reference pose by more than a second angle (ωupper), and wherein the first angle (ωlower) is smaller than the second angle (ωupper).

60. The method according to any one of claims 56 - 59, wherein a head pose rate of rotation (ωP) is determined, and wherein, when the head pose rate of rotation (ωP) is below a threshold value, the probing poses (Pn) include only one probing pose around each rotational degree of freedom.

61. The method according to any one of claims 56 - 60, wherein pose information indicative of the reference pose (P') is encoded and included in the output bitstream (b2).

62. The method according to any one of claims 56 - 61, wherein pose information indicative of the probing poses (Pn) is encoded and included in the output bitstream (b2).

63. The method according to any one of claims 56 - 62, wherein the reconstruction metadata (M) includes, for each time-frequency tile, a two-by-two matrix.

64. The method according to claim 63, further comprising quantizing and encoding the reconstruction metadata (M) based on symmetries in reconstruction metadata.

65. The method according to claim 61 or 62, further comprising encoding the reconstruction metadata (M) using differential coding between metadata relating to different probing poses.

66. An apparatus comprising a processor and a memory coupled to the processor, wherein the processor is adapted to cause the apparatus to carry out the method according to any one of claims 56 - 65.

67. A computer-readable storage medium storing a program comprising instructions that, when executed by a processor, cause the processor to carry out the method according to any one of claims 56 - 65.

68. A method of audio processing with pose correction around multiple rotational axes, the method comprising: receiving a bitstream (b2) from a main device; decoding the bitstream to obtain a reference binaural representation (Binref) and first reconstruction metadata associated with a set of probing poses (Pn) representing deviation from a reference pose (P') by rotation around the multiple rotational axes; detecting a current head-pose (P);for each of the rotational axes: selecting a probing pose closest to the detected pose along the rotation axis, determining axis-specific reconstruction metadata (Mα, Mβ, Mγ) based on the first metadata associated with the selected probing pose and a difference between the reference pose and the current head pose along the rotational axis; and determining a binaural output corresponding to the current head pose based on the reference binaural representation (Binref), and the axis-specific reconstruction metadata.

69. The method according to claim 68, wherein the set of probing poses is obtained from the bitstream.

70. The method according to claim 68, wherein the set of probing poses is obtained by adding a set of offsets to the reference pose.

71. The method according to claim 70, wherein the set of offsets is pre-defined.

72. The method according to claim 70, wherein the set of offsets is obtained from the bitstream.

73. The method of any one of claims 68 - 72, wherein an indication of the reference pose (P') is obtained from the bitstream.

74. The method of any one of claims 68 - 73, wherein the method further includes transmitting an indication of the current head pose (P) to the main device.

75. The method according to claim 74, wherein the reference pose (P') is estimated based on an expected transmission delay.

76. The method according to any one of claims 68 - 75, wherein the first reconstruction metadata includes, for each time-frequency tile, a two-by-two matrix.

77. The method according to claim 76, wherein the axis-specific reconstruction metadata, for each time-frequency tile, is a two-by-two matrix, and wherein the binaural output for a time- frequency tile is obtained by multiplying the reference binaural representation for this time- frequency tile with the corresponding two-by-two matrix for each axis-specific reconstruction metadata.

78. An apparatus comprising a processor and a memory coupled to the processor, wherein the processor is adapted to cause the apparatus to carry out the method according to any one of claims 68 - 77.

79. A computer-readable storage medium storing a program comprising instructions that, when executed by a processor, cause the processor to carry out the method according to any one of claims 68 - 77.

80. A lightweight processing device (3), comprising: a decoder (22, 23) configured to decode a bitstream (b2) to obtain a reference binaural representation (Binref) and first reconstruction metadata associated with a set of probing poses (Pn) representing deviation from a reference pose (P') by rotation around multiple rotational axes; a head-tracker (25) configured to detect a current head-pose (P); a binaural reconstruction block (26) configured to: for each of the rotational axes, select a probing pose closest to the detected pose along the rotation axis, and determine axis-specific reconstruction metadata (Mα, Mβ, Mγ) based on the first metadata associated with the selected probing pose and a difference between the reference pose and the current head pose (P) along the rotational axis; and determine a binaural output corresponding to the current head pose based on the reference binaural representation (Binref), and the axis-specific reconstruction metadata.