Head tracking split rendering and head-related transfer function personalization

Abstract

Systems, methods, and computer program products are described for direction of arrival (DOA)-based head tracking split rendering and head related transfer function (HRTF) personalization. Head tracking audio rendering is performed separately by two devices. A first device receives a primary bitstream representation of encoded audio. The second device tracks head pose information. The first device decodes the primary bitstream using a primary decoder and encodes the decoded bitstream into a pre-rendered binaural signal and post-rendered metadata. The second device decodes the pre-rendered binaural signal and the post-rendered metadata from the intermediate bitstream, and provides the decoded pre-rendered binaural signal and post-rendered metadata to a lightweight renderer. A lightweight renderer renders the pre-rendered binaural signal into binaural audio based on the post-rendered metadata, the head pose information, the generic HRTF, and the personalized HRTF.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Application No. 63 / 405,538 filed on September 12, 2022 and U.S. Provisional Application No. 63 / 422,331 filed on November 3, 2022, each of which is hereby incorporated by reference in its entirety. Technical Field

[0003] The present disclosure relates to audio processing. In particular, the present disclosure relates to audio rendering. Background Art

[0004] Extended reality XR (AR / MR / VR) will increasingly rely on terminal devices with very limited power. Augmented reality AR glasses are a prominent example. In order to make the glasses as lightweight as possible, they cannot be equipped with heavy batteries. Therefore, in order to achieve reasonable computing time, only numerical calculations of very limited complexity can be performed on the processor included in the glasses. On the other hand, immersive audio is an essential media component of XR services. These services can typically support adjusting the presented immersive audio / visual scene in response to 3DoF or 6DoF user (head) movements. To perform the corresponding immersive audio reproduction with high quality, a higher numerical complexity is usually required.

[0005] A potential solution to this problem is to perform the rendering not on the device itself, but on an entity of the mobile / wireless network to which the terminal device is connected, or on a powerful mobile user equipment (UE) to which the terminal device is tethered. In this case, the terminal device will, for example, only receive audio that has been binaurally rendered. The 3DoF / 6DoF head pose information (head tracking metadata) will need to be transmitted to the rendering entity (network entity / UE). The problem with this is that the transmission delay between the terminal device and the network entity / UE can reach 100ms or more. Therefore, rendering on the network entity / UE will mean having to rely on outdated head tracking metadata, and the binauralized audio played by the terminal rendering device will not match the actual head pose of the head / terminal device. This delay is called motion-to-sound delay. If the delay is too large, the terminal user will perceive this as a drop in quality.

[0006] For the video component of immersive media rendering, this problem is being addressed through a split rendering approach, where an approximate portion of the video scene is rendered by the network entity / UE and the final video scene adjustment is done on the terminal device. For audio, this area is less explored so far. Summary of the invention

[0007] In various audio services such as immersive voice and audio services (IVAS), it is desirable to be able to track the user's head movements during audio rendering and adjust the audio accordingly, thereby providing the user with an immersive audio experience. This requires immersive audio decoding and binaural rendering using a set of head-related transfer functions (HRTFs), wherein the selection of a specific HRTF may depend on the properties of the immersive audio signal and the user's head movement (or head posture). Depending on the immersive audio format, decoding and head tracking binaural rendering may be computationally complex operations. For example, scene-based audio (e.g., high-order high-fidelity stereo), channel-based audio (e.g., with a 7.1.4 channel layout), or object-based audio with many objects may each rely on a large number of constituent audio components, and due to the large number of these components, the decoding and rendering calculations are very complex. This means that decoding the bitstream and performing binaural rendering in response to the user's head movement requires a lot of computational processing. Computational complexity requires power and generates heat, which may cause problems for small portable devices (such as AR glasses, etc.).

[0008] It is an object of the present invention to overcome the problems described herein and to provide split rendering, wherein head pose specific processing may be performed on a second device.

[0009] According to some embodiments, this object and other objects are achieved by a method according to claim 1 or claim 14. According to another embodiment, this object and other objects are achieved by a device held by a user according to claim 22.

[0010] Techniques for head tracking separate rendering and head related transfer function (HRTF) personalization based on direction of arrival (DOA) are described. Head tracking audio decoding and binaural rendering can be performed separately by two or more devices. In some examples, a first device can coordinate with a second device to perform separate decoding and rendering operations. A first device (e.g., a smart phone) receives a main bitstream representation of encoded audio. The first device decodes the main bitstream using a main decoder and a binaural renderer and renders it into a pre-rendered binaural signal, and encodes the pre-rendered binaural signal and post-rendering metadata, the post-rendering metadata including information about the HRTF associated with the binaural rendering. The first device provides the pre-rendered binaural signal and the post-rendering metadata as a multiplexed intermediate bitstream to the second device. The second device (e.g., a headset, AR glasses, or earbuds) tracks current head posture information. The second device decodes the pre-rendered binaural signal and the post-rendering metadata from the intermediate bitstream, and provides the decoded pre-rendered binaural signal and the post-rendering metadata to a lightweight renderer. The lightweight renderer renders the pre-rendered binaural signals into binaural audio based on the post-rendering metadata, the current head pose information, the generic HRTF and optionally the personalized HRTF.

[0011] The post-rendering metadata includes at least an indication of the pre-rendered HRTFs that have been used in binaural pre-rendering. The pre-rendered HRTFs are associated with the directions of arrival (DOA) (usually two angles) of the main directional components of the audio content relative to an assumed head pose. The indication of the pre-rendered HRTFs can be the DOA or some kind of index, allowing the device held by the user to identify the correct HRTF.

[0012] In some embodiments, the indication of the pre-rendered HRTF also includes one or several parameters that can be personalized.

[0013] Rendering may involve: calculating a compensated stereo audio signal by applying a HRTF compensation operation to the binaural audio signal, the HRTF compensation operation being configured to compensate for the effects of the pre-rendered HRTF; and calculating a binaural output signal by applying a post-rendered HRTF to the compensated stereo signal. These steps may be performed in a single operation. The HRTF compensation operation may involve, for example, the inverse of a pre-rendered HRTF obtained by accessing a lookup table. Other approaches may also be used to achieve compensation of the pre-rendered HRTF.

[0014] The binauralization described herein is performed using head-related transfer functions (HRTFs), but can equally be performed using binaural room impulse responses (BRIRs). Further, it should be noted that all HRTF processing needs to be performed for each time frame and each frequency band (usually represented as time / frequency tiles).

[0015] In some applications, the metadata also includes an assumed head pose. In other embodiments, the device held by the user is configured to send the current head pose to the main device. Optionally, the second device encodes at least a portion of the head pose information into a head pose bitstream and provides the bitstream to the first device. The first device decodes the head pose bitstream to obtain the head pose information, and then applies the head pose information to the main decoder and pre-render. The main decoder / pre-render decodes and pre-renders the main bitstream based on the received head pose information (also called the assumed head pose) and the general HRTF. In this case, the device held by the user can estimate the assumed head pose based on the expected transmission delay.

[0016] Information about the assumed head pose is transmitted to the second device along with other information unless the device derives the assumed head pose based on prior knowledge, which can be based on (head pose) information previously transmitted to the first device or based on an assumed head pose pre-agreed between the two devices.

[0017] In addition, the present disclosure relates to further inventive concepts, relating to techniques for DOA-based head tracking separated rendering using suitable prototype signals and optional diffuse signals. A first device decodes a main bitstream using a main decoder and renders the decoded bitstream into a main directional component (referred to as a prototype signal), zero or more diffuse signals, and post-rendering metadata. The first device then encodes the prototype signal, zero or more diffuse signals (or parameters representing them), and post-rendering metadata, and provides them to a second device as a multiplexed intermediate bitstream. The second device decodes the prototype signal, zero or more diffuse signals, and post-rendering metadata from the intermediate bitstream, and provides the decoded prototype signal, zero or more diffuse signals, and post-rendering metadata to a lightweight renderer. The lightweight renderer renders the prototype signal and zero or more diffuse signals as binaural audio based on the post-rendering metadata, information related to head posture, a general HRTF, and optionally a personalized HRTF.

[0018] Compared to traditional rendering techniques, the techniques described in this specification can achieve various technical advantages. Separating processing between two devices can reduce the amount of processing on the wearable device, thereby extending battery life. The wearable device performs lightweight rendering based on the user's current head posture, without having to rely solely on binaural rendering of a heavy rendering device that may only have access to delayed / outdated head posture information, thereby reducing motion-to-sound latency caused by outdated head posture information that may be used during rendering. The allocation of processing between the first devices can be flexible, for example, by adjusting the amount of head posture information transmitted from the second device to the first device (from none to all), thereby allowing matching of various wearable devices with different processing capabilities. Based on the specific embodiments described below and the associated drawings, other advantages, features and benefits in addition to those explicitly described above will be apparent.

[0019] This disclosure is provided to introduce a series of concepts in a simplified form and is not intended to identify key or important features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter. For example, the term "technique" may refer to (multiple) systems, (multiple) methods, computer-readable instructions, (multiple) modules, algorithms, hardware logic, and / or (multiple) operations as allowed by the context and the entire document as described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a block diagram of an example system implementing head-tracked separated rendering.

[0021] Figure 2 is a flowchart showing processing in a first device or a master device.

[0022] Figure 3is a flow chart showing processing in a second device or device held by a user.

[0023] Figure 4 An example technique for DOA-based separate rendering using pre-rendered binaural signals is shown.

[0024] Figure 5 Example techniques for HRTF personalization are shown.

[0025] Figure 6 An example technique for DOA-based split rendering using a prototype signal is shown. DETAILED DESCRIPTION

[0026] The following detailed description refers to the accompanying drawings, which form a part of the detailed description and show by way of illustration specific example configurations in which the concepts described may be practiced. These configurations are described in sufficient detail to enable those skilled in the art to practice the technology disclosed herein, and it should be understood that other configurations may be utilized and other changes may be made without departing from the spirit or scope of the proposed concepts. Therefore, the following detailed description should not be taken as having a limiting meaning, and the scope of the proposed concepts is limited only by the appended claims.

[0027] The systems and methods disclosed in this application can be implemented as software, firmware, hardware, or a combination thereof. In hardware implementations, the task division does not necessarily correspond to the division of physical units; on the contrary, a physical component can have multiple functions, and a task can be performed by several physical components in collaboration.

[0028] The computer hardware may be, for example, a server computer, a client computer, a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a cellular phone, a smart phone, a web device, a network router, a switch or a bridge, or any machine capable of (sequentially or otherwise) executing instructions specifying actions to be taken by the computer hardware. Further, the present disclosure will relate to any collection of computer hardware that executes instructions, alone or in combination, to perform any one or more of the concepts discussed herein.

[0029] Some or all of the components may be implemented by one or more processors that accept a computer-readable (also referred to as machine-readable) code containing a set of instructions that, when executed by one or more processors, perform at least one method described herein. Any processor that is capable of executing (sequentially or otherwise) a set of instructions that specifies an action to be taken is included. Thus, an example is a typical processing system (e.g., computer hardware) that includes one or more processors. Each processor may include one or more of a CPU, a graphics processing unit, and a programmable DSP unit. The processing system may further include a memory subsystem that includes a hard drive, an SSD, a RAM, and / or a ROM. A bus subsystem may be included for communication between components. The software may reside in the memory subsystem and / or the processor during its execution by the computer system.

[0030] The one or more processors may operate as standalone devices or may be connected to other processor(s), such as via a network. Such a network may be constructed on a variety of different network protocols and may be the Internet, a wide area network (WAN), a local area network (LAN), or any combination thereof.

[0031] The software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or transient medium). As is well known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include, but are not limited to, various forms of physical (non-transitory) storage media, such as ROM, PROM, EPROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage devices, cassettes, tapes, disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Further, it is well known to those skilled in the art that (transitory) communication media typically embodies computer-readable instructions, data structures, program modules or other data in the form of modulated data signals or other transmission mechanisms such as carrier waves, and includes any information transfer medium.

[0032] The present disclosure assumes the use of an immersive audio codec (such as the IVAS codec) in some extended reality XR applications. The main decoding and pre-rendering can be done by a first device (user equipment UE) or an edge or other network node of a hypothetical 5G system. The second device contains a post-decoder and a (lightweight) post-renderer. In this way, the overall operation can be divided into operations of multiple devices. The first device (main device) can be a mobile device such as a laptop, tablet or smartphone, or a fixed device such as a workstation or server. The first device can also be a combination of several processing devices. The second device can be a device held (e.g., worn) by a user, such as a pair of augmented reality AR glasses.

[0033] A fundamental assumption and creative insight applied in the field of disaggregated rendering is that, for each time-frequency tile, the audio consists of a main directional component and a diffuse (omnidirectional) component. The directional component is assumed to be a prototype signal S arriving from a certain direction of arrival (DOA), while the diffuse component is a decorrelated version of this prototype signal. This concept has proven to be very powerful in spatial audio coding methods such as DirAC or metadata-assisted spatial audio (MASA) coding.

[0034] Based at least on these assumptions, various example implementations may include the following steps:

[0035] 1. The pre-render binauralizes the decoded immersive audio using a set of generic HRTFs (or BRIRs) given a head pose P', which is either transmitted from a lightweight device equipped with a head tracker, or may just be preset values ​​that do not necessarily correspond to any actual head pose of the user, but may be reasonable default values ​​such as a head pose looking straight ahead. Applying HRTFs during the binaural pre-rendering operation can be done by specifically selecting an HRTF for each time / frequency tile. The selection of the HRTF is based on the direction of arrival (DOA) of the main components of the immersive audio content relative to the assumed head pose.

[0036] 2. The first or master device encodes and transmits the binauralized audio channels, an indication of the HRTFs and / or DOA angles used, and an assumed head pose P'.

[0037] 3. The post-renderer of the second device aims to adjust the received left and right binaural signals relative to the current head pose P (if deviating from P').

[0038] 4. Given the HRTF or DOA angles and head pose applied by the pre-renderer, the second device computes left and right HRTF compensated signals essentially by inverse HRTF filtering the left and right audio channels and optionally linearly combining them.

[0039] 5. The HRTF compensated signal is filtered by the second device using the correct HRTF corresponding to the correct head posture.

[0040] 6. By appropriately choosing the weights of the linear combination, potential errors in the diffusion component can be mitigated.

[0041] The main idea can also be applied to HRTF personalization, where the pre-render uses generic HRTFs and the post-render compensates for these generic HRTFs and then applies the personalized HRTFs.

[0042] In the following, reference Figures 1 to 3 To illustrate example implementations of the novel concepts described herein.

[0043] Figure 1 1 is a block diagram of an example system implementing head tracking split rendering arranged according to various aspects of the present invention. The example system includes a first device 10 and a second device 20. The first device 10 may also be referred to as a master device, and the second device 20 may also be referred to as a mobile device or a user-held device.

[0044] exist Figure 1 In the embodiment of the present invention, a first device or master device 10 comprises a decoder / renderer 11, an optional head pose decoder 12, encoders 13 and 14, and a multiplexer 15. The decoder / renderer 11 (e.g., an IVAS decoder) receives (step S1) a master bitstream b comprising encoded immersive audio content. 1 , decode the immersive audio content (step S2), and perform binaural rendering on the decoded audio content using the HRTF associated with the direction of arrival DOA relative to the user's assumed head posture P' (step S3). The HRTF used is usually taken from a set of Generic HRTF (for various directions of arrival DOA). Such processing is generally too computationally complex to be performed in a mobile device or a (lightweight) device held by the user. Assume that the head pose P' may be either a suitable default head pose or an actual user head pose received from a device held by the user, which may optionally be decoded by a head pose decoder 12 (step S21). This decoded user head pose may represent the user's most recent but not completely current head pose. The renderer 11 outputs a binaural signal L 1 , R 1 and post-rendering metadata M. The post-rendering metadata M comprises an indication of the HRTF used, for example expressed as a direction of arrival DOA of the main directional component of the immersive audio content (the DOA is relative to the assumed head pose P') or as an index of the HRTF used. The post-rendering metadata M may also comprise an indication of the head pose P' associated with binaural rendering. The encoders 13, 14 are arranged to convert the binaural signal L1 , R 1 and post-rendering metadata M are respectively encoded (step S4) into encoded signals b 11 and b 12 The multiplexer 15 is arranged to convert the encoded binaural signal b 11 and the encoded metadata b 12 Multiplex or combine (step S5) into an intermediate bit stream b 2 , the intermediate bit stream is transmitted (step S6) to the second device 20.

[0045] The second device 20 (which may be a device 20 held by a user) includes a demultiplexer 21, decoders 22 and 23, an encoder 25, a renderer 26, and a head tracker 24. The device 20 held by the user receives (step S11) an intermediate bit stream b 2 , and the demultiplexer 21 converts the intermediate bit stream b 2 Separated into coded signal b 21 and b 22 The coded signal is received by the corresponding decoders 22 and 23. The decoders 22 and 23 responsively process the coded signal b 21 and b 22 Decode (step S12) to obtain the decoded binaural signal L 2 , R 2 and decoded metadata M'. As described above, the metadata M' comprises an indication of the HRTF used, for example by an index or a direction of arrival relative to a head pose looking straight ahead.

[0046] The head tracker 124 (which may be included in the device 120 held by the user or may also be connected to the device) detects (step S13) the current head pose P of the user's head. The encoder 25 may optionally be used to encode (step S131) ​​the detected head pose P as b P , and the encoded detected head pose b P Transmitted to the master device 10.

[0047] The metadata M' may also include an assumed head pose P' for use in the renderer 11. Alternatively, in an embodiment where the detected head pose P is transmitted to the main device 10, the device held by the user may estimate the assumed head pose based on an expected transmission delay. In principle, the assumed head pose may be assumed to be a head pose detected at a point in time corresponding to the expected transmission delay.

[0048] Finally, the renderer 26 receives the decoded binaural audio signal L 2 , R 2, DOA or HRTF used, assumed head posture P' and current head posture P, and calculate the output binaural signal L out , R out The processing includes: identifying (step S14) a post-rendering HRTF corresponding to the detected current head posture P; calculating a compensated stereo audio signal by applying an HRTF compensation operation to the binaural audio signal (step S15), wherein the HRTF compensation operation is configured to compensate for the effects of the pre-rendered HRTF; and finally applying (step S16) the identified post-rendering HRTF. For such processing, which will be discussed below, the renderer 26 is provided with HRTF data, which is typically a set of Generic HRTF (for various directions of arrival DOA). A set of Personalized HRTF.

[0049] Figure 2 is a flowchart showing the processing in the first device (or master device), which includes the above-mentioned steps S1 to S6 and the optional step S21.

[0050] The process comprises: in step S1 "receiving bitstream", receiving a bitstream; and in step S2 "decoding", decoding the bitstream by a decoder to obtain decoded immersive audio content. In step S21 "decoding posture", the process may include the optional steps of receiving and decoding an indication of a current user head posture from a device held by a second user, and determining a hypothetical user head posture based on the current user head posture. In step S3 "pre-rendering", the process involves binauralizing the immersive audio content by a pre-renderer to generate a pre-rendered binaural signal, the binauralizing using a pre-rendered HRTF from a set of HRTFs and a hypothetical head posture of the user. In step S4 "encoding", the process involves encoding the pre-rendered binaural signal and encoding post-rendering metadata, the metadata indicating the pre-rendered HRTF. In step S5 "combining", the process involves combining the encoded binaural audio signal and the encoded post-rendering metadata in a multiplexer to form a bitstream including a binaural audio representation. In step S6 "Transmit", the process involves transmitting the bitstream to the second device (or device held by the user).

[0051] Figure 3 is a flowchart showing the processing in the second device (or the device held by the user), which includes the above-mentioned steps S11 to S16 and the optional step S131.

[0052] The process includes receiving a bitstream including a binaural pre-rendered representation of immersive audio content from a first device (or master device) in step S11 "receiving a bitstream". The binaural pre-rendering has been obtained relative to a hypothetical head pose P'. In step S12 "decoding", the process involves decoding the bitstream to obtain a binaural audio signal and associated post-rendering metadata. The metadata indicates the pre-rendered HRTF used in the binaural pre-rendering, wherein the pre-rendered HRTF is associated with the hypothetical head pose P'. In step S13 "detecting a current pose", the process involves obtaining user head pose information indicating a current head pose P. In step 131 "encoding a pose", the process may include the optional step of encoding the detected head pose P using an encoder and transmitting an indication of the current head pose P to the master device. As described above, the master device may then use the current pose received from the device held by the second user as the hypothetical pose. Therefore, in this case, the device held by the second user can estimate the hypothetical head pose P' based on the expected transmission delay (and the previously transmitted current pose). In step S14 "Identify post-rendered HRTF", the process involves identifying a post-rendered HRTF based on the metadata, the assumed head pose P' and the current head pose P. In step S15 "Calculate compensated audio", the process involves calculating a compensated stereo audio signal by applying an HRTF compensation operation to the binaural audio signal, the HRTF compensation operation being configured to compensate for the effects of the pre-rendered HRTF.

[0053] Various example HRTF compensation operations applicable to calculating compensated stereo audio signals are described herein. These operations may involve any number of methods that appropriately offset and adjust the various effects produced by the pre-rendered HRTF operations. In some examples, HRTF compensation may involve inverse mathematical operations of pre-rendered HRTFs. In some other examples, HRTF compensation may be implemented by lookup table type operations, wherein, in order to reduce memory requirements, lightweight devices may rely on a less dense set of HRTFs than those used in pre-rendered devices. Therefore, a set of inverse HRTFs available to lightweight devices may include appropriate approximations of the inverse of the HRTFs available in the HRTF set at the pre-rendered device. In other examples, HRTF compensation may be implemented by numerical approximation methods, which may include linear or nonlinear interpolation or a combination thereof. In yet other examples, HRTF compensation may be implemented by best fit type approximations. Combinations of various methods are equally applicable and are considered to be within the scope of this disclosure.

[0054] Back to Figure 3In step S16 "Apply post-rendering HRTF", the process involves calculating the binaural output signal by applying the post-rendering HRTF to the compensated stereo signal. Steps S15 and S16 may be performed as a single operation.

[0055] Figure 2 and Figure 3 The illustrated process includes a set of blocks that represent a series of operations or steps that can be implemented in hardware, software, or a combination thereof. In the context of software, a block can represent a computer-executable instruction that, when executed by one or more processors, performs the operations. Typically, computer-executable instructions can include routines, programs, objects, components, data structures, etc. that perform or implement functions. The order in which these operations are described is not intended to be construed as limiting, and any number of the described blocks can be combined in any order, separated into additional blocks, and / or operated in parallel to implement the process.

[0056] Method at prerender 11:

[0057] The basic assumption is that, for each time-frequency tile, the audio consists of a main directional component and a diffuse (omnidirectional) component. The directional component is assumed to be a prototype signal S arriving from some DOA with azimuth and elevation angles α expressed in some room coordinate system. 0 , γ 0 . The diffuse component is a decorrelated version of the prototype signal S.

[0058] Here, the pre-renderer synthesis is done by convolving the directional component (per time-frequency tile) with the HRTF corresponding to the DOA and adding the diffuse component. Both components are added with the corresponding weight r dir and r diff :

[0059] L 1 =r dir S*h L (α′,γ′)+r diff Ψ L (S), R 1 =r dir S*h R (α′,γ′)+r diff Ψ R (S),

[0060] Among them Ψ L and R is a decorrelator.

[0061] Here, α′, γ′ are the azimuth and elevation of the directional components relative to the head pose P′ assumed at the pre-renderer 11. If the head pose P′ is expressed in the same room coordinate system, then, at least under some further restrictive assumptions, α′=α 0 -α P′ , γ′=γ 0 -γ P′ .

[0062] Method at the post-renderer:

[0063] The post-renderer aims to adjust the received left and right binaural signals L relative to the current head pose P (if it deviates from P'). 2 and R 2 .

[0064] An important insight is that the correct output signal will be

[0065]

[0066] signal S and decorrelator signal Ψ L and R Not available. Alternatively, and Will use the available signal L 2 and R 2 Approximated by parametric methods. Assuming that the HRTF applied by the pre-renderer is known, the left and right HRTF compensation signals can be calculated. One possibility to obtain these signals is to derive them as a weighted combination of the HRTF-compensated left and right channel signals. Therefore, k L and k R is a suitable weighting factor or operator.

[0067] The left and right channel signals after HRTF compensation are respectively and

[0068] Then, the left and right HRTF compensation signals are

[0069] as well as

[0070]

[0071] In a simple example, κ L =κ R = 1 (no weight), so

[0072] and

[0073]

[0074] Using these signals, the left and right output signals of the post-renderer are obtained as follows:

[0075] L out =h L (α, γ)*L C and R out =h R (α, γ)*R C .

[0076] This method can correct the directional component in the output signal relative to the current head posture, i.e., r dir S*h L (α, γ) and r dir S*h R (α, γ).

[0077] However, there is an error in the diffusion component, which can be quantified as follows:

[0078]

[0079] as well as

[0080]

[0081] Assuming that HRTFs may be decomposed into delay and gain / shape operations, the delay changes involved in the decorrelated diffusion component may be perceptually insignificant, while gain / shape changes may result in timbre deviation or color effects. Given the set of HRTFs involved, the k L , k R to mitigate this error. In a more general form, κ L , κ R These can be linear and non-linear operators, such as (frequency selective) filter operators or gain limiters to avoid that the output samples exceed a predefined digital range.

[0082] Adaptively choose the weight κ L , κ R The benefit of is illustrated by considering the example of the case where the head pose change is limited to yaw, i.e., the rotation occurs only about the z-axis, so the directional components are equal with respect to the elevation angles of the assumed head pose and the current head pose (γ=γ′).

[0083] First consider the case where there is almost no yaw rotation. Therefore, α is approximately equal to α′. In this case, it is preferred to use κ L =κ R = 1 to select the weight, which means that the received left and right binaural signals L 2 and R 2 output with little modification, and ΔL diff≈ΔR diff ≈ 0. Therefore, for (sufficiently) small yaw deviations (e.g., less than 20 degrees), the weight is set to κ L =κ R =1 is a good choice.

[0084] Next, consider the case where the yaw change is 180 degrees, which results in α being approximately equal to α′±180 degrees. Now, preferably according to κ L =κ R = 0 to select the weight, which means that the received left and right binaural signals L 2 and R 2 As part of the adjustment process, they are (effectively) swapped. This results in the following diffuse components for the left and right output channels:

[0085] as well as

[0086]

[0087] Considering that the right ear HRTF can be approximated by the left ear HRTF with an azimuth shift of 180 degrees, and similarly, the left ear HRTF can be approximated by the right ear HRTF with an azimuth shift of 180 degrees, it can be found that in the above equation, the term and can be approximated to 1. This results in the following approximation for the diffuse components of the left and right output channels:

[0088] L out，diff ≈r diff *Ψ R (S), and

[0089] R out，diff ≈r diff *Ψ L (S).

[0090] It can be concluded that if the yaw difference between the current head pose and the hypothesized head pose is close to 180 degrees, the weight is set to κ L =κ R =0 is a good choice.

[0091] The third case considered is a yaw rotation of 90 degrees, which results in α being equal to α′ ± 90 degrees. Now, it can be said that there is no reason to give priority to the available signal L when constructing the HRTF compensation signal. 2 or R 2 This is because this may result in asymmetric behavior between the left and right channels of the solution. In the case of a yaw rotation close to 90 degrees, when choosing κ L =κ R When =0.5, symmetrical behavior can be achieved.

[0092] This discussion leads to the conclusion that in response to determining the yaw rotation Δ yaw The weight κ is adaptively selected based on the direction component relative to the difference between the azimuth of the assumed head pose P′ and the current head pose P. L , κ R The preferred solution:

[0093]

[0094] Note that similar embodiments can be formulated based on the roll angle of the hypothesized head pose and the current head pose.

[0095] Figure 4 An example technique for DOA-based separation rendering using pre-rendered binaural signals is shown. The figure shows how the sound wave front is assumed to be from the azimuth angle α ′ The DOA of α and α arrives at the listener's head 30. The pre-renderer only has access to the head pose P' at angle α'. Therefore, binaural synthesis is done using the HRTF corresponding to the head pose P' at angle α'. One major effect is that the interaural time difference (ITD) of the wavefront between the left and right ears is Δ' LR . There is also a corresponding interaural level difference (ILD) and spectral difference. The applied HRTF simulates this effect by imposing / imprinting the appropriate ITD, ILD and spectrum. The figure further shows the hypothetical situation in the post-renderer, which knows the actual head pose P with angle α. The figure shows how the hypothetical wavefront arrives from a different DOA relative to the actual head pose P (angle α, not α′) - which in turn produces a different ITDΔ LR It is worth noting that there are other ILDs and spectra corresponding to actual head poses. Figure 4 A main idea of ​​the present disclosure visualized in FIG. 1 is to compensate Δ′ by first LR Then apply Δ LR To change ITD from Δ′ LR becomes Δ LR Similarly, the ILD and spectrum are modified by compensating the ILD and spectrum corresponding to the head posture P′ and applying the ILD and spectrum of the HRTF corresponding to the actual head posture P.

[0096] The above description leads to the following simplified method (see also Figure 4 ):

[0097] Assumptions:

[0098] The rear-front axis A of the listener 30 defines the x-axis of the right-handed coordinate system. In addition, in many relevant situations, the user may move his head mainly around the yaw axis (z-axis), and the sound sources of most immersive audio content are close to the horizontal plane. Therefore, the elevation angle of the DOA is relatively close to zero degrees (e.g., in the interval of [-20, 20] degrees).

[0099] Under these assumptions, according to the simplified formula, only the azimuth component of the DOA causes significant ITD, which further affects the gain and spectral shape.

[0100] The bounded elevation components of DOA (pitch, roll) affect the gain and spectral shape but not the ITD.

[0101] The HRTF filter can be decomposed into a delay operation and a gain / shape operation:

[0102] h L (α′, γ′) = δ L (α′)*gs L (α′, γ′), h R (α′, γ′) = δ R (α′)*gs R (α′, γ′).

[0103] The listener's head pose P' is assumed during pre-rendering. The pre-renderer renders at the azimuth component α' of the DOA during playback, which deviates from the true azimuth α.

[0104] The post-renderer renders at the azimuth component α of the DOA corresponding to the listener's head pose P during playback.

[0105] The interaural time difference (ITD) between the pre-rendered signal and the signal adjusted by the post-renderer is calculated as follows:

[0106]

[0107] where d e : interaural distance, and c: speed of sound.

[0108] Therefore, the post-renderer should reduce the ITD of the directional component in a given time-frequency tile from Δ′ LR Adjust to Δ LR .

[0109] In addition to the ITD adjustment, the post-renderer also adjusts the interaural level difference and spectral shape based on the comparison of the real head pose with the pre-renderer's assumed head pose.

[0110] It is worth noting that similar formulas can be obtained even if a bounded elevation component of the DOA is not assumed. Even in this case, the operation of the post-renderer can be decomposed into ITD adjustment, interaural level difference, and spectral shape adjustment. However, in this case, the amount of ITD adjustment required will depend on the azimuth and elevation of the DOA assumed during pre-rendering, and the effective values ​​during post-rendering.

[0111] Figure 5 An example technique for HRTF personalization is shown. The figure shows how a hypothetical wavefront arriving at the listener's head 30 from a DOA angle α produces different ITDs depending on the size of the listener's head. Pre-rendering using a generic HRTF can assume that the listener's head size has a generic interaural distance This will result in The corresponding generic ITD, the corresponding ILD and the spectral shape of the left and right audio signals. The personalized HRTF will be based on the (more) correct listener head size. Therefore, this will produce a more correct, personalized ITD As well as more correct corresponding ILD and spectral shapes of left and right audio signals. The overall idea of ​​HRTF personalization is that the post-renderer will compensate for the generic HRTF and apply the effect of the personalized HRTF. The overall concept is very similar to the head pose correction at the post-renderer mentioned above. Therefore, the two concepts are compatible with each other and can be easily combined.

[0112] For the host device 10 with the pre-render 11, the same description as above is applicable. However, the pre-render 11 relies on a set of generic HRTFs Rendering is performed, and the post-renderer 26 in the device 20 held by the user uses a set of personalized HRTF Make adjustments so the post-renderer knows the common HRTF used by the pre-renderer.

[0113] The post-renderer 26 aims to generate a 3D image with respect to the current head pose P (if it deviates from P') and with respect to a set of personalized HRTFs Adjust the received left and right binaural signals L 2 and R 2 .

[0114] The correct output signal will be

[0115]

[0116] signal S and decorrelator signal Ψ L and R Not available. Alternatively, and Will use the available signal L 2 and R 2Approximated by a parametric method. Assuming that the HRTF applied by the pre-renderer is known, the left and right HRTF compensation signals are calculated as the linear combination of the left and right channel signals after HRTF compensation:

[0117] as well as

[0118]

[0119] Using these signals, the left and right output signals of the post-renderer are obtained as follows:

[0120] L out =h p，L (α, γ)*L C and R out =h p，R (α, γ)*R C .

[0121] This method can correct the directional component in the output signal relative to the actual head pose and the personalized HRTF, i.e., r dir S*h p，L (α, γ) and r dir S*h p，R (α, γ).

[0122] However, there is an error in the diffuse component, which can be quantified as follows:

[0123]

[0124] as well as

[0125]

[0126] Assuming that HRTFs can be decomposed into delay and gain / shape operations, the delay changes involved in the decorrelated diffusion component may be perceptually insignificant, while gain / shape changes may result in timbre deviation or color effects. Given a set of HRTFs involved, the delay of the decorrelated diffusion component can be reduced by an appropriate choice of κ L , κ R The embodiment of adaptively selecting weights in response to yaw and / or roll deviations between the current head pose and the assumed head pose is still fully applicable.

[0127] Specific aspects of the embodiments

[0128] The post-renderer receives the direction of arrival (DOA) information. This DOA information can be expressed as the azimuth and elevation (DOA angles) α′, γ′ of the main directional components of the immersive audio content relative to the assumed head pose P'. It should be noted that the DOA is determined per time-frequency tile. The index of the HRTF used is another form for providing DOA information to the post-renderer.

[0129] Furthermore, the post-renderer must be aware of the head pose P′ assumed at the pre-renderer. The corresponding information can be transmitted to the post-renderer (i.e. in metadata). It is also possible to rely on the fact that P′ corresponds to the real head pose at an earlier time instant, which has been transmitted from the post-renderer to the pre-renderer. Assuming that the transmission delay from the post-renderer to the pre-render is known a priori or can be estimated, this will make the transmission of P′ to the post-renderer unnecessary. One way to estimate the transmission delay from the post-renderer to the pre-render is based on measuring the round-trip delay from the post-renderer to the pre-render and back to the post-renderer (e.g. using timestamps).

[0130] Parameter r dir 、r diff , κ L , κ R are mathematically related to each other. This interdependence can be exploited, for example, to help find the right κ L , κ R This avoids the need for a decorrelator in the post-renderer. The benefit of this approach is that it avoids the complexity of the post-renderer.

[0131] If the decorrelator is to be used in a post-renderer, a suitable decorrelator input signal is

[0132]

[0133] It compensates / cancels the directional component because it produces:

[0134]

[0135] Preferably, the pre-rendered binaural channel signal L 1 , R 1 Transmitting in the complex-valued quadrature mirror filterbank (CQMF) / frequency domain will avoid performing a forward operation from time to CQMF / frequency domain in the post-renderer, which is advantageous in terms of complexity and latency.

[0136] A significant difference between the present method and conventional techniques is that the present method relies on compensation of the HRTF filter operation of the pre-renderer and applies the HRTF used in the ideal case. In contrast, alternative techniques rely on transforming the binaural output channels using a linear transform whose coefficients are obtained by LMS method and interpolation.

[0137] An example implementation of DOA-based split rendering using a prototype signal includes the following steps:

[0138] 1. The pre-renderer or decoder generates a prototype signal (S). Some example methods for generating S are as follows:

[0139] a. Take the high-fidelity stereo W or omni channel representation from the decoder output using any known technique and use that as S.

[0140] b. Get the representation of the main feature signal from the decoder output and use it as S.

[0141] c. Use a set of common HRTFs (or BRIRs) to pre-render the decoded immersive audio to generate S=aL+bR; where L and R are the left and right channels of the pre-rendered bin signal, a and b are complex or purely real gain factors for each time-frequency tile and can be dynamically calculated values ​​or statically predetermined values ​​(e.g., a=0.5 and b=0.5).

[0142] 2. The master device transmits the encoded prototype signal S, the assumed head pose P' and / or the assumed DOA angle (or equivalent information) and the diffusion parameters.

[0143] 3. The post-renderer decodes the prototype signal bits and generates S' (if the codec used to encode S has zero latency and is lossless, then S' should be the same as S).

[0144] 4. The post-renderer aims to generate left and right binaural signals relative to the current head pose P (if deviated from P').

[0145] 4. The post-renderer adjusts the DOA angle sent by the master device based on the difference between P and P'. Together with S', the HRTF at the post-renderer and the adjusted DOA angle, the post-renderer generates the directional components of the post-rendered binaural signal. The diffuseness parameter is used with the decorrelated S' to fill the diffuse energy in the post-rendered binaural signal.

[0146] Method at the post-renderer:

[0147] The post-renderer aims to adjust the received DOA relative to the current head pose P (if it deviates from P'), and then compares it with the prototype signal S' and a set of HRTFs The post-renderer generates the head-tracking binaural signal as follows.

[0148] α=α′-(α P -α P′ ), γ=γ′-(γ P -γ P′ ).

[0149]

[0150] r diff is the diffusion parameter sent by the master device, and r dir is the directional gain that can be calculated using DOA and spherical harmonics.

[0151] Advantages of this approach:

[0152] - Low bitrate mode is achieved by only encoding the S channel and transferring it to the post-renderer.

[0153] - No HRTF compensation is required and the error in diffusion compensation can be reduced to 0.

[0154] Disadvantages of this approach:

[0155] - For the potential situation where the device held by the user only outputs the decoded binaural audio signal without any further processing or post-rendering operation, there is no pre-rendered binaural audio signal readily available on the device held by the user.

[0156] Figure 6 An example technique for DOA-based split rendering using a prototype signal is shown.

[0157] exist Figure 6 In the embodiment, the master device 110 includes a decoder / renderer 111, a head pose decoder 112, encoders 113, 114, and a multiplexer 115. The decoder / renderer 111 (eg, an IVAS decoder) receives a master bitstream b 1 And perform rendering synthesis of a prototype signal S, which has an arrival direction DOA relative to an assumed head posture P' of the user. The assumed head posture P' can be an appropriate default head posture, or it can be an actual user head posture received from a device held by the user, which can be optionally decoded by a head posture decoder 112. This decoded user head posture will represent the user's most recent but not completely current head posture. Here, the renderer 111 outputs the prototype signal S and metadata M, which includes at least the arrival direction DOA of the prototype signal. The encoders 113, 114 encode the prototype signal S and the metadata M, and the multiplexer 115 converts the encoded prototype signal S into a multiplexer. 11and the encoded metadata b 12 Multiplexed into an intermediate bit stream b 2 .

[0158] The device 120 held by the user includes a demultiplexer 121, decoders 122, 123, a head tracker 124, an encoder 125, and a post-renderer 126. The demultiplexer 121 receives the intermediate bit stream and separates it into two encoded signals b 21 and b 22 , and the two decoders 122, 123 responsively decode these signals to obtain a decoded prototype signal S' and decoded metadata M', for example, the DOA and (optionally) the assumed head pose P' used in the renderer 111. A head tracker 124 (which may be included in or connected to the device 120 held by the user) detects the current head pose P of the user's head. The encoder 125 encodes the detected head pose P and transmits it to the main device 110. Finally, the post-renderer 126 receives the decoded prototype signal S', the DOA, the assumed head pose P' and the current head pose P, and calculates the output binaural signal L out , R out For this processing, which will be discussed below, the post-renderer 126 is provided with HRTF data, typically a set of Generic HRTF (for various directions of arrival DOA). A set of Personalized HRTF.

[0159] Example use cases that can benefit from disaggregated rendering of immersive audio

[0160] The techniques described in this specification can be implemented in various use cases. It is assumed that the main audio processing / audio signal enhancement and subsequent pre-rendering are completed on some powerful devices or network nodes, while the post-rendering is completed on lightweight terminal devices (such as AR glasses, etc.).

[0161] Some examples are provided below.

[0162] 1. AR / MR involving audio

[0163] Audio Zoom / Amplifier: Similar to a magnifying glass, but for sound. Users can zoom in on sounds of interest.

[0164] Overlay of real world objects and sounds: Real world objects / items will be associated with sounds. Useful for assistive systems for the visually impaired, but not limited to this.

[0165] Conversation Enhancement / Intelligent Ambient Noise Reduction: Helps people with cocktail party issues by letting active voices drown out ambient noise.

[0166] Emotional Sound Ambience: Like mood lights. Sounds will be associated with real-world environments, objects, and personal preferences.

[0167] 2. Use case characteristics

[0168] These use cases will typically rely on audio / visual capture, some scene analysis, and generation of an augmented sound signal. In some cases, this can also be overlaid with immersive sound from a network node or the far end of the communication.

[0169] The use case will typically rely on audio / visual rendering with head tracking.

[0170] 3. Other non-AR / MR use cases

[0171] Immersive voice communication (two-party conferencing) and immersive content streaming with AR glasses as the terminal device may be IVAS use cases.

[0172] Some of these use cases may rely on head-tracked audio rendering, some may not.

[0173] Some use cases may involve one-to-many immersive distribution of head-tracked audio.

[0174] Aspects of the systems described herein can be implemented in a suitable computer-based sound processing network environment to process digital or digitized audio files. Portions of the adaptive audio system may include one or more networks comprising any desired number of independent machines including one or more routers (not shown) for buffering and routing data transmitted between computers. Such networks may be constructed on a variety of different network protocols and may be the Internet, a wide area network (WAN), a local area network (LAN), or any combination thereof.

[0175] One or more components, blocks, processes or other functional components may be implemented by a computer program executed by a processor-based computing device that controls the system. It should also be noted that any number of combinations of hardware, firmware and / or data and / or instructions embodied in various machine-readable or computer-readable media may be used to describe the various functions disclosed herein from the perspective of their behavior, register transfers, logic components and / or other characteristics. Computer-readable media that may embody such formatted data and / or instructions include, but are not limited to, various forms of physical (non-transitory), non-volatile storage media (such as optical, magnetic or semiconductor storage media).

[0176] Although one or more embodiments have been described by way of example and with respect to specific embodiments, it should be understood that one or more embodiments are not limited to the disclosed embodiments. On the contrary, it is intended to cover various modifications and similar arrangements that are obvious to those skilled in the art. Therefore, the scope of the appended claims should be given the broadest interpretation so as to cover all such modifications and similar arrangements.

[0177] Further details and embodiments of the present invention may be understood from the following enumerated exemplary embodiment EEE:

[0178] EEE1. A method for processing audio, the method comprising:

[0179] receiving, by a first device, a primary bitstream representation of encoded audio;

[0180] Obtaining user head posture information by the second device;

[0181] Determining, by the first device, a downmix signal including at least one channel and metadata from the main bitstream;

[0182] Providing, by the first device, the downmix signal and the metadata to a second device;

[0183] The downmix signal is rendered into output binaural audio by a lightweight renderer of the second device based on the metadata and the user head posture information.

[0184] EEE2. The method as described in EEE1, wherein the downmix signal includes a pre-rendered binaural signal.

[0185] EEE3. The method as described in EEE2, wherein determining the pre-rendered binaural signal and rendering metadata comprises:

[0186] decoding, by a primary renderer of the first device, the primary bitstream representation to generate decoded audio;

[0187] The decoded audio is binauralized by a prerender of the first device to generate the prerendered binaural signal and rendering metadata, wherein the prerender performs the binauralization using at least one of the following:

[0188] Generic Head Related Transfer Function (HRTF) or Binaural Room Impulse Response (BRIR), or

[0189] The user head posture information, the user head posture information, the user information is obtained from at least one of the following:

[0190] a head tracker of the second device,

[0191] A storage device storing preset values, or

[0192] Assume a direction of arrival (DOA) angle.

[0193] EEE4. The method as described in EEE3, wherein the metadata includes at least one of the following:

[0194] An indication of the HRTF or BRIR used by the pre-render,

[0195] the assumed user head pose used by the pre-render, or

[0196] The assumed DOA angle used by the pre-render.

[0197] EEE5. A method as described in EEE 4, wherein rendering the pre-rendered binaural signal as output binaural audio includes: the lightweight renderer adjusts the left channel and the right channel of the pre-rendered binaural signal based on an assumed user head posture used by the pre-render, relative to the current user head posture obtained by the head tracker.

[0198] EEE6. A method as described in any one of EEE2-5, wherein rendering the pre-rendered binaural signal comprises:

[0199] According to the HRTF or assumed DOA angle used by the pre-renderer, the left channel and the right channel of the pre-rendered binaural signal are subjected to inverse HRTF filtering; and

[0200] The inverse HRTF filtered signals are linearly combined.

[0201] EEE7. A method as described in EEE6, wherein the inverse HRTF filtering includes using the current user head posture obtained by the head tracker of the second device to correct the HRTF used by the pre-render.

[0202] EEE8. A method as described in EEE6 or 7, wherein linearly combining the inverse HRTF filtered signals includes mitigating errors in the diffuse component by selecting weights of the linear combination.

[0203] EEE9. The method of any one of claims 2 to 8, comprising applying HRTF personalization, wherein the pre-render applies a generic HRTF, and the lightweight renderer compensates for the generic HRTF and then applies a personalized HRTF.

[0204] EEE10. The method as described in EEE1, wherein the downmix signal includes a prototype signal.

[0205] EEE11. The method as described in EEE10, wherein the prototype signal includes a single channel.

[0206] EEE12. The method as described in EEE10 or 11, wherein calculating the prototype signal comprises:

[0207] decoding, by a primary decoder of the first device, the primary bitstream representation to generate decoded audio; and

[0208] A gain is applied to the decoded audio, and the decoded audio to which the gain is applied is added to the decoded audio.

[0209] EEE13. The method as described in any one of EEE10-12, comprising:

[0210] Calculating the prototype signal and the DOA angle based on the assumed head posture P' and the diffusion parameter;

[0211] sending the hypothesized head pose P', the DOA angle of the hypothesized head pose P', the diffusion parameter and the prototype signal to a post-renderer device;

[0212] adjusting the DOA angle based on an actual head pose P at the post-renderer device;

[0213] Calculating the directional component using the prototype signal and a set of HRTFs and the adjusted DOA angle;

[0214] calculating a diffuse component using the diffuseness parameter and a decorrelated version of the prototype signal; and

[0215] The directional and diffuse components are added to produce the rendered binaural output.

[0216] EEE14. A method as described in any one of EEE1-13, wherein the first device includes a smart phone, the second device includes a wearable audio device, a visual device, or an AR device, and the main bitstream includes an immersive audio and video service (IVAS) bitstream.

[0217] EEE15. A system comprising one or more processors configured to perform the operations described in any one of EEE1-14.

[0218] EEE16. A computer program product configured to cause one or more processors to perform the operations described in any one of EEE1-14.

Claims

1. A method for processing audio in a processing device held by a user, the method comprising: Receiving a bitstream from a host device, the bitstream comprising a binaural pre-rendered representation of immersive audio content, wherein the binaural pre-rendering is obtained relative to a hypothesized head pose P'; decoding the bitstream to obtain a binaural audio signal and associated post-rendering metadata, the metadata indicating a pre-rendered HRTF used in the binaural pre-rendering, wherein the pre-rendered HRTF is associated with the hypothesized head pose P'; Obtaining user head posture information indicating a current head posture P; identifying a post-rendering HRTF based on the metadata, the hypothesized head pose P' and the current head pose P; calculating a compensated stereo audio signal by applying a HRTF compensation operation to the binaural audio signal, the HRTF compensation operation being configured to compensate for an effect of the pre-rendered HRTF; and A binaural output signal is calculated by applying the post-rendering HRTF to the compensated stereo signal.

2. The method of claim 1, wherein: The steps of calculating the compensated stereo audio signal and calculating the binaural output signal are performed in a single operation.

3. The method according to claim 1 or 2, wherein: The HRTF compensation operation involves the inverse of the pre-rendered HRTF.

4. The method of claim 3, wherein: The inverse of the pre-rendered HRTF is obtained by accessing a lookup table in a table including approximations of the inverse of the pre-rendered HRTF.

5. The method according to claim 3 or 4, wherein: The step of calculating the compensated stereo audio signal comprises: An inverse left HRTF is applied to the left channel of the binaural audio signal to form an inverse filtered left channel, an inverse right HRTF is applied to the right channel of the binaural audio signal to form an inverse filtered right channel, and the inverse filtered left channel and the inverse filtered right channel are combined to form the left channel and the right channel of the compensated stereo audio signal, respectively.

6. The method of claim 5, wherein: The inverse filtered left channel and the inverse filtered right channel are linearly combined, and a weight of the linear combination is selected to mitigate errors in diffuse components of the binaural output signal.

7. The method of claim 6, wherein: The weights are adaptive, preferably responsive to the difference between the hypothesized head pose P' and the current head pose P.

8. A method as claimed in any one of the preceding claims, wherein: The post-rendered HRTF is personalized to a user of a processing device held by the user.

9. A method as claimed in any one of the preceding claims, wherein: The post-rendering metadata further comprises an indication of the hypothesized head pose P'.

10. The method according to any one of claims 1 to 9, wherein: The method further includes transmitting an indication of the current head pose P to the master device and estimating the hypothesized head pose P' based on an expected transmission delay.

11. A method as claimed in any one of the preceding claims, wherein: The processing device held by the user includes a wearable audio device, a visual device or an AR device.

12. A method as claimed in any one of the preceding claims, wherein: The bitstream comprises an immersive audio and video services (IVAS) bitstream.

13. A method as claimed in any one of the preceding claims, wherein: The metadata includes a direction of arrival (DOA) associated with a dominant directional component of the immersive audio content, the DOA being indicative of the pre-rendered HRTF.

14. A method for processing audio, the method comprising: Receive a bit stream; The decoder decodes the bit stream to obtain decoded immersive audio content; performing binauralization on the immersive audio content by a pre-renderer to generate a pre-rendered binaural signal, the binauralization being performed using the pre-rendered HRTFs in a set of HRTFs and an assumed head pose of the user; encoding the pre-rendered binaural signal; encoding post-rendering metadata, the metadata indicating the pre-rendering HRTF; combining the encoded binaural audio signal and the encoded post-rendering metadata in a multiplexer to form a bitstream including a binaural audio representation; as well as The bitstream is transmitted to a device held by the user.

15. The method of claim 134, wherein: The metadata includes a direction of arrival associated with a dominant directional component of the immersive audio content.

16. The method according to claim 14 or 15, wherein: The metadata additionally includes the hypothesized head pose.

17. The method of any one of claims 14 to 16, further comprising: receiving an indication of a current user head pose from a device held by the user, and The hypothesized user head pose is determined based on the current user head pose.

18. The method according to any one of claims 14 to 17, wherein: The method is executed in a smartphone.

19. A system comprising one or more processors configured to perform the method of any one of claims 1 to 18.

20. A computer program product configured to cause one or more processors to perform the operations of any one of claims 1 to 18.

21. A processing device held by a user, comprising: a decoder configured to decode a bitstream comprising a binaural pre-rendered representation of immersive audio content and obtain a binaural audio signal and associated post-rendering metadata, the metadata indicating a pre-rendered HRTF used in the binaural pre-rendering, the pre-rendered HRTF being associated with a hypothesized head pose; a head tracker for obtaining head posture information of a user indicating a current head posture; A renderer, the renderer being configured to: identifying a post-rendering HRTF based on the metadata, the hypothesized head pose P' and the current head pose P; calculating a compensated stereo audio signal by applying a HRTF compensation operation to the binaural audio signal, the HRTF compensation operation being configured to compensate for an effect of the pre-rendered HRTF; as well as A binaural output signal is calculated by applying the post-rendering HRTF to the compensated stereo signal.

22. The apparatus of claim 21, wherein: The HRTF compensation operation involves the inverse of the pre-rendered HRTF.

23. The apparatus of claim 22, wherein: The renderer is configured to obtain the inverse of the pre-rendered HRTF by accessing a look-up table in a table comprising approximations of the inverse of the pre-rendered HRTF.

24. The apparatus of claim 22 or 23, wherein: The renderer is configured to calculate the compensated stereo audio signal by: An inverse left HRTF is applied to the left channel of the binaural audio signal to form an inverse filtered left channel, an inverse right HRTF is applied to the right channel of the binaural audio signal to form an inverse filtered right channel, and the inverse filtered left channel and the inverse filtered right channel are combined to form the left channel and the right channel of the compensated stereo audio signal, respectively.

25. The apparatus of claim 24, wherein: The inverse filtered left channel and the inverse filtered right channel are linearly combined, and a weight of the linear combination is selected to mitigate errors in diffuse components of the binaural output signal.

26. The method of claim 25, wherein: The weights are adaptive, preferably responsive to the difference between the hypothesized head pose P' and the current head pose P.

27. Apparatus as claimed in any one of claims 21 to 26, wherein: The post-rendered HRTF is personalized to a user of a processing device held by the user.

28. A device as claimed in any one of claims 21 to 27, incorporated into a wearable audio device, a visual device or an AR device.

Images