Rendering interface for audio data in augmented reality system

Abstract

An apparatus configured to process a bitstream may implement various techniques. The apparatus includes a memory configured to store a bitstream representing at least one audio element in an augmented reality scene and audio descriptive information associated with the at least one audio element. The device also includes processing circuitry coupled to the memory and configured to execute a scene manager and an audio unit. The scene manager is configured to construct a scene graph comprising at least one node representing the at least one audio element based on the at least one audio element, and modify the audio descriptive information based on the scene graph to obtain modified audio descriptive information. The audio unit is configured to render the at least one audio element to one or more speaker feeds based on the modified audio descriptive information, and output the one or more speaker feeds.

Description

[0001] This application claims priority to U.S. Patent Application No. 18 / 467,869, filed on September 15, 2023, U.S. Provisional Application No. 63 / 377,169, filed on September 26, 2022, and U.S. Provisional Application No. 63 / 578,618, filed on August 24, 2023, each of which is hereby incorporated by reference in its entirety. U.S. Patent Application No. 18 / 467,869, filed on September 15, 2023, claims the benefit of U.S. Provisional Application No. 63 / 377,169, filed on September 26, 2022, and U.S. Provisional Application No. 63 / 578,618, filed on August 24, 2023. Technical Field

[0002] The present disclosure relates to the processing of audio data. Background Art

[0003] Computer-mediated reality systems are being developed to allow computing devices to augment or add to, remove or subtract from, or generally modify an existing reality experienced by a user. Computer-mediated reality systems (which may also be referred to as "extended reality systems" or "XR systems") may include, for example, virtual reality (VR) systems, augmented reality (AR) systems, and mixed reality (MR) systems. The perceived success of a computer-mediated reality system is generally related to the ability of such computer-mediated reality system to provide a truly immersive experience in terms of both the visual and audio experiences, where the visual and audio experiences are consistent in a manner that the user expects. Although the human visual system is more sensitive than the human auditory system (e.g., in terms of perceived positioning of various objects within a scene), ensuring an adequate auditory experience is an increasingly important factor in ensuring a truly immersive experience, especially as the visual experience improves to allow better positioning of visual objects that enable users to better identify the source of audio content. Summary of the invention

[0004] The present disclosure as a whole relates to techniques for providing a separate audio interface that facilitates rendering at an audio playback system. These techniques may enable an audio playback system to synchronize playback of audio elements with playback of visual elements. An audio playback system may include an interface (such as an application programming interface, or API) that the audio system may expose to facilitate interaction with a scene manager that manages playback of one or more visual elements supporting an extended reality (XR) scene.

[0005] In some cases, the audio elements may not be captured at the same time as the visual elements, or may be added later (e.g., during an XR-mediated meeting, such as an XR video conference). Thus, the audio playback system may call a scene manager to match one or more visual elements with one or more audio elements (e.g., by comparing a name or other unique identifier (UID) associated with each of the one or more visual elements and the one or more audio elements). The scene manager may modify the audio metadata defining the pose (which may refer to position and / or orientation) of the one or more audio elements to more closely correspond to the matched one or more visual elements. The scene manager may then output the modified audio metadata to an audio unit of the audio playback system, which may render the audio elements to one or more speaker feeds. The audio playback system may then output the one or more speaker feeds to one or more speakers (which may also be referred to as loudspeakers, headphone speakers, or more generally as transducers).

[0006] Thus, these techniques can improve the operation of an audio playback system because the audio playback system can more accurately reproduce the sound field (based on one or more speaker feeds) to potentially improve the immersive experience of the XR system. That is, the audio playback system can modify the audio metadata to more closely match the corresponding visual elements, thereby increasing the immersion of the XR experience through higher resolution audio metadata, rather than rendering the audio elements based on low resolution audio metadata that may not match the corresponding visual elements. Thus, various aspects of the techniques described in this disclosure can improve the audio playback system itself.

[0007] In one example, the techniques relate to a device configured to process an audio bitstream, the device comprising: a memory configured to store a visual bitstream representing at least one visual element in an augmented reality scene and an audio bitstream representing at least one audio element in the augmented reality scene; and a processing circuit coupled to the memory and configured to: map the at least one visual element to the at least one audio element based on visual metadata associated with the at least one visual element and audio metadata associated with the at least one audio element; modify the audio metadata based on the mapping of the at least one visual element to the at least one audio element to obtain modified audio metadata; render the at least one audio element to one or more speaker feeds based on the modified audio metadata; and output the one or more speaker feeds.

[0008] In another example, the techniques relate to a method of processing at least one audio element, the method comprising: mapping at least one visual element to at least one audio element based on visual metadata associated with the at least one visual element and audio metadata associated with the at least one audio element; modifying the audio metadata based on the mapping of the at least one visual element to the at least one audio element to obtain modified audio metadata; rendering the at least one audio element to one or more speaker feeds based on the modified audio metadata; and outputting the one or more speaker feeds.

[0009] In another example, the techniques relate to a non-transitory computer-readable storage medium having stored thereon instructions that, when executed, cause one or more processors to: map at least one visual element to at least one audio element based on visual metadata associated with the at least one visual element and audio metadata associated with the at least one audio element; modify the audio metadata based on the mapping of the at least one visual element to the at least one audio element to obtain modified audio metadata; render the at least one audio element to one or more speaker feeds based on the modified audio metadata; and output the one or more speaker feeds.

[0010] In another example, the techniques relate to a device configured to process an audio bitstream, the device comprising: a memory configured to store a visual bitstream representing at least one visual element in an extended reality scene and an audio bitstream representing at least one audio element in the extended reality scene; and a processing circuit coupled to the memory and configured to execute a scene manager and an audio unit, wherein the scene manager is configured to: map the at least one visual element to the at least one audio element based on visual metadata associated with the at least one visual element and audio metadata associated with the at least one audio element; modify the audio metadata based on the mapping of the at least one visual element to the at least one audio element to obtain modified audio metadata; and register a callback with the audio unit, the audio unit being configured to request the modified audio metadata through the callback before rendering the at least one audio element, and wherein the audio unit is configured to: render the at least one audio element to one or more speaker feeds based on the modified audio metadata; and output the one or more speaker feeds.

[0011] In another example, the techniques relate to a method of processing at least one audio element, the method comprising: mapping, by a scene manager executed by a processing circuit and based on visual metadata associated with the at least one visual element and audio metadata associated with the at least one audio element, the at least one visual element to the at least one audio element; modifying, by the scene manager and based on the mapping of the at least one visual element to the at least one audio element, the audio metadata to obtain modified audio metadata; rendering, by an audio unit executed by the processing circuit and based on the modified audio metadata, the at least one audio element to one or more speaker feeds; and outputting, by the audio unit, the one or more speaker feeds.

[0012] In another example, the techniques relate to a non-transitory computer-readable storage medium having instructions stored thereon that, when executed, cause one or more processors to: execute a scene manager configured to: map at least one visual element to at least one audio element based on visual metadata associated with the at least one visual element and audio metadata associated with the at least one audio element; and modify the audio metadata based on the mapping of the at least one visual element to the at least one audio element to obtain modified audio metadata; and execute an audio unit configured to: render the at least one audio element to one or more speaker feeds based on the modified audio metadata; and output the one or more speaker feeds.

[0013] In another example, the techniques relate to a device configured to process an audio bitstream, the device comprising: a memory configured to store a visual bitstream representing at least one visual element in an augmented reality scene and an audio bitstream representing at least one audio element in the augmented reality scene; and a processing circuit coupled to the memory and configured to execute a scene manager, an audio processing unit, and an audio unit, wherein the scene manager is configured to: map the at least one visual element to the at least one audio element based on visual metadata associated with the at least one visual element and audio metadata associated with the at least one audio element; and map the at least one visual element to the at least one audio element based on the at least one visual element. The method further comprises modifying the audio metadata based on a mapping of at least one visual element to the at least one audio element to obtain modified audio metadata; and configuring the audio processing unit to modify the audio metadata based on the mapping of the at least one visual element to the at least one audio element to obtain modified audio metadata, wherein the audio processing unit is configured to: replace the audio metadata in the audio bitstream with the modified audio metadata based on the configuration; and output the audio bitstream to the audio unit, and wherein the audio unit is configured to: render the at least one audio element to one or more speaker feeds based on the modified audio metadata; and output the one or more speaker feeds.

[0014] In another example, the techniques relate to a method of processing at least one audio element, the method comprising: mapping, by a scene manager executed by a processing circuit and based on visual metadata associated with at least one visual element and audio metadata associated with the at least one audio element, the at least one visual element to the at least one audio element; modifying, by the scene manager and based on the mapping of the at least one visual element to the at least one audio element, the audio metadata to obtain modified audio metadata; configuring, by the scene manager, an audio processing unit to modify the audio metadata based on the mapping of the at least one visual element to the at least one audio element to obtain modified audio metadata; replacing, by the audio processing unit and based on the configuration, the audio metadata in the audio bitstream with the modified audio metadata; and outputting, by the audio processing unit, the audio bitstream to an audio unit executed by the processing circuit; rendering, by the audio unit and based on the modified audio metadata, the at least one audio element to one or more speaker feeds; and outputting, by the audio unit, the one or more speaker feeds.

[0015] In another example, the techniques relate to a non-transitory computer-readable storage medium having instructions stored thereon that, when executed, cause one or more processors to: execute a scene manager configured to map at least one visual element to at least one audio element based on visual metadata associated with the at least one visual element and audio metadata associated with the at least one audio element, and to modify the audio metadata based on the mapping of the at least one visual element to the at least one audio element to obtain modified audio metadata, and to configure an audio processing unit to modify the audio metadata based on the mapping of the at least one visual element to the at least one audio element to obtain modified audio metadata; execute the audio processing unit to replace audio metadata in an audio bitstream with the modified audio metadata based on the configuration, and output the audio bitstream to an audio unit; and execute the audio unit configured to render the at least one audio element to one or more speaker feeds based on the modified audio metadata, and output the one or more speaker feeds.

[0016] In another example, the techniques relate to a device configured to process an audio bitstream, the device comprising: a memory configured to store a visual bitstream representing at least one visual element in an extended reality scene and an audio bitstream representing at least one audio element in the extended reality scene; and a processing circuit coupled to the memory and configured to execute a scene manager and an audio unit, wherein the scene manager is configured to: map the at least one visual element to the at least one audio element based on visual metadata associated with the at least one visual element and audio metadata associated with the at least one audio element; construct a scene graph based on the mapping of the at least one visual element to the at least one audio element, the scene graph comprising a parent node representing the at least one visual element and a child node dependent on the parent node and representing the at least one audio element; and modify the audio metadata based on the scene graph to obtain modified audio metadata, and wherein the audio unit is configured to: render the at least one audio element to one or more speaker feeds based on the modified audio metadata; and output the one or more speaker feeds.

[0017] In another example, these technologies relate to a method of processing at least one audio element, the method comprising: mapping the at least one visual element to the at least one audio element by a scene manager executed by a processing circuit and based on visual metadata associated with the at least one visual element and audio metadata associated with the at least one audio element; constructing a scene graph by the scene manager and based on the mapping of the at least one visual element to the at least one audio element, the scene graph comprising a parent node representing the at least one visual element and a child node that depends on the parent node and represents the at least one audio element; modifying the audio metadata by the scene manager and based on the scene graph to obtain modified audio metadata; rendering the at least one audio element to one or more speaker feeds by an audio unit executed by the processing circuit and based on the modified audio metadata; and outputting the one or more speaker feeds by the audio unit.

[0018] In another example, the techniques relate to a non-transitory computer-readable storage medium having instructions stored thereon, which, when executed, cause one or more processors to: execute a scene manager, the scene manager being configured to: map at least one visual element to at least one audio element based on visual metadata associated with the at least one visual element and audio metadata associated with the at least one audio element; construct, by the scene manager and based on the mapping of the at least one visual element to the at least one audio element, a scene graph, the scene graph comprising a parent node representing the at least one visual element and child nodes dependent on the parent node and representing the at least one audio element; and modify, by the scene manager and based on the scene graph, the audio metadata to obtain modified audio metadata; and execute an audio unit, the audio unit being configured to: render the at least one audio element to one or more speaker feeds based on the modified audio metadata; and output the one or more speaker feeds.

[0019] In another example, the techniques relate to a device configured to process a bitstream, the device comprising: a memory configured to store a bitstream representing at least one audio element in an augmented reality scene and audio descriptive information associated with the at least one audio element; and a processing circuit coupled to the memory and configured to execute a scene manager and an audio unit, wherein the scene manager is configured to: build a scene graph based on the at least one audio element, the scene graph comprising at least one node representing the at least one audio element; and modify the audio descriptive information based on the scene graph to obtain modified audio descriptive information, and wherein the audio unit is configured to: render the at least one audio element to one or more speaker feeds based on the modified audio descriptive information; and output the one or more speaker feeds.

[0020] In another example, the techniques relate to a method comprising: obtaining a bitstream representing at least one audio element in an augmented reality scene and audio descriptive information associated with the at least one audio element; and constructing a scene graph based on the at least one audio element, the scene graph comprising at least one node representing the at least one audio element; and modifying the audio descriptive information based on the scene graph to obtain modified audio descriptive information, and rendering the at least one audio element to one or more speaker feeds based on the modified audio descriptive information; and outputting the one or more speaker feeds.

[0021] In another example, the techniques relate to a non-transitory computer-readable medium having instructions stored thereon that, when executed, cause a processing circuit to: obtain a bitstream representing at least one audio element in an augmented reality scene and audio descriptive information associated with the at least one audio element; and construct a scene graph based on the at least one audio element, the scene graph comprising at least one node representing the at least one audio element; and modify the audio descriptive information based on the scene graph to obtain modified audio descriptive information, and render the at least one audio element to one or more speaker feeds based on the modified audio descriptive information; and output the one or more speaker feeds.

[0022] The details of one or more examples of the disclosure are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of various aspects of the technology will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1A and Figure 1B is a diagram illustrating a system that can perform various aspects of the techniques described in this disclosure.

[0024] Figures 2 to 4 is an example Figure 1A and / or Figure 1B A block diagram of an example architecture of a playback system for performing various aspects of the audio rendering techniques described in this disclosure is shown in an example of FIG.

[0025] Figure 5A and Figure 5B is a diagram illustrating an example of an XR device.

[0026] Figure 6 An example of a wireless communication system supporting audio streaming according to aspects of the present disclosure is illustrated.

[0027] Figure 7 is an example Figure 1A and / or Figure 1BA block diagram of an example architecture of a playback system for performing various aspects of the audio rendering techniques described in this disclosure is shown in an example of FIG.

[0028] Figure 8 is a diagram illustrating an example of an audio playback system according to various aspects of the technology described in this disclosure when performing a graph update.

[0029] Fig. 9 is a diagram illustrating an example Listener Space Descriptor File (LSDF) alignment in accordance with various aspects of the techniques described in this disclosure.

[0030] Fig.10 is a flow diagram illustrating example operation of the content consumer device of FIG. 1 in performing various aspects of the techniques described in this disclosure.

[0031] Fig.11 is a diagram illustrating various aspects of the technology described in this disclosure. Figure 7 The API exposed by the Audio Unit is shown in the table that provides example functions. DETAILED DESCRIPTION

[0032] There are many different ways to represent a sound field. Example formats include channel-based audio formats, object-based audio formats, and scene-based audio formats. Channel-based audio formats refer to 5.1 surround sound formats, 7.1 surround sound formats, 22.2 surround sound formats, or any other channel-based format that positions audio channels to specific locations around the listener in order to recreate a sound field.

[0033] An object-based audio format may refer to a format in which audio objects (typically encoded using pulse code modulation (PCM) and referred to as PCM audio objects) are specified to represent a sound field. Such audio objects may include metadata identifying the position of the audio objects relative to a listener or other reference point in the sound field, so that the audio objects can be rendered to one or more speaker channels for playback in an effort to recreate the sound field. The techniques described in the present disclosure may be applicable to any of the above formats, including scene-based audio formats, channel-based audio formats, object-based audio formats, or any combination thereof.

[0034] A scene-based audio format may include a set of hierarchical elements that define a sound field in three dimensions. An example of a set of hierarchical elements is a set of spherical harmonic coefficients (SHCs). The following expression shows a description or representation of a sound field using SHCs:

[0035]

[0036] This expression shows that at time t, at any point in the sound field The pressure at pi Can be Uniquely represents. Here, c is the speed of sound (about 343 m / s), is the reference point (or observation point), j n (·) is a spherical Bessel function of order n, and is a spherical harmonic basis function of order n and suborder m (which may also be referred to as a spherical basis function). It can be appreciated that the term in the square brackets is a frequency domain representation of the signal (i.e., ), which can be approximated by various time-frequency transforms such as discrete Fourier transform (DFT), discrete cosine transform (DCT) or wavelet transform. Other examples of hierarchical sets include wavelet transform coefficient sets and other coefficient sets of multiresolution basis functions.

[0037] They can be physically acquired (e.g., recorded) by various microphone array configurations, or alternatively, they can be derived from a channel-based or object-based description of the sound field. SHC (which may also be referred to as panoramic sound coefficients) represent scene-based audio, where the SHC can be input into an audio encoder to obtain encoded SHC that can facilitate more efficient transmission or storage. For example, a method involving (1+4) 2 (25, and hence fourth-order) coefficients.

[0038] As described above, SHC can be derived from microphone recordings from using a microphone array. Various examples of how SHC can be physically obtained from a microphone array are described in the following document: Poletti, M., "Three-Dimensional Surround Sound Systems Based on Spherical Harmonics", Audio Engineering Society Journal, Vol. 53, No. 11, November 2005, pp. 1004-1025.

[0039] The following equation may illustrate how the SHC may be derived from an object-based description. The coefficients for the sound field corresponding to a single audio object are It can be expressed as:

[0040]

[0041] where i is is a spherical Hankel function (of the second kind) of order n, and is the position of the object. Knowing the source energy of the object as a function of frequency g(ω) (e.g., using time-frequency analysis techniques such as fast Fourier transform of a pulse code modulated (PCM) stream) allows the conversion of each PCM object and corresponding position into Furthermore, it can be shown (since the above is a linear and orthogonal decomposition) that for each object The coefficients are added. In this way, multiple PCM objects can be represented by The coefficients are represented (e.g. as the sum of the coefficient vectors of a single object). These coefficients may contain information about the sound field (pressure as a function of 3D coordinates) and are represented above at the observation point The transformation from a single object to a representation of the entire sound field is nearby.

[0042] Computer-mediated reality systems (which may also be referred to as "extended reality systems" or "XR systems") are being developed to take advantage of many of the potential benefits provided by panoramic sound coefficients. For example, the panoramic sound coefficients may represent a three-dimensional sound field in a manner that potentially enables accurate three-dimensional (3D) positioning of sound sources within the sound field. Thus, an XR device may render the panoramic sound coefficients as speaker feeds that accurately reproduce the sound field when played through one or more speakers.

[0043] Using Atmos for XR can enable the development of multiple use cases that rely on the more immersive sound field provided by Atmos, especially for computer gaming applications and real-time visual streaming applications. In these highly dynamic use cases that rely on low-latency reproduction of the sound field, XR devices may prefer Atmos over other representations that are more difficult to manipulate or involve complex rendering. Figure 1A and Figure 1B More information about these use cases is available.

[0044] Although described in the present disclosure with respect to VR devices, various aspects of the described techniques may be performed in the context of other devices such as mobile devices. In this case, a mobile device (such as a so-called smart phone) may present a displayed world via a screen that may be mounted on the head of the user 102 or viewed as would normally be done using the mobile device. Thus, any information on the screen may be part of the mobile device. The mobile device may be able to provide tracking information and thereby allow both a VR experience (when mounted on the head) and a normal experience of viewing the displayed world, wherein the normal experience may still allow the user to view the displayed world in what may be a VR-lite type of experience (e.g., lifting the device and rotating or translating the device to view different parts of the displayed world).

[0045] Figure 1A and Figure 1Bis a diagram illustrating a system that can perform various aspects of the technology described in this disclosure. Figure 1A As shown in the example of , system 10 includes a source device 12 and a content consumer device 14. Although described in the context of source device 12 and content consumer device 14, these techniques may be implemented in any context in which any layered representation of a sound field is encoded to form a bitstream representing audio data. In addition, source device 12 may represent any form of computing device capable of generating a layered representation of a sound field, and is generally described herein in the context of being a VR content creator device. Similarly, content consumer device 14 may represent any form of computing device capable of implementing the audio stream interpolation techniques and audio playback described in the present disclosure, and is generally described herein in the context of being a VR client device.

[0046] Source device 12 may be operated by an entertainment company or other entity that can generate multi-channel audio content for consumption by content consumer devices such as content consumer device 14. In many VR scenarios, source device 12 generates audio content in conjunction with visual content. Source device 12 includes content capture device 300 and content sound field representation generator 302.

[0047] The content capture device 300 may be configured to interface with or otherwise communicate with one or more microphones 5A-5N (“microphones 5”). Microphones 5 may represent devices capable of capturing a sound field and representing it as corresponding scene-based audio data 11A-11N (which may also be referred to as panoramic sound coefficients 11A-11N or “panoramic sound coefficients 11”). or other types of 3D audio microphones. In the context of scene-based audio data 11 (which is another way of referring to the panoramic sound coefficients 11), each of the microphones 5 may represent a cluster of microphones arranged within a single housing according to a set geometry that helps generate the panoramic sound coefficients 11. Thus, the term microphone may refer to a group of microphones (which are actually geometrically arranged transducers) or a single microphone (which may be referred to as a point microphone or point transducer).

[0048] The panoramic sound coefficient 11 may represent an example of an audio stream. Therefore, the panoramic sound coefficient 11 may also be referred to as the audio stream 11. Although mainly described with respect to the panoramic sound coefficient 11, these techniques may be performed with respect to other types of audio streams, including pulse code modulation (PCM) audio streams, channel-based audio streams, object-based audio streams, etc.

[0049] In some examples, the content capture device 300 may include an integrated microphone integrated into the housing of the content capture device 300. The content capture device 300 may interface with the microphone 5 wirelessly or via a wired connection. Rather than capturing audio data via the microphone 5 or in conjunction with capturing audio data, the content capture device 300 may process the panoramic sound coefficients 11 after inputting the panoramic sound coefficients 11 wirelessly via some type of removable storage device and / or via a wired input process, or alternatively or in conjunction with the foregoing, generate or otherwise create (from stored sound samples, such as is common in gaming applications, etc.). Thus, various combinations of content capture devices 300 and microphones 5 are possible.

[0050] The content capture device 300 may also be configured to interface with or otherwise communicate with a sound field representation generator 302. The sound field representation generator 302 may include any type of hardware device capable of interfacing with the content capture device 300. The sound field representation generator 302 may use the panoramic sound coefficients 11 provided by the content capture device 300 to generate various representations of the same sound field represented by the panoramic sound coefficients 11.

[0051] For example, to generate different representations of the sound field using the panoramic sound coefficients (which is also an example of an audio stream), the sound field representation generator 24 can use a decoding scheme for the panoramic sound representation of the sound field, which is called mixed-order panoramic sound (MOA), as discussed in more detail in U.S. application serial number 15 / 672,058, entitled “MIXED-ORDER AMBISONICS (MOA) AUDIO DATA FO COMPUTER-MEDIATED REALITY SYSTEMS,” filed on August 8, 2017 and published on January 3, 2019 as U.S. Patent Publication No. 20190007781.

[0052] To generate a particular MOA representation of a sound field, the sound field representation generator 24 may generate a partial subset of the full set of panoramic sound coefficients (wherein the term "subset" is not used in a strict mathematical sense to include zero or more (if not all) of the full set, but may refer to one or more, but not all, of the full set). For example, each MOA representation generated by the sound field representation generator 24 may provide accuracy with respect to some areas of the sound field, but have lower accuracy in other areas. In one example, the MOA representation of the sound field may include eight (8) uncompressed panoramic sound coefficients, while a third-order panoramic sound representation of the same sound field may include sixteen (16) uncompressed panoramic sound coefficients. Thus, each MOA representation of the sound field generated as a partial subset of the panoramic sound coefficients may be less storage intensive and less bandwidth intensive (if and when sent as part of the bitstream 27 over the transmission channel shown) (compared to a corresponding third-order panoramic sound representation of the same sound field generated from the panoramic sound coefficients).

[0053] Although described with respect to an MOA representation, the techniques of the present disclosure may also be performed with respect to a first-order Atmos (FOA) representation, where all Atmos coefficients associated with the first-order spherical basis functions and the zero-order spherical basis functions are used to represent the sound field. In other words, the sound field representation generator 302 may use all Atmos coefficients of a given order N to represent the sound field, rather than using a partial non-zero subset of the Atmos coefficients to represent the sound field, resulting in a total number of Atmos coefficients equal to (N+1). 2 .

[0054] In this regard, the Atmos audio data (which is another way of referring to an MOA representation or a full-order representation such as the Atmos coefficients in the first-order representation described above) may include Atmos coefficients associated with spherical basis functions having an order of one or less (which may be referred to as "first-order Atmos audio data"), Atmos coefficients associated with spherical basis functions having a mixed order and sub-orders (which may be referred to as the "MOA representation" discussed above), or Atmos coefficients associated with spherical basis functions having an order greater than one (which is referred to as the "full-order representation" above).

[0055] In some examples, the content capture device 300 can be configured to wirelessly communicate with the sound field representation generator 302. In some examples, the content capture device 300 can communicate with the sound field representation generator 302 via one or both of a wireless connection or a wired connection. Via the connection between the content capture device 300 and the sound field representation generator 302, the content capture device 300 can provide various forms of content, which for the purposes of discussion are described herein as part of the panoramic sound coefficients 11.

[0056] In some examples, the content capture device 300 may utilize various aspects of the sound field representation generator 302 (in terms of hardware or software capabilities of the sound field representation generator 302). For example, the sound field representation generator 302 may include dedicated hardware that is configured to (or dedicated software that, when executed, causes one or more processors to perform operations) perform psychoacoustic audio coding (such as the Unified Speech and Audio Codec denoted “USAC” proposed by the Moving Picture Experts Group (MPEG)), the MPEG-H 3D audio coding standard, the MPEG-I immersive audio standard, or a proprietary standard such as AptX. TM (including various versions of AptX, such as Enhanced AptX (E-AptX), AptX live, AptX stereo and AptX High Definition (AptX HD)), Advanced Audio Coding (AAC), Audio Codec 3 (AC-3), Apple Lossless Audio Codec (ALAC), MPEG-4 Audio Lossless Streaming (ALS), Enhanced AC-3, Free Lossless Audio Codec (FLAC), Monkey Audio, MPEG-1 Audio Layer II (MP2), MPEG-1 Audio Layer III (MP3), Opus and Windows Media Audio (WMA).

[0057] The content capture device 300 may not include psychoacoustic audio encoder dedicated hardware or dedicated software, but may provide the audio aspects of the content 301 in a non-psychoacoustic audio coded form. The sound field representation generator 302 may assist in the capture of the content 301 by performing psychoacoustic audio encoding at least in part with respect to the audio aspects of the content 301.

[0058] The sound field representation generator 302 may also assist in content capture and transmission by generating one or more bitstreams 21 based at least in part on audio content (e.g., MOA representation, third-order Atmos representation, and / or first-order Atmos representation) generated from the Atmos coefficients 11. The bitstream 21 may represent a compressed version of the Atmos coefficients 11 (and / or a partial subset thereof of the MOA representation used to form the sound field) and any other different types of content 301 (such as a compressed version of spherical visual data, image data, or text data).

[0059] The sound field representation generator 302 may generate a bitstream 21 for transmission, for example, across a transmission channel (which may be a wired or wireless channel), a data storage device, etc. The bitstream 21 may represent an encoded version of the panoramic sound coefficients 11 (and / or a partial subset thereof for forming an MOA representation of the sound field), and may include a main bitstream and another side bitstream, which may be referred to as side channel information. In some cases, the bitstream 21 representing a compressed version of the panoramic sound coefficients 11 may conform to a bitstream produced according to the MPEG-H 3D audio coding standard and / or the MPEG-I standard for "coded representation of immersive media".

[0060] Content consumer device 14 may be operated by an individual and may represent a VR client device. Although described with respect to a VR client device, content consumer device 14 may represent other types of devices, such as an augmented reality (AR) client device, a mixed reality (MR) client device (or any other type of head mounted display device or extended reality (XR) device), a standard computer, a headset, headphones, or any other device capable of tracking the head movements and / or general translational movements of the individual operating client consumer device 14. Figure 1A As shown in the example of , content consumption device 14 includes audio playback system 16A, which can refer to any form of audio playback system capable of rendering Atmos coefficients (whether in the form of first-order, second-order and / or third-order Atmos representations and / or MOA representations) for playback as multi-channel audio content.

[0061] Content consumer device 14 may retrieve bitstream 21 directly from source device 12. In some examples, content consumer device 12 may interface with a network, including a fifth generation (5G) cellular network, to retrieve bitstream 21 or otherwise cause source device 12 to send bitstream 21 to content consumer device 14.

[0062] Although in Figure 1A 14, but the source device 12 may output the bitstream 21 to an intermediate device located between the source device 12 and the content consumer device 14. The intermediate device may store the bitstream 21 for later delivery to the content consumer device 14 that may request the bitstream. The intermediate device may include a file server, a web server, a desktop computer, a laptop computer, a tablet computer, a mobile phone, a smart phone, or any other device capable of storing the bitstream 21 for later retrieval by an audio decoder. The intermediate device may reside in a content delivery network capable of streaming the bitstream 21 (and possibly in conjunction with sending a corresponding visual data bitstream) to a subscriber such as the content consumer device 14 that requests the bitstream 21.

[0063] Alternatively, source device 12 may store bitstream 21 to a storage medium such as a compact disc, digital visual disc, high-definition visual disc, or other storage medium, most of which are capable of being read by a computer and may therefore be referred to as a computer-readable storage medium or a non-transitory computer-readable storage medium. In this context, a transmission channel may refer to a channel through which the content stored to the medium is transmitted (and may include retail stores and other store-based delivery mechanisms). In any case, the technology of the present disclosure should therefore not be limited in this regard. Figure 1A .

[0064] As described above, the content consumer device 14 includes an audio playback system 16. The audio playback system 16 may represent any system capable of playing back multi-channel audio data. The audio playback system 16A may include a plurality of different audio renderers 22. The renderers 22 may each provide different forms of audio rendering, wherein the different forms of rendering may include one or more of various ways of performing vector-based amplitude panning (VBAP) and / or one or more of various ways of performing sound field synthesis. As used herein, "A and / or B" means "A or B" or "both A and B".

[0065] The audio playback system 16A may further include an audio decoding device 24. The audio decoding device 24 may represent an audio device configured to decode the bitstream 21 to output reconstructed Atmos coefficients 11A′-11N′ (which may form a complete first-order, second-order and / or third-order Atmos representation or a subset thereof, which form a MOA representation of the same sound field or a decomposition thereof, such as a main audio signal, ambient Atmos coefficients, and vector-based signals described in the MPEG-H 3D Audio Coding Standard and / or the MPEG-I Immersive Audio Standard).

[0066] Thus, the panoramic sound coefficients 11A'-11N' ("the panoramic sound coefficients 11'") may be similar to the complete set or a partial subset of the panoramic sound coefficients 11, but may differ due to lossy operations (e.g., quantization) and / or transmission via a transmission channel. After decoding the bitstream 21 to obtain the panoramic sound coefficients 11', the audio playback system 16 may obtain the panoramic sound audio data 15 from the different streams of the panoramic sound coefficients 11' and render the panoramic sound audio data 15 to the output speaker feed 25. The speaker feed 25 may drive one or more speakers (for ease of illustration, in Figure 1A (not shown in the example of ). The panoramic sound representation of the sound field can be normalized in a variety of ways, including N3D, SN3D, FuMa, N2D or SN2D.

[0067] In order to select an appropriate renderer or, in some cases, generate an appropriate renderer, the audio playback system 16A may obtain speaker information 13 indicating the number of speakers and / or the spatial geometry of the speakers. In some cases, the audio playback system 16A may obtain the speaker information 13 using a reference microphone and output a signal for activating (or in other words, driving) the speaker in a manner that dynamically determines the speaker information 13 via the reference microphone. In other cases, or in conjunction with the dynamic determination of the speaker information 13, the audio playback system 16A may prompt the user to interface with the audio playback system 16A and input the speaker information 13.

[0068] The audio playback system 16A may select one of the audio renderers 22 based on the speaker information 13. In some cases, the audio playback system 16A may generate one of the audio renderers 22 based on the speaker information 13 when none of the audio renderers 22 are within a certain threshold similarity metric (in terms of speaker geometry) to the speaker geometry specified in the speaker information 13. In some cases, the audio playback system 16A may generate one of the audio renderers 22 based on the speaker information 13 without first attempting to select an existing one of the audio renderers 22.

[0069] When outputting the speaker feeds 25 to headphones, the audio playback system 16A can utilize one of the renderers 22 that provides binaural rendering for headphone speaker playback using a head-related transfer function (HRTF) or other function capable of rendering to the left and right speaker feeds 25. The term "speaker" or "transducer" can generally refer to any speaker, including a loudspeaker, a headphone speaker, etc. The one or more speakers can then play back the rendered speaker feeds 25.

[0070] Although described as rendering the speaker feeds 25 from the Atmos audio data 15, references to rendering of the speaker feeds 25 may refer to other types of rendering, such as rendering incorporated directly into decoding the Atmos audio data 15 from the bitstream 21. Examples of alternative renderings may be found in Annex G of the MPEG-H 3D Audio Coding Standard, where rendering occurs during main and background signal formation prior to synthesis of the sound field. References to rendering of the Atmos audio data 15 should therefore be understood to refer to both rendering of the actual Atmos audio data 15 and to decomposition of the Atmos audio data 15 or representations thereof, such as the main audio signals, ambience Atmos coefficients and / or vector-based signals (which may also be referred to as V-vectors) described above.

[0071] As described above, the content consumption device 14 may represent a VR device in which a human wearable display is mounted in front of the eyes of a user operating the VR device. Figure 5A and Figure 5B 400A and 400B are diagrams illustrating examples of VR devices. Figure 5A In the example of , VR device 400A is coupled to or otherwise includes headphones 404 that can reproduce the sound field represented by panoramic audio data 15 (which is another way to refer to panoramic sound coefficients 15) through playback of speaker feeds 25. Speaker feeds 25 can represent analog or digital signals that can cause a membrane within a transducer of the headphones 404 to vibrate at various frequencies. Such a process is generally referred to as driving the headphones 404.

[0072] Visual, audio, and other sensory data can play an important role in a VR experience. To participate in a VR experience, a user 402 can wear a VR device 400A (which can also be referred to as a VR headset 400A) or other wearable electronic device. A VR client device (such as a VR headset 400A) can track the head movements of the user 402 and adjust the visual data shown via the VR headset 400A to account for the head movements, thereby providing an immersive experience in which the user 402 can experience the virtual world shown in the visual data in visual three dimensions.

[0073] While VR (and other forms of AR and / or MR, which may generally be referred to as computer-mediated reality devices) may allow a user 402 to visually reside in a virtual world, a VR headset 400A may generally lack the ability to auditorily place the user in the virtual world. In other words, a VR system (which may include a computer responsible for rendering visual data and audio data—for ease of illustration, in FIG. 4A ) may be configured to provide a user with a virtual world. Figure 5A , and the VR headset 400A) may not be able to auditorily support full three-dimensional immersion.

[0074] Figure 5Bis a diagram illustrating an example of a wearable device 400B that can operate according to various aspects of the technology described in the present disclosure. In various examples, the wearable device 400B can represent a VR headset (such as the VR headset 400A described above), an AR headset, an MR headset, or any other type of XR headset. Augmented reality "AR" can refer to computer-rendered images or data that are overlaid on the real world where the user is actually located. Mixed reality "MR" can refer to computer-rendered images or data that are locked to a specific location in the real world, or can refer to a variation on VR in which part of the computer-rendered 3D elements and part of the real elements photographed are combined into an immersive experience that simulates the user's physical presence in the environment. Extended reality "XR" can represent a general term for VR, AR, and MR. More information about XR terminology can be found in the document entitled "Virtual Reality, Augmented Reality, and Mixed Reality Definitions" published by Jason Peterson on July 7, 2017.

[0075] Wearable device 400B may represent other types of devices, such as watches (including so-called “smart watches”), glasses (including so-called “smart glasses”), headphones (including so-called “wireless headphones” and “smart headphones”), smart clothing, smart jewelry, etc. Whether representing a VR device, watch, glasses and / or headphones, wearable device 400B may communicate with a computing device supporting wearable device 400B via a wired connection or a wireless connection.

[0076] In some cases, the computing device supporting the wearable device 400B can be integrated within the wearable device 400B, and thus, the wearable device 400B can be considered to be the same device as the computing device supporting the wearable device 400B. In other cases, the wearable device 400B can communicate with a separate computing device that can support the wearable device 400B. In this regard, the term "support" should not be understood as requiring a separate dedicated device, but one or more processors configured to perform various aspects of the technology described in the present disclosure can be integrated within the wearable device 400B, or integrated within a computing device separate from the wearable device 400B.

[0077] For example, when wearable device 400B represents an example of VR device 400B, a separate dedicated computing device (such as a personal computer including one or more processors) can render audio and video content, and wearable device 400B can determine translational head movement according to various aspects of the technology described in this disclosure, where based on the translational head movement, the dedicated computing device can render the audio content (as a speaker feed). As another example, when wearable device 400B represents smart glasses, wearable device 400B can include one or more processors that both determine translational head movement (by docking within one or more sensors of wearable device 400B) and render the speaker feed based on the determined translational head movement.

[0078] As shown, wearable device 400B includes one or more directional speakers and one or more tracking and / or recording cameras. In addition, wearable device 400B includes one or more inertial, tactile and / or health sensors, one or more eye tracking cameras, one or more high-sensitivity audio microphones, and optical / projection hardware. The optical / projection hardware of wearable device 400B can include durable translucent display technology and hardware.

[0079] Wearable device 400B also includes connectivity hardware, which may represent one or more network interfaces that support multi-mode connectivity, such as 4G communications, 5G communications, Bluetooth, etc. Wearable device 400B also includes one or more ambient light sensors and bone conduction transducers. In some cases, wearable device 400B may also include one or more passive and / or active cameras with fisheye lenses and / or telephoto lenses. Although in Figure 5B Although not shown in the figure, the wearable device 400B can also include one or more light emitting diode (LED) lights. In some examples, the LED lights can be referred to as "super bright" LED lights. In some specific implementations, the wearable device 400B can also include one or more rear cameras. It should be understood that the wearable device 400B can present a variety of different form factors.

[0080] Additionally, tracking and recording cameras and other sensors can facilitate determining translation distance. Figure 5B Not shown in the example of , the wearable device 400B may include other types of sensors for detecting translation distance.

[0081] Although specific examples of wearable devices such as those described above Figure 5B The VR device 400B discussed in the example Figure 1A and Figure 1B Other devices described in the examples of FIG. 1 and FIG. 2 are described, but a person skilled in the art will understand that Figures 1A to 5BThe related description may apply to other examples of wearable devices. For example, other wearable devices (such as smart glasses) may include sensors by which translational head movement is obtained. As another example, other wearable devices (such as smart watches) may include sensors by which translational movement is obtained. Therefore, the techniques described in this disclosure should not be limited to a specific type of wearable device, but any wearable device can be configured to perform the techniques described in this disclosure.

[0082] exist Figure 1A In the example of , source device 12 also includes camera 200. Camera 200 may be configured to capture visual data and provide the captured raw visual data to content capture device 300. Content capture device 300 may provide the visual data to another component of source device 12 for further processing into the viewport-divided portions.

[0083] The content consumption device 14 also includes a wearable device 800. It should be understood that in various specific implementations, the wearable device 800 may be included in the content consumption device 14, or may be externally coupled to the content consumption device. Figure 5A and Figure 5B As discussed, wearable device 800 includes display hardware and speaker hardware for outputting visual data (eg, associated with various viewports) and for rendering audio data.

[0084] In any case, the audio aspects of VR have been divided into three separate immersion categories. The first category provides the lowest level of immersion and is called three degrees of freedom (3DOF). 3DOF refers to audio rendering that takes into account the movement of the head in three degrees of freedom (yaw, pitch, and roll), thereby allowing the user to look around freely in any direction. However, 3DOF cannot account for translational head movements where the head is not centered in the optical and acoustic center of the sound field.

[0085] The second category, referred to as 3DOF plus (3DOF+), provides three degrees of freedom (yaw, pitch, and roll) in addition to limited spatial translational movement due to the head being positioned away from the optical and acoustic centers within the sound field. 3DOF+ can provide support for perceptual effects such as motion parallax, which can enhance immersion.

[0086] The third category, known as six degrees of freedom (6DOF), renders audio data in a way that takes into account the three degrees of freedom of head movement (yaw, pitch, and roll), and also takes into account the translation of the user in space (x, y, and z translation). Spatial translation can be sensed by sensors that track the user's position in the physical world or through input controllers.

[0087] 3DOF rendering is the latest technology for audio in VR. As a result, the audio aspects of VR are not as immersive as the visual aspects, potentially reducing the overall immersion experienced by the user and introducing positioning errors (e.g., such as when the auditory playback does not precisely match or fully correlate with the visual scene).

[0088] While 3DOF rendering is the current state of the art, more immersive audio rendering such as 3DOF+ and 6DOF rendering may result in higher complexity in terms of processor cycle consumption, memory and bandwidth consumption, etc. Additionally, 6DOF rendering may require additional granularity in terms of pose (which may refer to position and / or orientation), which results in higher complexity and may also complicate certain XR scenarios in terms of asynchronously capturing audio data and visual data.

[0089] For example, consider an XR scenario involving real-time audio data capture (e.g., XR conferencing, visual conferencing, visual chat, metaverse, XR gaming, etc.), where avatars (example visual objects) converse with each other using microphones to capture real-time audio and convert such real-time audio into audio objects (which may also be referred to as audio elements, as audio objects are not necessarily defined in an object format). 3DOF rendering may attempt to position audio elements into the general area of ​​the corresponding avatar, thereby providing loose co-location of audio elements to visual objects (which may also be referred to as visual elements). The lack of tighter co-location of audio elements relative to visual elements may reduce immersion and potentially lead to difficulty interpreting visual scenes.

[0090] In addition, multiple reference playback systems (which may be referred to as reference architectures) proposed via various standards such as Advanced Coding (AC) Fourth Generation (AC-4), MPEG-H 3D Audio Coding Standard, MPEG-I Immersive Coding Standard, Third Generation Partnership Project (3GPP) Standard, etc. may combine audio decoding with audio rendering in a so-called single-chip audio system. However, in some cases, an external renderer (which may refer to a renderer that is not configured to operate in the context of audio decoding) may be useful, for example, when rendering to a special class of devices that is not covered by built-in rendering (meaning rendering built-in to the audio decoding system). In these cases, the signal in the bitstream should generally be decoded and presented to the external renderer to avoid quality constraints compared to rendering to an intermediate format in the audio decoder and then re-rendered by the external renderer. In these cases, there are two instantiations using decoder built-in rendering and external rendering.

[0091] According to various aspects of the techniques described in this disclosure, the playback system 16 may include a separate audio unit 50 that provides a separate audio interface to facilitate rendering at the playback system 16. These techniques may enable the playback system 16 to synchronize the playback of audio elements with the playback of visual elements (which may refer to AR / VR / XR video elements, video data elements, and / or any elements to be viewed). The audio unit 50 may include an interface (such as an application programming interface, or API) that the audio unit 50 may expose to facilitate interaction with a scene manager 23 that manages the playback of one or more visual elements supporting an extended reality (XR) scene.

[0092] The scene manager 23 may represent a unified interface for renderer components to access audio streams associated with audio elements in a so-called scene state. The scene state may reflect the current state of all scene elements (e.g., video elements and / or audio elements), transforms / anchors, and geometry. Other components of the renderer may subscribe to changes in the scene state. Before rendering begins, all elements in the entire scene are created, and the associated metadata is updated to reflect the state of the expected scene configuration when playback begins. The audio stream is input to the renderer as PCM floating samples. The source of the audio stream may be, for example, a decoded MPEG-H audio stream or locally captured audio.

[0093] In some cases, the audio elements may not be captured at the same time as the visual elements, or may be added later (e.g., during an XR-mediated meeting, such as an XR video conference). Accordingly, the playback system 16 may call the scene manager 23 to match one or more visual elements with one or more audio elements (e.g., by comparing a name or other unique identifier (UID) associated with each of the one or more visual elements and the one or more audio elements). The scene manager 23 may modify the audio metadata defining the pose (which may refer to position and / or orientation) of the one or more audio elements to more closely correspond to the matched one or more visual elements. The scene manager 23 may then output the modified audio metadata to the audio unit 50 of the audio playback system, which may render the audio elements to one or more speaker feeds 25. The playback system 16 may then output the one or more speaker feeds 25 to one or more speakers (which may also be referred to as loudspeakers, headphone speakers, or more generally as transducers).

[0094] In operation, the scene manager 23 may map at least one visual element to at least one audio element based on visual metadata associated with the at least one visual element and audio metadata associated with the at least one audio element (both of which may be obtained by parsing the bitstream 21 during decoding, for example, by parsing the audio data specified via the bitstream 21 by the audio decoding device 24). The scene manager 23 may identify unique identifiers (UIDs) and / or names in the visual metadata and the audio metadata, and compare the UIDs and / or names (UID / names) associated with each visual element and each audio element to map one or more of the visual elements to one or more of the audio elements.

[0095] The scene manager 23 can then modify the audio metadata based on the mapping of at least one visual element to at least one audio element to obtain modified audio metadata. The audio unit 50 can present an API that the scene manager 23 can call to provide the modified audio metadata to the audio unit 50 via an API call.

[0096] The audio unit 50 may render at least one audio element to one or more speaker feeds 25 based on the modified audio metadata, thereby outputting the one or more speaker feeds 25 to one or more transducers (e.g., headphones, speakers, etc.). That is, the audio unit 50 may configure one or more of the audio renderers 22 to be based on the pose information specified by the modified audio metadata. The pose information (which may be expressed as a "pose" associated with a visual element and / or an audio element) may define the position and orientation of the corresponding element (where both the position and orientation represent 6DOF position information).

[0097] Assuming that the audio element may be captured asynchronously from the visual element (which may be computer generated and not necessarily captured but otherwise asynchronously generated), the pose of the audio element may not closely correspond to the pose of the video element. Therefore, the scene manager 23 may modify the pose of the audio element in the audio metadata to more accurately reflect the pose of the visual element in the XR scene. As described below with respect to Figures 2 to 4 As described, the API of audio unit 50 can facilitate updating the pose of audio elements in near real time to reduce latency, thereby improving the immersion of XR scenes (particularly when consumed via a wearable device such as XR device 800).

[0098] Thus, these techniques may improve the operation of playback system 16, as playback system 16 may more accurately reproduce a sound field (based on one or more speaker feeds) to potentially improve the immersive experience of an XR system such as system 10. That is, playback system 16 may modify audio metadata to more closely match corresponding visual elements, thereby increasing the immersion of the XR experience through higher resolution audio metadata, rather than rendering audio elements based on low resolution audio metadata that may not match corresponding visual elements. Thus, various aspects of the techniques described in this disclosure may improve playback system 16 itself.

[0099] Figure 1B is a block diagram illustrating another example system 100 configured to perform various aspects of the techniques described in this disclosure. System 100 is similar in that Figure 1A The system 10 shown, except for Figure 1A The audio renderer 22 is shown to be replaced by a binaural renderer 102 which is capable of performing binaural rendering using one or more HRTFs or other functions which can be rendered as left and right speaker feeds 103 .

[0100] The audio playback system 16B may output the left and right speaker feeds 103 to headphones 104, which may represent another example of a wearable device and may be coupled to an additional wearable device such as a watch, the aforementioned VR headset, smart glasses, smart clothing, a smart ring, a smart bracelet, or any other type of smart jewelry (including a smart necklace), etc. to facilitate reproduction of the sound field. The headphones 104 may be coupled to the additional wearable device wirelessly or via a wired connection.

[0101] Additionally, the headset 104 may be connected via a wired connection (such as a standard 3.5 mm audio jack, a Universal System Bus (USB) connection, an optical audio jack, or other form of wired connection) or wirelessly (such as via Bluetooth TM The headset 104 may be coupled to the audio playback system 16 (e.g., a wireless network connection, a wireless network connection, etc.). The headset 104 may reproduce the sound field represented by the panoramic sound coefficients 11 based on the left and right speaker feeds 103. The headset 104 may include a left earphone speaker and a right earphone speaker, both of which are powered (or in other words, driven) by the corresponding left and right speaker feeds 103.

[0102] Although about Figure 5A and Figure 5BThe VR device shown in the example of is described, but these techniques can be performed by other types of wearable devices, including watches (such as so-called "smart watches"), glasses (such as so-called "smart glasses"), headphones (including wireless headphones coupled via a wireless connection or smart headphones coupled via a wired or wireless connection), and any other type of wearable device. Therefore, these techniques can be performed by any type of wearable device through which a user can interact with the wearable device while wearing it.

[0103] Figures 2 to 4 is an example Figure 1A and / or Figure 1B A block diagram of an example architecture of a playback system for performing various aspects of the audio rendering techniques described in the present disclosure is shown in the example of FIG. Figure 2 , the playback system 216 may represent an example of the playback system 16. The playback system 216 includes a runtime system 220, a media access function 222, a scene manager 23, and an audio subsystem 50.

[0104] The runtime system 220 may represent a unit configured to support processing of sensor data, viewport rendering, and simultaneous localization and mapping (SLAM) processing. The runtime system 220 may be configured to transmit visual elements to a graphics language format (glTF) that specifies a 3D scene (which is an example of an XR scene) in relation to a 3D scene. TM ) to operate. Scene Manager 23 can handle glTF TM Elements to unpack and use the underlying asset (which is another way to refer to a visual element and / or a visual element). Scene Manager 23 can also handle TM Formatted bitstream separated from the audio bitstream.

[0105] The media access function (MAF) 222 may represent a unit configured to obtain media content, such as a visual bitstream specifying at least one visual element and an audio bitstream specifying an audio element (or in other words, an audio source element). The MAF 222 may implement media data (which is another way of referring to media content) to be communicated over various delivery networks, such as the Internet via wired, wireless, or other communication networks, including various cellular networks, such as fifth generation, or 5G, cellular networks. 3GPP TR 26.998 may represent an example standard by which an immersive XR media codec / profile is provided by which a glass-type XR device is integrated into a 5G network.

[0106] In any case, the audio unit 50 may expose an API 51 through which the audio unit 50 is interfaced to register a callback 53. The callback 53 may represent a function that is passed as an argument to another function (e.g., the audio unit 50) that is executed later (e.g., before rendering each frame of the audio bitstream). The scene manager 23 may map the at least one visual element 223 to the at least one audio element 225 based on the visual metadata associated with the at least one visual element 223 and the audio metadata associated with the at least one audio element 225.

[0107] As described above, the visual metadata may include the pose of at least one visual element 223 in the XR scene, and the audio metadata may include the pose of at least one audio element in the XR scene. The scene manager 23 may then modify the audio metadata based on the mapping of at least one visual element to at least one audio element to obtain modified audio metadata. That is, the scene manager 23 may modify the position of at least one audio element 225 based on the pose of at least one visual element 223 to obtain a modified position of at least one audio element 225 in the XR scene.

[0108] In some examples, the modified pose of at least one audio element is different from the pose of the at least one audio element. The modified pose of at least one audio element 225 is different from the pose of at least one audio element 225 in terms of rotation angle, elevation angle and / or translation distance.

[0109] In other words, the audio element 225 may define an original and / or audio mesh that may be a low-resolution (or, in other words, low-detail) version of the corresponding video element 223. To map the audio element 225 to the associated video element 223, the scene manager 23 may perform in-band and / or out-of-band mapping. For in-band mapping, the scene manager 23 may analyze the audio metadata associated with the audio element 225 (defined in the audio bitstream) to identify the association between the audio element 225 and the video element 223, as defined by the glTF TM For out-of-band mapping, an extension to the glTF node may define an identifier plus an alignment transform to map a visual node (representing a video element 223 ) to a corresponding audio element 225 .

[0110] The scene manager 23 may then register a callback 53 with the audio, the callback including a transformation for modifying the audio metadata to obtain modified audio metadata. The audio unit 50 may interface with the media access function 222 to obtain at least one audio element 225, whereby the audio unit 50 may render the at least one audio element to one or more speaker feeds 25 / 103 based on the modified audio metadata (as represented by the callback 53). The audio unit 50 may then output the one or more speaker feeds 25 / 103.

[0111] In some cases, when mapping visual elements 223 to audio elements 225, scene manager 23 may determine that none of at least one visual element 223 is mapped to an audio element 225 (e.g., audio element 225 is non-diegetic, meaning that audio element 225 is not heard by characters in the XR scene—such as transitions or background music heard by the audience but not by characters in the XR scene). In this case, scene manager 23 may set the unmatched audio element 225 to a universal world coordinate system (e.g., (0,0,0) in an (X,Y,Z) coordinate system). Scene manager 23 may then register a callback 53 via API 51 of audio unit 50 to maintain the universal world position of the corresponding audio element 225.

[0112] The audio unit 50 may render each frame of the audio element 225 based on this callback 53 that sets the audio metadata of the audio element 225 to the universal world coordinate system. The audio unit 50 may present the audio element 225 as background audio to each of the speaker feeds 25 / 103.

[0113] As described above, the callback 53 can define the translation and effectively represent the modified audio metadata. In some cases, the audio unit 50 can respond to the callback 53 and request the modified audio metadata before rendering the at least one audio element 225. In some examples, the audio unit 50 is configured to respond to the callback and request the modified audio metadata before rendering each frame of audio data of the at least one audio element 225.

[0114] Therefore, the relatively simple 3DOF model with a center point and shared pose information from 3GPP Technical Specification (TS) 26.118 is extended to operate in a reference space that facilitates the above-mentioned mapping between visual elements and audio elements to support 6DOF in XR systems. The pose may no longer be just head rotation, but may also include the position of the user / camera / audio listener in the 6DOF XR space. In addition, the scene itself may be more complex in XR, including interactions, multiple elements, etc. In the case of 6DOF and XR, the audio unit 50 can provide appropriate rendering using significantly more information than the time synchronization information and pose in 3DOF to properly render the immersive experience.

[0115] To overcome the limitations of 3DOF, the following needs to be defined for an audio listener: 1) the type of audio listener, 2) the pose of the audio listener (which can be obtained via the XR device 800 and / or other sensors), and 3) the consistency of the head-related transfer function (HRTF) with the underlying avatar representing the listener in the XR scene. For each audio source, the following needs to be defined: 1) the type of audio source, 2) the corresponding visual element, 3) the pose relative to the global coordinate system, and 4) timing information (which can be used to synchronize the audio element 225 with the corresponding video element 223 and start / stop time based on a schedule or interactivity).

[0116] The pose is defined relative to a reference space defined / selected by the scene manager 23. XR anchoring and interactivity applies to all media types, where anchoring can associate a given video / audio element with a set location in the XR world, real world, etc. The audio unit 50 can act as a so-called "black box" that exchanges information with the scene manager 23 via one or more APIs 51.

[0117] In this context, the audio unit 50 may operate in a 6DOF / XR audiovisual system 10. A 6DOF / XR experience is described by a scene, which may include one or more elements to which both audio and visual properties have been assigned. A 6DOF XR audiovisual experience may include the ability to move freely in at least a constrained position. An XR scene may allow modification, updating, and interaction with the scene and / or elements in the scene such that audio or visual properties may change.

[0118] The scene manager 23 (which may also be referred to as the rendering engine 23) may utilize hooks to modify or insert spatial audio metadata information, which are then used by the audio unit 50 to decode and render the 6DOF audio in synchronization with the visual experience (provided by the visual elements). In this example, the so-called "hooks" are implemented in the form of callbacks 53, which may be periodic or on-demand (e.g., as required by the audio unit 50 and / or the scene manager 23).

[0119] In this manner, various aspects of these techniques can enable steering of 6DOF audio units. Additionally, various aspects of these techniques can ensure consistency between a visual scene and an audio scene driven by a single source while keeping the audio rendering process separate. An MPEG-I Audio system compliant with MPEG-I Audio 23090-4 can perform these techniques.

[0120] exist Figure 3 In the example of , playback system 236 can represent another example of audio playback system 16. Playback system 236 can be similar to playback system 216, except that MAF 222 includes an audio processing system 55, which can also be referred to as an audio pre-processing system 55. The audio pre-processing system 55 can expose an API 51, which the scene manager 23 can call to interface and configure the audio pre-processing system 55 to modify the audio metadata associated with the audio element 225.

[0121] The scene manager 23 may configure the audio pre-processing system 55 to modify the audio metadata based on the mapping of at least one visual element 223 to at least one audio element 225 to obtain modified audio metadata. The audio processing system 55 may insert a scene update message or rewrite an existing message to define updated audio metadata associated with the audio element 225. In this manner, the audio processing system 55 may update the listener pose and / or scene geometry associated with the audio scene element 225 (which is another way of referring to the audio element 225). The audio processing system 55 may replace the audio metadata in the audio bitstream (defining the audio element 225) with the modified audio metadata based on the configuration (defined via the API 55). The audio processing system 55 may output the audio bitstream (which may be referred to as a modified 6DOF audio bitstream) to the audio unit 50.

[0122] exist Figure 4 In the example of , playback system 266 can represent another example of audio playback system 16. Playback system 266 can be similar to playback system 216 and / or 236, except that scene manager 23 can construct a scene graph based on a mapping of at least one visual element 223 to at least one audio element 225, the scene graph including a parent node representing at least one video element 225 and a child node that depends on the parent node and represents at least one audio element 223. The scene graph can be similar to the scene graph defined via various game platforms such as the Unity development platform. The scene manager 23 can modify the audio metadata based on the scene graph to obtain the modified audio metadata.

[0123] In this example, the scene manager 23 is further configured to output the modified audio metadata to the audio unit 50. In some examples, the scene manager 23 is further configured to output the modified audio metadata to the audio unit 50 via an API 51 exposed by the audio unit 50.

[0124] By managing synchronization within the scene manager 23, the scene manager 23 can reduce the latency associated with configuring the audio processing system 55 and / or configuring the callback 53. The scene manager 23 can maintain a single scene and update the single scene instead of relying on separate sets of nodes for visual and audio elements. The scene manager 23 can call the API 51 to provide 6DOF audio metadata to the audio unit 50.

[0125] In this example, the scene manager 23 can provide scene information (eg, gITF TM Inheritance of visual elements 223 from visual elements 223 to audio elements 225 that depend on visual elements 223 in the scene graph. An example scene description may be as follows:

[0126]

[0127] For MPEG-I audio bitstreams, the following metadata may be defined:

[0128]

[0129] The inherited audio metadata of the audio element 225 may be defined as follows:

[0130]

[0131] The above aspects of these techniques enable the following use cases. First, interactivity, where the runtime system 220 captures user actions that modify the position of audiovisual elements. Figure 2 In the example of , the playback system 216 may need to update the visual node and the audio node in the scene manager 23 respectively through the callback 53 of the audio unit 50. Figure 3 In the example of , the audio pre-processing system 55 may insert the updated position into the 6DoF audio bitstream. Figure 4 In the example of , the scene manager 23 interfaces with the audio unit 50 using the modified audio metadata for rendering.

[0132] In instances where an audio element is not mapped to a visual representation (non-context audio), the audio playback system 216 may render the non-context audio without requiring callback 53. The playback system 236 may bypass pre-processing of the non-context audio element. The playback system 266 may pass or add the non-context audio element as a child node to the world node {position=0,0,0}.

[0133] Figure 6 An example of a wireless communication system 100 supporting audio streaming according to various aspects of the present disclosure is illustrated. The wireless communication system 100 includes a base station 105, a UE 115, and a core network 130. In some examples, the wireless communication system 100 may be a long term evolution (LTE) network, an advanced LTE (LTE-A) network, an LTE-A Pro network, or a new radio (NR) network. In some cases, the wireless communication system 100 may support enhanced broadband communications, ultra-reliable (e.g., mission-critical) communications, low-latency communications, or communications with low-cost and low-complexity devices.

[0134] The base station 105 may communicate wirelessly with the UE 115 via one or more base station antennas. The base station 105 described herein may include or may be referred to by those skilled in the art as a base transceiver, a radio base station, an access point, a radio transceiver, a NodeB, an eNodeB (eNB), a next generation NodeB or a giga NodeB (any of which may be referred to as a gNB), a home NodeB, a home eNodeB, or some other appropriate term. The wireless communication system 100 may include different types of base stations 105 (e.g., macro cell base stations or small cell base stations). The UE 115 described herein is capable of communicating with various types of base stations 105 and network equipment (including macro eNBs, small cell eNBs, gNBs, relay base stations, etc.).

[0135] Each base station 105 may be associated with a particular geographic coverage area 110 in which it supports communications with respective UEs 115. Each base station 105 may provide communications coverage for the respective geographic coverage area 110 via a communication link 125, and the communication link 125 between the base station 105 and the UE 115 may utilize one or more carriers. The communication link 125 shown in the wireless communication system 100 may include an uplink transmission from the UE 115 to the base station 105, or a downlink transmission from the base station 105 to the UE 115. Downlink transmissions may also be referred to as forward link transmissions, and uplink transmissions may also be referred to as reverse link transmissions.

[0136] The geographic coverage area 110 for the base station 105 can be divided into sectors, which constitute a portion of the geographic coverage area 110, and each sector can be associated with a cell. For example, each base station 105 can provide communication coverage for a macro cell, a small cell, a hotspot, or other types of cells, or various combinations thereof. In some examples, the base station 105 can be mobile and thus provide communication coverage for a mobile geographic coverage area 110. In some examples, different geographic coverage areas 110 associated with different technologies can overlap, and overlapping geographic coverage areas 110 associated with different technologies can be supported by the same base station 105 or different base stations 105. The wireless communication system 100 may include, for example, a heterogeneous LTE / LTE-A / LTE-A Pro or NR network, in which different types of base stations 105 provide coverage for various geographic coverage areas 110.

[0137] UE 115 can be dispersed throughout the wireless communication system 100, and each UE 115 can be stationary or mobile.UE 115 can also be referred to as a mobile device, a wireless device, a remote device, a handheld device, or a subscriber device, or some other appropriate term, wherein "device" can also be referred to as a unit, a station, a terminal, or a client.UE 115 can also be a personal electronic device, such as a cellular phone, a personal digital assistant (PDA), a tablet computer, a laptop computer, or a personal computer. In an example of the present disclosure, UE 115 can be any audio source in the audio source described in the present disclosure, including a VR headset, an XR headset, an AR headset, a vehicle, a smart phone, a microphone, a microphone array, or any other device including a microphone, or can send a captured and / or synthesized audio stream. In some examples, the synthesized audio stream can be an audio stream stored in a memory or previously created or synthesized. In some examples, UE115 may also refer to a wireless local loop (WLL) station, an Internet of Things (IoT) device, an Internet of Everything (IoE) device, or an MTC device, etc., which may be implemented in various items such as appliances, vehicles, instruments, etc.

[0138] Some UEs 115, such as MTC or IoT devices, may be low-cost or low-complexity devices and may provide automated communication between machines (e.g., via machine-to-machine (M2M) communication). M2M communication or MTC may refer to data communication technology that allows devices to communicate with each other or with a base station 105 without human intervention. In some examples, M2M communication or MTC may include communications from devices that exchange and / or use audio metadata that indicates privacy restrictions and / or password-based privacy data to switch, block, and / or disable various audio streams and / or audio sources, as will be described in more detail below.

[0139] In some cases, UE 115 is also able to communicate directly with other UE 115 (e.g., using peer-to-peer (P2P) or device-to-device (D2D) protocols). One or more UE 115 in a group of UEs 115 utilizing D2D communication may be within the geographic coverage area 110 of the base station 105. Other UEs 115 in such a group may be outside the geographic coverage area 110 of the base station 105, or otherwise unable to receive transmissions from the base station 105. In some cases, multiple groups of UEs 115 communicating via D2D communication may utilize a one-to-many (1:M) system, in which each UE 115 transmits to each other UE 115 in the group. In some cases, the base station 105 facilitates the scheduling of resources for D2D communication. In other cases, D2D communication is performed between UEs 115 without involving the base station 105.

[0140] The base stations 105 may communicate with the core network 130 and with each other. For example, the base stations 105 may interface with the core network 130 via a backhaul link 132 (e.g., via S1, N2, N3, or other interface). The base stations 105 may communicate with each other directly (e.g., directly between the base stations 105) or indirectly (e.g., via the core network 130) over a backhaul link 134 (e.g., via X2, Xn, or other interface).

[0141] In some cases, the wireless communication system 100 may utilize both licensed and unlicensed radio frequency spectrum bands. For example, the wireless communication system 100 may adopt licensed assisted access (LAA), LTE-unlicensed (LTE-U) radio access technology, or NR technology in unlicensed bands such as the 5GHz ISM band. When operating in an unlicensed radio frequency spectrum band, wireless devices such as base stations 105 and UE 115 can use a listen-before-talk (LBT) process to ensure that the frequency channel is clear before sending data. In some cases, operations in unlicensed bands may be based on carrier aggregation configurations combined with component carriers operating in licensed bands (e.g., LAA). Operations in unlicensed spectrum may include downlink transmissions, uplink transmissions, peer-to-peer transmissions, or a combination of these items. Duplexing in unlicensed spectrum may be based on frequency division duplexing (FDD), time division duplexing (TDD), or a combination of both.

[0142] Figure 7 is an example Figure 1A and / or Figure 1B A block diagram of an example architecture of a playback system for performing various aspects of the audio rendering techniques described in this disclosure is shown in the example of FIG. Figure 7 In the example of Figure 2Another example of a playback system 216 is shown in the example of FIG. The playback system 716 may include a video playback system 216 that complies with openXR TM specification (or in other words, openXR TM The playback system 716 may also include an audio unit 750 (which represents an example of the audio unit 250), which may conform to the MPEG-I Audio standard.

[0143] The audio unit 750 may expose an API 751, which may represent an example of the API 51. The API 751 may also conform to the MPEG-I audio standard, which includes Fig.11 The examples shown in the table accept the specified input functions defined in (and can be followed by Fig.11 descriptions listed in the table).

[0144] according to Fig.11 As shown in the table in the example of , API 751 can allow the scene manager 23 to call the init function, configure function, start function, pause function, resume function, stop function, update function, updateGraph function and registerCallback function. The init function can accept an MPEG-I audio bitstream buffer / Uniform Resource Locator (ULR) or a description of an immersive audio scene, which allows the scene manager 23 to initialize the MPEG-I audio renderer by providing an MPEG-I audio bitstream URL or buffer pointer (in Figure 7 ) (or alternatively, audio unit 750 may be initialized by providing a spatial audio scene description extracted from a scene description).

[0145] The configure function can accept clocks, node maps, bounding boxes and coordinate system alignments and / or XR space and AR anchors as inputs, which the scene manager 23 can call to specify the initial configuration of the audio unit 750, with the potential goal of synchronizing the audio scene with the visual scene, but establishing a common timeline, exchanging node maps, aligning coordinate systems, and / or defining XR space and anchors. As an example, the Listener Space Descriptor File (LSDF) anchors can be aligned with MPEG_anchors in gITF. The start function, pause function, resume function, and / or stop function can accept an audio source identifier (ID) and / or action time as input, which allows the scene manager 23 to control the playback of a specific audio source for interactivity purposes.

[0146] The update function may accept as input an array of one or more of a node identifier (ID), a translation, rotation and scaling (TRS) matrix and / or a timestamp. The scene manager 23 may call the update function to update the node position and orientation in the audio scene, and the scene manager 23 may call the update function to provide a TRS matrix that is relative to the initial pose at the configuration time and is not incremental, where this may be a sequence of (TRS matrix, timestamp) to possibly support animation.

[0147] The updateGraph function may accept as input an add node (specified by a node identifier, a parent node identifier, and / or one or more attributes), a remove node (specified by a node identifier), and / or an update node (specified by a node identifier and / or one or more attributes). The scene manager 23 may call the updateGraph function to add or delete a set of audio nodes to the internal representation of the audio scene graph in the audio renderer (of the audio unit 750).

[0148] The registerCallback function may accept a callback function and / or one or more events (e.g., NEED_LISTENER_POSE) as input. The scene manager 23 may call the callback function to provide a hook for the audio renderer (of the audio unit 750) to call when a certain event is detected, for example, when the audio renderer needs an updated listener pose.

[0149] In terms of node mapping, an implicit mapping mechanism can be assumed and it is the author's responsibility to ensure appropriate and consistent node naming. In the example of gITF, the node name attribute is used for mapping, where mapping can only be applied at the node level. gITF nodes and audio scene nodes are mapped together, which may mean that the nodes are considered to be at the same hierarchical level (rather than parent-child). For all nodes in the mapping (as an example), any changes to the nodes should trigger an update call (reference scene manager 23 in this example can call the update function). The mapping can convey the transformation applied to the audio node to align the two nodes, where the transformation can be provided as new in-band scene metadata in the MPEG-I audio stream (possibly as the TRS matrix referenced above). If no transformation is provided, it is assumed that the audio transform and the corresponding visual node are aligned, or alternatively, the transformation is derived from the initial transformation at the alignment time, where, for example, TRS_visual -1 Multiply by TRS_audio. The following table specifies how the mapping is signaled as a syntax element in an MPEG-I audio bitstream:

[0150]

[0151] Other possible functions exposed by API 751 may include functions related to audio listener space and / or XR space definition. For audio listener space functions, scene manager 23 may gain an understanding of the scene (e.g., a 3D reconstruction of the user's physical environment) by interacting with runtime system 720, where scene manager 23 updates the LSDF using an update mechanism (or in other words, function). For XR space definition functions, audio unit 750 (e.g., audio renderer) may pass information about trackables that the audio renderer may track in AR physical space, and scene manager 23 may instruct runtime system 720 to create and track a new application XR space, and retrieve or otherwise obtain an initial pose and share the initial pose with the audio renderer (of audio unit 750). The audio renderer may retrieve the actual pose of the trackable in that XR space by using the callback function referenced above.

[0152] In terms of graph updates, nodes may be added to and removed from the graph, and one or more properties and components of the node may be updated. In addition, nodes are added / removed based on events in the application, where in some examples adding a node may refer to a user inserting a 3D asset into a scene (e.g., a new user joining a shared space), and in some examples removing a node may refer to a user leaving a shared space. Node updates may affect components of the node, where, for example, such updates may change the material of the node and / or change the geometry of the node.

[0153] Figure 8 is a diagram illustrating an example of an audio playback system according to various aspects of the technology described in this disclosure when performing a graph update. Figure 8 As shown in the example of Figure 7 The example of the audio playback system 716 shown in the example of shows how the scene manager 23 can call the graph update function via API 851 (which can represent an example of API 751) exposed by the audio renderer 850 (which can represent an example of the audio unit 750).

[0154] In response to user #1 leaving the collaborative XR session (represented by scene graph 870), scene manager 23 (in Figure 8In the example shown in FIG. 8 , the scene manager 23 may call the update function described above to remove node 872 (specifying a user #1 node, a whiteboard node, an avatar node, and an audio source node) from the scene graph 870. In response to user #2 joining the collaborative XR session, the scene manager 23 may call the update function to add a node 874 representing user #2 (specifying a user #2 node, a video display node, an avatar node, and an audio source node) to the scene graph 870, wherein the scene manager 23 may call the update function to update a subset of nodes 874 in the scene graph 870 (represented as nodes 876 specifying an avatar node and an audio source node).

[0155] In terms of anchor alignment for AR and / or XR, the scene manager 23 can receive constructed scene information from the runtime system 720, where the scene manager 23 can then extract acoustic related information and create an LSDF. If the LSDF is already available at the audio renderer 850, mapping of the identified objects is performed to align the LSDF to the reconstructed physical space (which can match the bounding box by rotating, translating and scaling the LSDF acoustic environment geometry according to the TRS matrix). The physical anchor is mapped and / or aligned with a trackable object of the anchor, which can be, for example, a floor, QR code, or 2D image of a user tracked by the runtime system 720.

[0156] Although shown as including various nodes for both visual elements and audio elements, in some cases, the scene graph may include only audio elements or only video elements, where various relationships between other audio elements or other video elements may be provided by way of the scene graph. In some cases, a separate visual scene graph may be mapped to a separate audio scene graph to identify a common scene graph, such as described above with respect to Figure 8 In some examples, only the audio scene graph may be used to obtain modified audio metadata (which, as described below, may represent a single example of audio descriptive information). Therefore, all references to audio metadata herein may also be referred to as audio descriptive information.

[0157] Fig. 9 is a diagram illustrating an example Listener Space Descriptor File (LSDF) alignment according to various aspects of the techniques described in this disclosure. Fig. 9 In the example of Figure 7 A representation of the scene can be obtained as shown in Figure 1, where the LSDF and LSDF updates are aligned with the audio coordinate system. The alignment can include matching trackable objects (e.g., QR codes, images, floors, etc.) and / or scaling, rotation, and / or translation. The LSDF is then used for audio rendering.

[0158] Although described above with respect to audio metadata, various aspects of the techniques described in this disclosure may be applied to any type of audio descriptive information. While audio metadata may be specified in an audio bitstream representing an audio element, audio descriptive information may be specified in a variety of other auxiliary information, including: via different transmission formats associated with cellular communication standards, such as 3GPP 3G, 3GPP 4G, 3GPP 5G, 3GPP 6G, etc.; various wireless network standards, including personal area network standards (such as Bluetooth); TM ), IEEE 802.11 family of standards, etc.; MPEG standards related to audio (e.g., MPEG-1, MPEG-2, MPEG-4, etc.). Therefore, audio descriptive information can be associated with audio elements, but is not necessarily sent in the same audio bitstream as the audio elements, but rather sent as auxiliary information or other transmission mechanisms.

[0159] Fig.10 1 is a flowchart illustrating example operations of the content consumer device of FIG. 1 in performing various aspects of the techniques described in this disclosure. Fig.10 As shown in the example of FIG. 1 , the content consumer device 14 shown in the example of FIG. 1 may obtain a bitstream representing at least one audio element in an augmented reality scene and audio descriptive information associated with the at least one audio element ( 1000 ).

[0160] The content consumer device 14 may also construct a scene graph based on the at least one audio element, the scene graph including at least one node representing the at least one audio element (1002). The content consumer device 14 may also modify the audio descriptive information based on the scene graph to obtain modified audio descriptive information (1004). The content consumer device 14 may then render the at least one audio element to one or more speaker feeds based on the modified audio descriptive information (1006) and output the one or more speaker feeds (1008).

[0161] It should be appreciated that, according to examples, certain actions or events of any of the techniques described herein may be performed in a different order, may be added, combined, or omitted entirely (e.g., not all actions or events described are necessary to implement the technique). In addition, in some examples, actions or events may be performed concurrently (e.g., through multithreading, interrupt processing, or multiple processors) rather than sequentially.

[0162] In some examples, a VR device (or streaming device) can use a network interface coupled to a memory of the VR / streaming device to communicate exchange messages with an external device, wherein the exchange messages are associated with multiple available representations of a sound field. In some examples, the VR device can use an antenna coupled to the network interface to receive wireless signals, which include data packets, audio packets, visual packets, or transmission protocol data associated with multiple available representations of the sound field. In some examples, one or more microphone arrays can capture the sound field.

[0163] In some examples, the multiple available representations of the sound field stored to the memory device may include multiple object-based representations of the sound field, a higher-order Atmos representation of the sound field, a mixed-order Atmos representation of the sound field, a combination of the object-based representation of the sound field and a higher-order Atmos representation of the sound field, a combination of the object-based representation of the sound field and a mixed-order Atmos representation of the sound field, or a combination of the mixed-order representation of the sound field and a higher-order Atmos representation of the sound field.

[0164] In some examples, one or more of the multiple available representations of the sound field may include at least one high-resolution region and at least one low-resolution region, and wherein the presentation selected based on the steering angle provides higher spatial accuracy with respect to the at least one high-resolution region and less spatial accuracy with respect to the low-resolution region.

[0165] In one or more examples, the described functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functionality may be stored as one or more instructions or codes on a computer-readable medium or sent via a computer-readable medium and executed by a hardware-based processing unit. A computer-readable medium may include a computer-readable storage medium (which corresponds to a tangible medium such as a data storage medium) or a communication medium, which includes, for example, any medium that facilitates the transfer of a computer program from one place to another according to a communication protocol. Thus, a computer-readable medium may generally correspond to (1) a non-transient tangible computer-readable storage medium, or (2) a communication medium such as a signal or carrier wave. A data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, codes, and / or data structures for implementing the techniques described in the present disclosure. A computer program product may include a computer-readable medium.

[0166] By way of example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, flash memory, or any other medium that can be used to store the desired program code in the form of instructions or data structures and can be accessed by a computer. Moreover, any connection is appropriately referred to as a computer-readable medium. For example, if the instruction is sent from a website, server or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wireless technology (such as infrared, radio and microwave), the coaxial cable, fiber optic cable, twisted pair, DSL or wireless technology (such as infrared, radio and microwave) is included in the definition of the medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals or other transient media, but are instead directed to non-transient tangible storage media. As used herein, disks and optical disks include compact discs (CDs), laser optical disks, optical optical disks, digital versatile discs (DVDs), floppy disks and blue-ray discs, wherein disks generally reproduce data magnetically, while optical disks reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0167] Instructions may be executed by one or more processors, including fixed-function processing circuits and / or programmable processing circuits, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Therefore, the term "processor" as used herein may refer to any of the above structures or any other structure suitable for implementing the techniques described herein. In addition, in some aspects, the functionality described herein may be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into a combined codec. In addition, these techniques may be fully implemented in one or more circuits or logic elements.

[0168] The techniques of the present disclosure may be implemented in a variety of devices or apparatuses, including a wireless handset, an integrated circuit (IC), or a set of ICs (e.g., a chipset). Various components, modules, or units are described in the present disclosure to emphasize functional aspects of devices configured to perform the disclosed techniques, but do not necessarily require implementation by different hardware units. Specifically, as described above, the various units may be combined in a codec hardware unit, or the various units may be provided by a collection of interoperable hardware units (including one or more processors as described above) in combination with appropriate software and / or firmware.

[0169] In this manner, various aspects of the technology can implement the following embodiments.

[0170] Embodiment 1A. A device configured to process an audio bitstream, the device comprising: a memory configured to store a visual bitstream representing at least one visual element in an extended reality scene and the audio bitstream representing at least one audio element in the extended reality scene; and a processing circuit coupled to the memory and configured to: modify audio metadata associated with the at least one audio element based on the at least one visual element to the at least one audio element to obtain modified audio metadata; render the at least one audio element to one or more speaker feeds based on the modified audio metadata; and output the one or more speaker feeds.

[0171] Embodiment 1.5A. A device according to Embodiment 1A, wherein the processing circuit is further configured to: map the at least one visual element to the at least one audio element based on visual metadata associated with the at least one visual element and audio metadata associated with the at least one audio element.

[0172] Embodiment 2A. A device according to embodiment 1A, wherein the visual metadata includes the position of the at least one visual element in the extended reality scene, wherein the audio metadata includes the position of the at least one audio element in the extended reality scene, and wherein the processing circuit is configured to modify the position of the at least one audio element based on the position of the at least one visual element to obtain a modified position of the at least one audio element in the extended reality scene.

[0173] Embodiment 3A. The apparatus of Embodiment 2A, wherein the modified pose of the at least one audio element is different from the pose of the at least one audio element.

[0174] Embodiment 4A. The apparatus of any combination of Embodiments 2A and 3A, wherein the modified pose of the at least one audio element differs from the pose of the at least one audio element in terms of a rotation angle.

[0175] Embodiment 5A. The apparatus of any combination of Embodiments 2A through 4A, wherein the modified pose of the at least one audio element differs from the pose of the at least one audio element in terms of a translation distance.

[0176] Embodiment 6A. A device according to any combination of Embodiments 1A to 5A, wherein the at least one audio element includes a first audio element and a second audio element, wherein the processing circuit is configured to map the at least one visual element to the first audio element based on visual metadata associated with the at least one visual element and audio metadata associated with the first audio element, wherein the processing circuit is further configured to: determine that none of the at least one visual element is mapped to the second audio element; and based on the audio metadata associated with the second audio element, render the second audio element to the one or more speaker feeds.

[0177] Embodiment 7A. A device according to any combination of Embodiments 1A to 6A, wherein the visual metadata includes an identifier that uniquely identifies the at least one visual element, wherein the audio metadata includes an identifier that uniquely identifies the at least one audio element, and wherein the processing circuit is configured to map the at least one visual element to the at least one audio element based on the identifier that uniquely identifies the at least one visual element and the identifier that uniquely identifies the at least one audio element.

[0178] Embodiment 8A. A device according to Embodiment 7A, wherein the identifier that uniquely identifies the at least one visual element includes one or more of a visual element identifier and a visual element name, and wherein the identifier that uniquely identifies the at least one audio element includes one or more of an audio element identifier and an audio element name.

[0179] Embodiment 9A. The device of any combination of Embodiments 1A to 8A, further comprising one or more speakers configured to reproduce a sound field based on the one or more speaker feeds.

[0180] Embodiment 10A. A device according to any combination of Embodiments 1A to 9A, wherein the processing circuit is further configured to execute a scene manager and an audio unit, wherein the scene manager is configured to map the at least one visual element to the at least one audio element based on the visual metadata associated with the at least one visual element and the audio metadata associated with the at least one audio element, and wherein the audio unit is configured to render the at least one audio element to the one or more speaker feeds based on the modified audio metadata.

[0181] Embodiment 11A. A device according to embodiment 10A, wherein the scene manager is configured to modify the audio metadata based on the mapping of the at least one visual element to the at least one audio element to obtain modified audio metadata, and wherein the scene manager is further configured to register a callback with the audio unit, and the audio unit is configured to request the modified audio metadata through the callback before rendering the at least one audio element.

[0182] Embodiment 12A. The device of Embodiment 11A, wherein the scene manager registers the callback via an application programming interface exposed by the audio unit.

[0183] Embodiment 13A. The device of any combination of Embodiments 11A and 12A, wherein the audio unit is configured to request the modified audio metadata in response to the callback and before rendering the at least one audio element.

[0184] Embodiment 14A. The device of any combination of Embodiments 11A to 13A, wherein the audio unit is configured to request the modified audio metadata in response to the callback and before rendering each frame of audio data of the at least one audio element.

[0185] Embodiment 15A. A device according to embodiment 10A, wherein the processing circuit is further configured to execute an audio processing unit, wherein the scene manager is configured to configure the audio processing unit to modify the audio metadata based on the mapping of the at least one visual element to the at least one audio element to obtain the modified audio metadata, and wherein the audio processing unit is configured to: replace the audio metadata in the audio bitstream with the modified audio metadata based on the configuration; and output the audio bitstream to the audio unit.

[0186] Embodiment 16A. A device according to Embodiment 15A, wherein the scene manager is configured to configure the audio processing unit via an application programming interface exposed by the audio processing unit to modify the audio metadata based on the mapping of the at least one visual element to the at least one audio element to obtain the modified audio metadata.

[0187] Embodiment 17A. A device according to embodiment 10A, wherein the scene manager is further configured to: construct a scene graph based on the mapping of the at least one visual element to the at least one audio element, the scene graph comprising a parent node representing the at least one visual element and a child node that depends on the parent node and represents the at least one audio element; and modify the audio metadata based on the scene graph to obtain the modified audio metadata.

[0188] Embodiment 18A. The device of Embodiment 17A, wherein the scene manager is further configured to output the modified audio metadata to the audio unit.

[0189] Embodiment 19A. The device of Embodiment 17A, wherein the scene manager is further configured to output the modified audio metadata to the audio unit via an application programming interface exposed by the audio unit.

[0190] Embodiment 20A. A method for processing at least one audio element, the method comprising: modifying audio metadata associated with the at least one audio element based on at least one visual element to the at least one audio element to obtain modified audio metadata; rendering the at least one audio element to one or more speaker feeds based on the modified audio metadata; and outputting the one or more speaker feeds.

[0191] Embodiment 21.5A. The method of Embodiment 1A, further comprising mapping the at least one visual element to the at least one audio element based on visual metadata associated with the at least one visual element and audio metadata associated with the at least one audio element.

[0192] Embodiment 21A. A method according to embodiment 20A, wherein the visual metadata includes the position of the at least one visual element in the extended reality scene, wherein the audio metadata includes the position of the at least one audio element in the extended reality scene, and wherein modifying the audio metadata includes modifying the position of the at least one audio element based on the position of the at least one visual element to obtain a modified position of the at least one audio element in the extended reality scene.

[0193] Embodiment 22A. The method of Embodiment 21A, wherein the modified pose of the at least one audio element is different from the pose of the at least one audio element.

[0194] Embodiment 23A. The method of any combination of Embodiments 21A and 22A, wherein the modified pose of the at least one audio element differs from the pose of the at least one audio element in terms of a rotation angle.

[0195] Embodiment 24A. The method of any combination of Embodiments 21A to 23A, wherein the modified pose of the at least one audio element differs from the pose of the at least one audio element in terms of a translation distance.

[0196] Embodiment 25A. A method according to any combination of Embodiments 20A to 24A, wherein the at least one audio element includes a first audio element and a second audio element, wherein the method further includes: mapping the at least one visual element to the first audio element based on visual metadata associated with the at least one visual element and audio metadata associated with the first audio element; determining that none of the at least one visual element is mapped to the second audio element; and rendering the second audio element to the one or more speaker feeds based on the audio metadata associated with the second audio element.

[0197] Embodiment 26A. A method according to any combination of Embodiments 20A to 25A, wherein the visual metadata includes an identifier that uniquely identifies the at least one visual element, wherein the audio metadata includes an identifier that uniquely identifies the at least one audio element, and wherein the method further includes mapping the at least one visual element to the at least one audio element based on the identifier that uniquely identifies the at least one visual element and the identifier that uniquely identifies the at least one audio element.

[0198] Embodiment 27A. A method according to Embodiment 26A, wherein the identifier that uniquely identifies the at least one visual element includes one or more of a visual element identifier and a visual element name, and wherein the identifier that uniquely identifies the at least one audio element includes one or more of an audio element identifier and an audio element name.

[0199] Embodiment 28A. The method of any combination of Embodiments 20A to 27A, further comprising reproducing a sound field by one or more speakers and based on the one or more speaker feeds.

[0200] Embodiment 29A. A method according to any combination of Embodiments 20A to 28A, the method also including executing a scene manager and an audio subsystem, wherein the scene manager is configured to map the at least one visual element to the at least one audio element based on the visual metadata associated with the at least one visual element and the audio metadata associated with the at least one audio element, and wherein the audio unit is configured to render the at least one audio element to the one or more speaker feeds based on the modified audio metadata.

[0201] Embodiment 30A. A method according to Embodiment 29A, wherein modifying the audio metadata includes executing the scene manager to modify the audio metadata based on the mapping of the at least one visual element to the at least one audio element to obtain the modified audio metadata, and wherein the method also includes registering a callback with the audio unit by the scene manager, and the audio unit is configured to request the modified audio metadata through the callback before rendering the at least one audio element.

[0202] Embodiment 31A. The method of Embodiment 30A, further comprising registering the callback via an application programming interface exposed by the audio unit.

[0203] Embodiment 32A. The method of any combination of Embodiments 30A and 31A, further comprising: requesting the modified audio metadata in response to the callback and before rendering the at least one audio element.

[0204] Embodiment 33A. A method according to any combination of Embodiments 30A to 32A, the method further comprising: requesting the modified audio metadata in response to the callback and before rendering each frame of audio data of the at least one audio element.

[0205] Embodiment 34A. A method according to Embodiment 29A, wherein the method further includes configuring the audio processing unit to modify the audio metadata based on the mapping of the at least one visual element to the at least one audio element to obtain the modified audio metadata, and wherein the method further includes: replacing the audio metadata in the audio bitstream with the modified audio metadata based on the configuration; and outputting the audio bitstream to the audio unit.

[0206] Embodiment 35A. A method according to Embodiment 34A, wherein configuring the audio processing unit includes configuring the audio processing unit via an application programming interface exposed by the audio processing unit to modify the audio metadata based on the mapping of the at least one visual element to the at least one audio element to obtain the modified audio metadata.

[0207] Embodiment 36A. According to the method described in Embodiment 29A, the method also includes: constructing a scene graph based on the mapping of the at least one visual element to the at least one audio element, the scene graph including a parent node representing the at least one visual element and a child node that depends on the parent node and represents the at least one audio element; and modifying the audio metadata based on the scene graph to obtain the modified audio metadata.

[0208] Embodiment 37A. The method of Embodiment 36A, further comprising outputting the modified audio metadata to the audio unit.

[0209] Embodiment 38A. The method of Embodiment 36A, further comprising outputting the modified audio metadata to the audio unit via an application programming interface exposed by the audio unit.

[0210] Embodiment 39A. A non-transitory computer-readable storage medium having instructions stored thereon, which, when executed, cause one or more processors to: modify audio metadata associated with at least one audio element based on at least one visual element to obtain modified audio metadata; render the at least one audio element to one or more speaker feeds based on the modified audio metadata; and output the one or more speaker feeds.

[0211] Embodiment 1B. A device configured to process an audio bitstream, the device comprising: a memory, the memory being configured to store a visual bitstream representing at least one visual element in an extended reality scene and the audio bitstream representing at least one audio element in the extended reality scene; and a processing circuit, the processing circuit being coupled to the memory and configured to execute a scene manager and an audio unit, wherein the scene manager is configured to: modify audio metadata associated with the at least one audio element based on the at least one visual element to the at least one audio element to obtain modified audio metadata; and register a callback with the audio unit, the audio unit being configured to request the modified audio metadata through the callback before rendering the at least one audio element, and wherein the audio unit is configured to: render the at least one audio element to one or more speaker feeds based on the modified audio metadata; and output the one or more speaker feeds.

[0212] Embodiment 1.5B. The apparatus of Embodiment 1B, wherein the scene is configured to map the at least one visual element to the at least one audio element based on visual metadata associated with the at least one visual element and audio metadata associated with the at least one audio element.

[0213] Embodiment 2B. A device according to embodiment 1B, wherein the visual metadata includes the position of the at least one visual element in the extended reality scene, wherein the audio metadata includes the position of the at least one audio element in the extended reality scene, and wherein the scene manager is configured to modify the position of the at least one audio element based on the position of the at least one visual element to obtain a modified position of the at least one audio element in the extended reality scene.

[0214] Embodiment 3B. The apparatus of Embodiment 2B, wherein the modified position of the at least one audio element is different from the position of the at least one audio element.

[0215] Embodiment 4B. The device of any combination of Embodiments 2B and 3B, wherein the modified position of the at least one audio element differs from the position of the at least one audio element in terms of a rotational angle.

[0216] Embodiment 5B. The apparatus of any combination of Embodiments 2B through 4B, wherein the modified position of the at least one audio element differs from the position of the at least one audio element in terms of a translation distance.

[0217] Embodiment 6B. The apparatus of any combination of Embodiments 1B to 5B, wherein the at least one audio element comprises a first audio element and a second audio element, wherein the scene manager is configured to map the at least one visual element to the first audio element based on visual metadata associated with the at least one visual element and audio metadata associated with the first audio element, wherein the scene manager is configured to map the at least one visual element to the first audio element based on visual metadata associated with the at least one visual element and audio metadata associated with the first audio element The method is further configured to: determine that none of the at least one visual element maps to the second audio element; and render the second audio element to the one or more speaker feeds based on the audio metadata associated with the second audio element.

[0218] Embodiment 7B. A device according to any combination of Embodiments 1B to 6B, wherein the visual metadata includes an identifier that uniquely identifies the at least one visual element, wherein the audio metadata includes an identifier that uniquely identifies the at least one audio element, and wherein the scene manager is configured to map the at least one visual element to the at least one audio element based on the identifier that uniquely identifies the at least one visual element and the identifier that uniquely identifies the at least one audio element.

[0219] Embodiment 8B. A device according to Embodiment 7B, wherein the identifier that uniquely identifies the at least one visual element includes one or more of a visual element identifier and a visual element name, and wherein the identifier that uniquely identifies the at least one audio element includes one or more of an audio element identifier and an audio element name.

[0220] Embodiment 9B. The device of any combination of Embodiments 1B to 8B, further comprising one or more speakers configured to reproduce a sound field based on the one or more speaker feeds.

[0221] Embodiment 10B. The device of any combination of Embodiments 1B through 9B, wherein the scene manager registers the callback via an application programming interface exposed by the audio unit.

[0222] Embodiment 11B. The device of any combination of Embodiments 1B to 10B, wherein the audio unit is configured to request the modified audio metadata in response to the callback and before rendering the at least one audio element.

[0223] Embodiment 12B. The device of any combination of Embodiments 1B to 11B, wherein the audio unit is configured to request the modified audio metadata in response to the callback and before rendering each frame of audio data of the at least one audio element.

[0224] Embodiment 13B. A method for processing at least one audio element, the method comprising: modifying audio metadata associated with the at least one audio element by a scene manager executed by a processing circuit and based on at least one visual element and the at least one audio element to obtain modified audio metadata; rendering the at least one audio element to one or more speaker feeds by an audio unit executed by the processing circuit and based on the modified audio metadata; and outputting the one or more speaker feeds by the audio unit.

[0225] Embodiment 13.5B. The method according to Embodiment 13B further comprises: mapping, by the scene manager and based on visual metadata associated with the at least one visual element and the audio metadata associated with the at least one audio element, the at least one visual element to the at least one audio element.

[0226] Embodiment 14B. A method according to Embodiment 13B, wherein the visual metadata includes the position of the at least one visual element in the extended reality scene, wherein the audio metadata includes the position of the at least one audio element in the extended reality scene, and wherein modifying the audio metadata includes modifying the position of the at least one audio element based on the position of the at least one visual element to obtain a modified position of the at least one audio element in the extended reality scene.

[0227] Embodiment 15B. The method of Embodiment 14B, wherein the modified position of the at least one audio element is different from the position of the at least one audio element.

[0228] Embodiment 16B. The method of any combination of Embodiments 14B and 15B, wherein the modified position of the at least one audio element differs from the position of the at least one audio element in terms of a rotation angle.

[0229] Embodiment 17B. The method of any combination of Embodiments 14B through 16B, wherein the modified position of the at least one audio element differs from the position of the at least one audio element in terms of a translation distance.

[0230] Embodiment 18B. A method according to any combination of Embodiments 13B to 17B, wherein the at least one audio element includes a first audio element and a second audio element, wherein the method further includes: mapping the at least one visual element to the first audio element based on visual metadata associated with the at least one visual element and audio metadata associated with the first audio element; determining that none of the at least one visual element is mapped to the second audio element; and rendering the second audio element to the one or more speaker feeds based on the audio metadata associated with the second audio element.

[0231] Embodiment 19B. A method according to any combination of Embodiments 13B to 18B, wherein the visual metadata includes an identifier that uniquely identifies the at least one visual element, wherein the audio metadata includes an identifier that uniquely identifies the at least one audio element, and wherein the method further includes mapping the at least one visual element to the at least one audio element based on the identifier that uniquely identifies the at least one visual element and the identifier that uniquely identifies the at least one audio element.

[0232] Embodiment 20B. A method according to Embodiment 19B, wherein the identifier that uniquely identifies the at least one visual element includes one or more of a visual element identifier and a visual element name, and wherein the identifier that uniquely identifies the at least one audio element includes one or more of an audio element identifier and an audio element name.

[0233] Embodiment 21B. The method of any combination of Embodiments 13B to 20B, further comprising reproducing a sound field by one or more speakers and based on the one or more speaker feeds.

[0234] Embodiment 22B. The method of any combination of Embodiments 13B to 21B, further comprising registering the callback via an application programming interface exposed by the audio unit.

[0235] Embodiment 23B. The method of any combination of Embodiments 13B to 22B, further comprising: requesting the modified audio metadata in response to the callback and before rendering the at least one audio element.

[0236] Embodiment 24B. The method of any combination of Embodiments 13B to 23B, further comprising requesting the modified audio metadata in response to the callback and prior to rendering each frame of audio data of the at least one audio element.

[0237] Embodiment 25B. A non-transitory computer-readable storage medium having instructions stored thereon, which, when executed, cause one or more processors to: execute a scene manager, the scene manager being configured to modify audio metadata associated with at least one audio element based on at least one visual element and at least one audio element to obtain modified audio metadata; and execute an audio unit, the audio unit being configured to: render the at least one audio element to one or more speaker feeds based on the modified audio metadata; and output the one or more speaker feeds.

[0238] Embodiment 1C. A device configured to process an audio bitstream, the device comprising: a memory configured to store a visual bitstream representing at least one visual element in an extended reality scene and an audio bitstream representing at least one audio element in the extended reality scene; and a processing circuit coupled to the memory and configured to execute a scene manager, an audio processing unit, and an audio unit, wherein the scene manager is configured to: modify audio metadata associated with the at least one audio element based on the at least one visual element and the at least one audio element to obtain modified audio metadata; and configure the audio processing unit to modify the audio metadata based on a mapping of the at least one visual element to the at least one audio element to obtain the modified audio metadata, wherein the audio processing unit is configured to: replace the audio metadata in the audio bitstream with the modified audio metadata based on the configuration; and output the audio bitstream to the audio unit, and wherein the audio unit is configured to: render the at least one audio element to one or more speaker feeds based on the modified audio metadata; and output the one or more speaker feeds.

[0239] Embodiment 2C. A device according to Embodiment 1C, wherein the visual metadata includes the position of the at least one visual element in the extended reality scene, wherein the audio metadata includes the position of the at least one audio element in the extended reality scene, and wherein the scene manager is configured to modify the position of the at least one audio element based on the position of the at least one visual element to obtain a modified position of the at least one audio element in the extended reality scene.

[0240] Embodiment 3C. The apparatus of Embodiment 2C, wherein the modified position of the at least one audio element is different from the position of the at least one audio element.

[0241] Embodiment 4C. The apparatus of any combination of Embodiments 2C and 3C, wherein the modified position of the at least one audio element differs from the position of the at least one audio element in terms of a rotational angle.

[0242] Embodiment 5C. The apparatus of any combination of Embodiments 2C through 4C, wherein the modified position of the at least one audio element differs from the position of the at least one audio element in terms of a translation distance.

[0243] Embodiment 6C. A device according to any combination of Embodiments 1C to 5C, wherein the at least one audio element includes a first audio element and a second audio element, wherein the scene manager is further configured to: map the at least one visual element to the first audio element based on visual metadata associated with the at least one visual element and the audio metadata associated with the first audio element; determine that none of the at least one visual element is mapped to the second audio element; and render the second audio element to the one or more speaker feeds based on the audio metadata associated with the second audio element.

[0244] Embodiment 7C. A device according to any combination of Embodiments 1C to 6C, wherein the visual metadata includes an identifier that uniquely identifies the at least one visual element, wherein the audio metadata includes an identifier that uniquely identifies the at least one audio element, and wherein the scene manager is configured to map the at least one visual element to the at least one audio element based on the identifier that uniquely identifies the at least one visual element and the identifier that uniquely identifies the at least one audio element.

[0245] Embodiment 8C. A device according to Embodiment 7C, wherein the identifier that uniquely identifies the at least one visual element includes one or more of a visual element identifier and a visual element name, and wherein the identifier that uniquely identifies the at least one audio element includes one or more of an audio element identifier and an audio element name.

[0246] Embodiment 9C. The device of any combination of Embodiments 1C to 8C, further comprising one or more speakers configured to reproduce a sound field based on the one or more speaker feeds.

[0247] Embodiment 10C. A device according to any combination of Embodiments 1C to 9C, wherein the scene manager is configured to configure the audio processing unit via an application programming interface exposed by the audio processing unit to modify the audio metadata based on the mapping of the at least one visual element to the at least one audio element to obtain the modified audio metadata.

[0248] Embodiment 11C. A method for processing at least one audio element, the method comprising: modifying audio metadata associated with the at least one audio element by a scene manager executed by a processing circuit and based on at least one visual element and the at least one audio element to obtain modified audio metadata; configuring an audio processing unit by the scene manager to modify the audio metadata based on the mapping of the at least one visual element to the at least one audio element to obtain the modified audio metadata; replacing the audio metadata in the audio bitstream with the modified audio metadata by the audio processing unit and based on the configuration; and outputting the audio bitstream to an audio unit executed by the processing circuit by the audio processing unit; rendering the at least one audio element to one or more speaker feeds by the audio unit and based on the modified audio metadata; and outputting the one or more speaker feeds by the audio unit.

[0249] Embodiment 11.5C. A method according to Embodiment 11C, the method further comprising: mapping the at least one visual element to the at least one audio element by the scene manager and based on the visual metadata associated with the at least one visual element and the audio metadata associated with the at least one audio element.

[0250] Embodiment 12C. A method according to Embodiment 11C, wherein the visual metadata includes the position of the at least one visual element in the extended reality scene, wherein the audio metadata includes the position of the at least one audio element in the extended reality scene, and wherein modifying the audio metadata includes modifying the position of the at least one audio element based on the position of the at least one visual element to obtain a modified position of the at least one audio element in the extended reality scene.

[0251] Embodiment 13C. The method of Embodiment 12C, wherein the modified position of the at least one audio element is different from the position of the at least one audio element.

[0252] Embodiment 14C. The method of any combination of Embodiments 12C and 13C, wherein the modified position of the at least one audio element differs from the position of the at least one audio element in terms of a rotation angle.

[0253] Embodiment 15C. The method of any combination of Embodiments 12C through 14C, wherein the modified position of the at least one audio element differs from the position of the at least one audio element in terms of a translation distance.

[0254] Embodiment 16C. A method according to any combination of Embodiments 11C to 15C, wherein the at least one audio element includes a first audio element and a second audio element, wherein the method further includes: mapping the at least one visual element to the first audio element based on visual metadata associated with the at least one visual element and the audio metadata associated with the first audio element; determining that none of the at least one visual element is mapped to the second audio element; and rendering the second audio element to the one or more speaker feeds based on the audio metadata associated with the second audio element.

[0255] Embodiment 17C. A method according to any combination of Embodiments 11C to 16C, wherein the visual metadata includes an identifier that uniquely identifies the at least one visual element, wherein the audio metadata includes an identifier that uniquely identifies the at least one audio element, and wherein the method further includes mapping the at least one visual element to the at least one audio element based on the identifier that uniquely identifies the at least one visual element and the identifier that uniquely identifies the at least one audio element.

[0256] Embodiment 18C. A method according to Embodiment 17C, wherein the identifier that uniquely identifies the at least one visual element includes one or more of a visual element identifier and a visual element name, and wherein the identifier that uniquely identifies the at least one audio element includes one or more of an audio element identifier and an audio element name.

[0257] Embodiment 19C. The method of any combination of Embodiments 11C to 18C, further comprising reproducing a sound field by one or more speakers and based on the one or more speaker feeds.

[0258] Embodiment 20C. A method according to any combination of Embodiments 11C to 19C, wherein configuring the audio processing unit includes configuring the audio processing unit via an application programming interface exposed by the audio processing unit to modify the audio metadata based on the mapping of the at least one visual element to the at least one audio element to obtain the modified audio metadata.

[0259] Embodiment 21C. A non-transitory computer-readable storage medium having instructions stored thereon, which, when executed, cause one or more processors to: execute a scene manager, the scene manager being configured to: modify audio metadata based on at least one visual element and at least one audio element to obtain modified audio metadata, and configure an audio processing unit to modify the audio metadata based on a mapping of the at least one visual element to the at least one audio element to obtain the modified audio metadata; execute the audio processing unit to replace the audio metadata in an audio bitstream with the modified audio metadata based on the configuration, and output the audio bitstream to an audio unit; and execute the audio unit, the audio unit being configured to render the at least one audio element to one or more speaker feeds based on the modified audio metadata, and output the one or more speaker feeds.

[0260] Embodiment 1D. A device configured to process an audio bitstream, the device comprising: a memory, the memory being configured to store a visual bitstream representing at least one visual element in an extended reality scene and an audio bitstream representing at least one audio element in the extended reality scene, the audio bitstream comprising audio metadata; and a processing circuit, the processing circuit being coupled to the memory and configured to execute a scene manager and an audio unit, wherein the scene manager is configured to: construct a scene graph based on the at least one visual element and the at least one audio element, the scene graph comprising a parent node representing the at least one visual element and a child node that depends on the parent node and represents the at least one audio element; and modify the audio metadata based on the scene graph to obtain modified audio metadata, and wherein the audio unit is configured to: render the at least one audio element to one or more speaker feeds based on the modified audio metadata; and output the one or more speaker feeds.

[0261] Embodiment 1.5D. A device according to Embodiment 1D, wherein the scene is further configured to: map the at least one visual element to the at least one audio element based on visual metadata associated with the at least one visual element and audio metadata associated with the at least one audio element.

[0262] Embodiment 2D. A device according to Embodiment 1D, wherein the visual metadata includes the position of the at least one visual element in the extended reality scene, wherein the audio metadata includes the position of the at least one audio element in the extended reality scene, and wherein the scene manager is configured to modify the position of the at least one audio element based on the position of the at least one visual element to obtain a modified position of the at least one audio element in the extended reality scene.

[0263] Embodiment 3D. The apparatus of Embodiment 2D, wherein the modified position of the at least one audio element is different from the position of the at least one audio element.

[0264] Embodiment 4D. The apparatus of any combination of Embodiments 2D and 3D, wherein the modified position of the at least one audio element differs from the position of the at least one audio element in terms of a rotational angle.

[0265] Embodiment 5D. The apparatus of any combination of Embodiments 2D through 4D, wherein the modified position of the at least one audio element differs from the position of the at least one audio element in terms of a translation distance.

[0266] Embodiment 6D. A device according to any combination of Embodiments 1D to 5D, wherein the at least one audio element includes a first audio element and a second audio element, wherein the scene manager is further configured to: map the at least one visual element to the first audio element based on visual metadata associated with the at least one visual element and audio metadata associated with the first audio element; determine that none of the at least one visual element is mapped to the second audio element; and render the second audio element to the one or more speaker feeds based on the audio metadata associated with the second audio element.

[0267] Embodiment 7D. A device according to any combination of Embodiments 1D to 6D, wherein the visual metadata includes an identifier that uniquely identifies the at least one visual element, wherein the audio metadata includes an identifier that uniquely identifies the at least one audio element, and wherein the scene manager is configured to map the at least one visual element to the at least one audio element based on the identifier that uniquely identifies the at least one visual element and the identifier that uniquely identifies the at least one audio element.

[0268] Embodiment 8D. A device according to Embodiment 7D, wherein the identifier that uniquely identifies the at least one visual element includes one or more of a visual element identifier and a visual element name, and wherein the identifier that uniquely identifies the at least one audio element includes one or more of an audio element identifier and an audio element name.

[0269] Embodiment 9D. The device of any combination of Embodiments 1D to 8D, further comprising one or more speakers configured to reproduce a sound field based on the one or more speaker feeds.

[0270] Embodiment 10D. The device of any combination of Embodiments 1D to 9D, wherein the scene manager is further configured to output the modified audio metadata to the audio unit.

[0271] Embodiment 11D. The device of any combination of Embodiments 1D to 9D, wherein the scene manager is further configured to output the modified audio metadata to the audio unit via an application programming interface exposed by the audio unit.

[0272] Embodiment 12D. A method for processing at least one audio element, the method comprising: constructing a scene graph through a scene manager executed by a processing circuit and based on at least one visual element and the at least one audio element, the scene graph comprising a parent node representing the at least one visual element and a child node that depends on the parent node and represents the at least one audio element; modifying audio metadata associated with the at least one audio element by the scene manager and based on the scene graph to obtain modified audio metadata; rendering the at least one audio element to one or more speaker feeds through an audio unit executed by the processing circuit and based on the modified audio metadata; and outputting the one or more speaker feeds by the audio unit.

[0273] Embodiment 12.5D. According to the method of embodiment 12D, the method also includes: mapping the at least one visual element to the at least one audio element by the scene manager and based on the visual metadata associated with the at least one visual element and the audio metadata associated with the at least one audio element.

[0274] Embodiment 13D. A method according to Embodiment 12D, wherein the visual metadata includes the position of the at least one visual element in the extended reality scene, wherein the audio metadata includes the position of the at least one audio element in the extended reality scene, and wherein modifying the audio metadata includes modifying the position of the at least one audio element based on the position of the at least one visual element to obtain a modified position of the at least one audio element in the extended reality scene.

[0275] Embodiment 14D. The method of Embodiment 13D, wherein the modified position of the at least one audio element is different from the position of the at least one audio element.

[0276] Embodiment 15D. The method of any combination of Embodiments 13D and 14D, wherein the modified position of the at least one audio element differs from the position of the at least one audio element in terms of a rotation angle.

[0277] Embodiment 16D. The method of any combination of Embodiments 13D through 15D, wherein the modified position of the at least one audio element differs from the position of the at least one audio element in terms of a translation distance.

[0278] Embodiment 17D. A method according to any combination of Embodiments 12D to 16D, wherein the at least one audio element includes a first audio element and a second audio element, and wherein the method further includes: mapping the at least one visual element to the first audio element based on visual metadata associated with the at least one visual element and audio metadata associated with the first audio element; and determining that none of the at least one visual element is mapped to the second audio element; and rendering the second audio element to the one or more speaker feeds based on the audio metadata associated with the second audio element.

[0279] Embodiment 18D. A method according to any combination of Embodiments 12D to 17D, wherein the visual metadata includes an identifier that uniquely identifies the at least one visual element, wherein the audio metadata includes an identifier that uniquely identifies the at least one audio element, and wherein the method further includes mapping the at least one visual element to the at least one audio element based on the identifier that uniquely identifies the at least one visual element and the identifier that uniquely identifies the at least one audio element.

[0280] Embodiment 19D. A method according to Embodiment 18D, wherein the identifier that uniquely identifies the at least one visual element includes one or more of a visual element identifier and a visual element name, and wherein the identifier that uniquely identifies the at least one audio element includes one or more of an audio element identifier and an audio element name.

[0281] Embodiment 20D. The method of any combination of Embodiments 12D to 19D, further comprising reproducing a sound field by one or more speakers and based on the one or more speaker feeds.

[0282] Embodiment 21D. The method of any combination of Embodiments 12D to 20D, further comprising outputting the modified audio metadata to the audio unit.

[0283] Embodiment 22D. The method of any combination of Embodiments 12D to 20D, further comprising outputting the modified audio metadata to the audio unit via an application programming interface exposed by the audio unit.

[0284] Embodiment 23D. A non-transitory computer-readable storage medium having instructions stored thereon, which, when executed, cause one or more processors to: execute a scene manager, the scene manager being configured to: construct a scene graph by the scene manager and based on at least one visual element and at least one audio element, the scene graph comprising a parent node representing the at least one visual element and a child node that depends on the parent node and represents the at least one audio element; and modify audio metadata by the scene manager and based on the scene graph to obtain modified audio metadata; and execute an audio unit, the audio unit being configured to: render the at least one audio element to one or more speaker feeds based on the modified audio metadata; and output the one or more speaker feeds.

[0285] Embodiment 1E. A device configured to process a bitstream, the device comprising: a memory configured to store the bitstream representing at least one audio element in an extended reality scene and audio descriptive information associated with the at least one audio element; and a processing circuit coupled to the memory and configured to execute a scene manager and an audio unit, wherein the scene manager is configured to: construct a scene graph based on the at least one audio element, the scene graph comprising at least one node representing the at least one audio element; and modify the audio descriptive information based on the scene graph to obtain modified audio descriptive information, and wherein the audio unit is configured to: render the at least one audio element to one or more speaker feeds based on the modified audio descriptive information; and output the one or more speaker feeds.

[0286] Embodiment 2E. A device according to Embodiment 1E, wherein the scene manager is further configured to obtain at least one visual element, and wherein the scene manager is configured to construct the scene graph based on the at least one audio element and the at least one video element, the scene graph including a parent node representing the at least one visual element and a child node that is dependent on the parent node and represents the at least one audio element.

[0287] Embodiment 3E. The apparatus of Embodiment 2E, wherein the scene manager is configured to align the at least one visual element and the at least one audio element when constructing the scene graph.

[0288] Embodiment 4E. The apparatus of Embodiments 1E to 3E, wherein the scene manager is further configured to update the scene graph to add, remove, or edit the at least one node representing the at least one audio element.

[0289] Embodiment 5E. The apparatus of Embodiment 2E, wherein the scene manager is further configured to map the at least one visual element to the at least one audio element based on visual descriptive information associated with the at least one visual element and the audio descriptive information associated with the at least one audio element;

[0290] Embodiment 6E. A device according to Embodiment 5E, wherein the visual descriptive information includes the position of the at least one visual element in the extended reality scene, wherein the audio descriptive information includes the position of the at least one audio element in the extended reality scene, and wherein the scene manager is configured to modify the position of the at least one audio element based on the position of the at least one visual element to obtain a modified position of the at least one audio element in the extended reality scene.

[0291] Embodiment 7E. The apparatus of Embodiment 6E, wherein the modified position of the at least one audio element is different from the position of the at least one audio element.

[0292] Embodiment 8E. The apparatus of any combination of Embodiments 6E and 7E, wherein the modified position of the at least one audio element differs from the position of the at least one audio element in terms of a rotational angle.

[0293] Embodiment 9E. The apparatus of any combination of Embodiments 6E to 8E, wherein the modified position of the at least one audio element differs from the position of the at least one audio element in terms of a translation distance.

[0294] Embodiment 10E. A device according to any combination of Embodiments 5E to 9E, wherein the at least one audio element includes a first audio element and a second audio element, wherein the scene manager is further configured to: map the at least one visual element to the first audio element based on visual descriptive information associated with the at least one visual element and audio descriptive information associated with the first audio element; determine that none of the at least one visual element is mapped to the second audio element; and render the second audio element to the one or more speaker feeds based on the audio descriptive information associated with the second audio element.

[0295] Embodiment 11E. A device according to any combination of Embodiments 5E to 10E, wherein the visual descriptive information includes an identifier that uniquely identifies the at least one visual element, wherein the audio descriptive information includes an identifier that uniquely identifies the at least one audio element, and wherein the scene manager is configured to map the at least one visual element to the at least one audio element based on the identifier that uniquely identifies the at least one visual element and the identifier that uniquely identifies the at least one audio element.

[0296] Embodiment 12E. A device according to Embodiment 11E, wherein the identifier that uniquely identifies the at least one visual element includes one or more of a visual element identifier and a visual element name, and wherein the identifier that uniquely identifies the at least one audio element includes one or more of an audio element identifier and an audio element name.

[0297] Embodiment 13E. The device of any combination of Embodiments 5E to 12E, further comprising one or more speakers configured to reproduce a sound field based on the one or more speaker feeds.

[0298] Embodiment 14E. The apparatus of any combination of Embodiments 5E to 13E, wherein the scene manager is further configured to output the modified audio metadata to the audio unit.

[0299] Embodiment 15E. A device according to any combination of Embodiments 5E to 13E, wherein the scene manager is further configured to output the modified audio metadata to the audio unit via an application programming interface exposed by the audio unit.

[0300] Embodiment 15.5E. The apparatus of any combination of Embodiments 1E to 15E, wherein the bitstream is sent according to one or more of a wireless network protocol, a personal area network protocol, and a cellular network protocol.

[0301] Embodiment 16E. A method, the method comprising: obtaining a bitstream representing at least one audio element in an extended reality scene and audio descriptive information associated with the at least one audio element; and constructing a scene graph based on the at least one audio element, the scene graph comprising at least one node representing the at least one audio element; and modifying the audio descriptive information based on the scene graph to obtain modified audio descriptive information, and rendering the at least one audio element to one or more speaker feeds based on the modified audio descriptive information; and outputting the one or more speaker feeds.

[0302] Embodiment 17E. The method according to Embodiment 16E further includes obtaining at least one visual element, wherein constructing the scene graph includes constructing the scene graph based on the at least one audio element and the at least one video element, and the scene graph includes a parent node representing the at least one visual element and a child node that depends on the parent node and represents the at least one audio element.

[0303] Embodiment 18E. The method of Embodiment 15E, wherein constructing the scene graph comprises aligning the at least one visual element and the at least one audio element.

[0304] Embodiment 19E. The method according to Embodiments 16E to 18E, further comprising updating the scene graph to add, remove or edit the at least one node representing the at least one audio element.

[0305] Embodiment 20E. The method according to Embodiment 17E further comprises: mapping the at least one visual element to the at least one audio element based on the visual descriptive information associated with the at least one visual element and the audio descriptive information associated with the at least one audio element.

[0306] Embodiment 21E. A method according to embodiment 20E, wherein the visual descriptive information includes the position of the at least one visual element in the extended reality scene, wherein the audio descriptive information includes the position of the at least one audio element in the extended reality scene, and wherein modifying the audio descriptive information includes modifying the position of the at least one audio element based on the position of the at least one visual element to obtain a modified position of the at least one audio element in the extended reality scene.

[0307] Embodiment 22E. The method of Embodiment 21E, wherein the modified position of the at least one audio element is different from the position of the at least one audio element.

[0308] Embodiment 23E. A method as described in any combination of Embodiments 21E and 22E, wherein the modified position of the at least one audio element is different from the position of the at least one audio element in terms of a rotation angle.

[0309] Embodiment 24E. The method of any combination of Embodiments 21E to 23E, wherein the modified position of the at least one audio element differs from the position of the at least one audio element in terms of a translation distance.

[0310] Embodiment 25E. A method according to any combination of Embodiments 20E to 24E, wherein the at least one audio element includes a first audio element and a second audio element, and wherein the method further includes: mapping the at least one visual element to the first audio element based on visual descriptive information associated with the at least one visual element and audio descriptive information associated with the first audio element; determining that none of the at least one visual element is mapped to the second audio element; and rendering the second audio element to the one or more speaker feeds based on the audio descriptive information associated with the second audio element.

[0311] Embodiment 26E. A method according to any combination of Embodiments 20E to 25E, wherein the visual descriptive information includes an identifier that uniquely identifies the at least one visual element, wherein the audio descriptive information includes an identifier that uniquely identifies the at least one audio element, and wherein mapping the at least one visual element to the at least one audio element includes: mapping the at least one visual element to the at least one audio element based on the identifier that uniquely identifies the at least one visual element and the identifier that uniquely identifies the at least one audio element.

[0312] Embodiment 27E. A method according to Embodiment 26E, wherein the identifier that uniquely identifies the at least one visual element includes one or more of a visual element identifier and a visual element name, and wherein the identifier that uniquely identifies the at least one audio element includes one or more of an audio element identifier and an audio element name.

[0313] Embodiment 28E. The method of any combination of Embodiments 20E to 27E, further comprising outputting the modified audio metadata to the audio unit.

[0314] Embodiment 29E. The apparatus of any combination of Embodiments 20E to 27E, the method further comprising outputting the modified audio metadata to the audio unit via an application programming interface exposed by the audio unit.

[0315] Embodiment 29.5E. The apparatus of any combination of Embodiments 20E to 27E, wherein the bitstream is sent according to one or more of a wireless network protocol, a personal area network protocol, and a cellular network protocol.

[0316] Embodiment 30E. A non-transitory computer-readable medium having instructions stored thereon, which, when executed, cause a processing circuit to: obtain a bitstream representing at least one audio element in an augmented reality scene and audio descriptive information associated with the at least one audio element; and construct a scene graph based on the at least one audio element, the scene graph comprising at least one node representing the at least one audio element; and modify the audio descriptive information based on the scene graph to obtain modified audio descriptive information, and render the at least one audio element to one or more speaker feeds based on the modified audio descriptive information; and output the one or more speaker feeds.

[0317] Various examples have been described. These and other examples are within the scope of the following claims.

Claims

1. A device configured to process a bit stream, the device comprising: a memory configured to store the bitstream representing at least one audio element in an augmented reality scene and audio descriptive information associated with the at least one audio element; and a processing circuit coupled to the memory and configured to execute a scene manager and an audio unit, The scene manager is configured as follows: constructing a scene graph based on the at least one audio element, the scene graph comprising at least one node representing the at least one audio element; and modifying the audio descriptive information based on the scene graph to obtain modified audio descriptive information, and The audio unit is configured as follows: rendering the at least one audio element to one or more speaker feeds based on the modified audio descriptive information; as well as The one or more speaker feeds are output.

2. The device according to claim 1, wherein the scene manager is further configured to obtain at least one visual element, and The scene manager is configured to construct the scene graph based on the at least one audio element and the at least one visual element, wherein the scene graph includes a parent node representing the at least one visual element and a child node that depends on the parent node and represents the at least one audio element. 3 . The apparatus of claim 2 , wherein the scene manager is configured to align the at least one visual element and the at least one audio element when constructing the scene graph. 4 . The apparatus of claim 1 , wherein the scene manager is further configured to update the scene graph to add, remove, or edit the at least one node representing the at least one audio element.

5. The device of claim 2, wherein the scene manager is further configured to map the at least one visual element to the at least one audio element based on visual descriptive information associated with the at least one visual element and the audio descriptive information associated with the at least one audio element.

6. The device according to claim 5, wherein the visual descriptive information includes a position of the at least one visual element in the augmented reality scene, wherein the audio descriptive information includes a position of the at least one audio element in the augmented reality scene, and The scene manager is configured to modify the position of the at least one audio element based on the position of the at least one visual element to obtain a modified position of the at least one audio element in the augmented reality scene.

7. The apparatus of claim 6, wherein the modified position of the at least one audio element is different from the position of the at least one audio element.

8. The apparatus of claim 6, wherein the modified position of the at least one audio element differs from the position of the at least one audio element in terms of a rotation angle.

9. The apparatus of claim 6, wherein the modified position of the at least one audio element differs from the position of the at least one audio element in terms of a translation distance.

10. The device according to claim 5, The at least one audio element comprises a first audio element and a second audio element, wherein the scene manager is further configured to: mapping the at least one visual element to the first audio element based on visual descriptive information associated with the at least one visual element and audio descriptive information associated with the first audio element; determining that none of the at least one visual element maps to the second audio element; as well as The second audio element is rendered to the one or more speaker feeds based on the audio descriptive information associated with the second audio element.

11. The device according to claim 5, wherein the visual descriptive information comprises an identifier that uniquely identifies the at least one visual element, wherein the audio descriptive information comprises an identifier that uniquely identifies the at least one audio element, and Wherein the scene manager is configured to map the at least one visual element to the at least one audio element based on the identifier that uniquely identifies the at least one visual element and the identifier that uniquely identifies the at least one audio element.

12. The device according to claim 11, wherein the identifier that uniquely identifies the at least one visual element comprises one or more of a visual element identifier and a visual element name, and The identifier that uniquely identifies the at least one audio element comprises one or more of an audio element identifier and an audio element name.

13. The device of claim 5, further comprising one or more speakers configured to reproduce a sound field based on the one or more speaker feeds.

14. The device of claim 5, wherein the scene manager is further configured to output the modified audio descriptive information to the audio unit.

15. The device of claim 5, wherein the scene manager is further configured to output the modified audio descriptive information to the audio unit via an application programming interface exposed by the audio unit.

16. The apparatus of claim 1, wherein the bitstream is sent according to one or more of a wireless network protocol, a personal area network protocol, and a cellular network protocol.

17. A method comprising: Obtaining a bitstream representing at least one audio element in an augmented reality scene and audio descriptive information associated with the at least one audio element; constructing a scene graph based on the at least one audio element, the scene graph comprising at least one node representing the at least one audio element; modifying the audio descriptive information based on the scene graph to obtain modified audio descriptive information; rendering the at least one audio element to one or more speaker feeds based on the modified audio descriptive information; as well as The one or more speaker feeds are output.

18. The method according to claim 17, further comprising obtaining at least one visual element, Constructing the scene graph includes constructing the scene graph based on the at least one audio element and the at least one visual element, and the scene graph includes a parent node representing the at least one visual element and a child node that depends on the parent node and represents the at least one audio element.

19. The method of claim 18, wherein constructing the scene graph comprises aligning the at least one visual element and the at least one audio element.

20. The method according to claim 18, further comprising: The at least one visual element is mapped to the at least one audio element based on the visual descriptive information associated with the at least one visual element and the audio descriptive information associated with the at least one audio element.

21. The method according to claim 20, wherein the visual descriptive information includes a position of the at least one visual element in the augmented reality scene, wherein the audio descriptive information includes a position of the at least one audio element in the augmented reality scene, and Wherein modifying the audio descriptive information comprises: The position of the at least one audio element is modified based on the position of the at least one visual element to obtain a modified position of the at least one audio element in the augmented reality scene.

22. The method of claim 21, wherein the modified position of the at least one audio element is different from the position of the at least one audio element.

23. The method of claim 21, wherein the modified position of the at least one audio element differs from the position of the at least one audio element in terms of a rotation angle.

24. The method of claim 21, wherein the modified position of the at least one audio element differs from the position of the at least one audio element in terms of a translation distance.

25. The method according to claim 20, wherein the at least one audio element comprises a first audio element and a second audio element, and The method further comprises: mapping the at least one visual element to the first audio element based on visual descriptive information associated with the at least one visual element and audio descriptive information associated with the first audio element; determining that none of the at least one visual element maps to the second audio element; as well as The second audio element is rendered to the one or more speaker feeds based on the audio descriptive information associated with the second audio element.

26. The method according to claim 20, wherein the visual descriptive information comprises an identifier that uniquely identifies the at least one visual element, wherein the audio descriptive information comprises an identifier that uniquely identifies the at least one audio element, and Wherein mapping the at least one visual element to the at least one audio element comprises: The at least one visual element is mapped to the at least one audio element based on the identifier that uniquely identifies the at least one visual element and the identifier that uniquely identifies the at least one audio element.

27. The method according to claim 26, wherein the identifier that uniquely identifies the at least one visual element comprises one or more of a visual element identifier and a visual element name, and The identifier that uniquely identifies the at least one audio element comprises one or more of an audio element identifier and an audio element name.

28. The method according to claim 20, further comprising: The modified audio metadata is output to the audio unit via an application programming interface exposed by the audio unit.

29. The method of claim 20, wherein the bitstream is sent according to one or more of a wireless network protocol, a personal area network protocol, and a cellular network protocol.

30. A non-transitory computer readable medium having stored thereon instructions that, when executed, cause a processing circuit to: Obtaining a bitstream representing at least one audio element in an augmented reality scene and audio descriptive information associated with the at least one audio element; as well as constructing a scene graph based on the at least one audio element, the scene graph comprising at least one node representing the at least one audio element; as well as modifying the audio descriptive information based on the scene graph to obtain modified audio descriptive information, and rendering the at least one audio element to one or more speaker feeds based on the modified audio descriptive information; as well as The one or more speaker feeds are output.

Images