Audio capture and rendering for extended reality experiences

Abstract

In some examples, a content consumer device configured to play one or more of a plurality of audio streams includes a memory configured to store a plurality of audio streams and audio position information associated with the plurality of audio streams, the audio position information representing audio stream coordinates in an acoustic space in which the audio streams were captured or synthesized, or both. Each audio stream represents a sound field. The content consumer device also includes one or more processors coupled to the memory, configured to determine device position information representing device coordinates of the content consumer device in the acoustic space. The one or more processors are configured to select a subset of the plurality of audio streams based on the device position information and the audio position information, and output one or more speaker feeds based on the subset of the plurality of audio streams.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority to U.S. Patent Application No. 16 / 918,441 filed on July 1, 2020, U.S. Provisional Patent Application No. 62 / 870,573 filed on July 3, 2019, and U.S. Provisional Patent Application No. 62 / 992,635 filed on March 20, 2020, the entire contents of each of which are incorporated by reference. Technical Field

[0003] The present invention relates to the processing of media data, such as audio data. Background Art

[0004] Computer-mediated reality systems are being developed to allow computing devices to augment or add to, remove or subtract from, or generally modify an existing reality that a user experiences. Computer-mediated reality systems (which may also be referred to as "extended reality systems" or "XR systems") may include, as a few examples, virtual reality (VR) systems, augmented reality (AR) systems, and mixed reality (MR) systems. The perceived success of a computer-mediated reality system generally relates to the ability of such a computer-mediated reality system to provide a truly immersive experience in terms of both the video experience and the audio experience, where the video experience and the audio experience are aligned in a manner that the user desires. Although the human visual system is more sensitive than the human auditory system (e.g., in terms of perceived positioning of various objects within a scene), ensuring an adequate auditory experience is an increasingly important factor in ensuring a truly immersive experience, particularly as the video experience improves to allow better positioning of video objects, which enables users to better identify the source of audio content. Summary of the invention

[0005] The present invention generally relates to the auditory aspects of the user experience of computer-mediated reality systems, including virtual reality (VR), mixed reality (MR), augmented reality (AR), computer vision and graphics systems. Various aspects of the present technology can provide adaptive audio capture and rendering of acoustic space for extended reality systems. As used herein, the acoustic environment is represented as an indoor environment or an outdoor environment, or both an indoor environment and an outdoor environment. The acoustic environment may include one or more subacoustic spaces, and the subacoustic space may include various acoustic elements. Examples of outdoor environments may include cars, buildings, walls, forests, etc. The acoustic space may be an example of an acoustic environment, and may be an indoor space or an outdoor space. As used herein, an audio element is a sound captured by a microphone (e.g., directly from a near-field source or from a reflection of a far-field source, whether real or synthetic), or a previously synthesized sound field, or a monophonic sound synthesized from text to speech, or a reflection of a virtual sound from an object in an acoustic environment. An audio element may also be referred to as a receiver herein.

[0006] In one example, various aspects of the present technology may be directed to a content consumer device configured to play one or more of a plurality of audio streams, the content consumer device comprising: a memory configured to store a plurality of audio streams and audio position information associated with the plurality of audio streams, the audio position information representing audio stream coordinates in an acoustic space of captured audio streams or audio stream coordinates in a virtual acoustic space of synthesized audio streams, or both, each audio stream representing a sound field; and one or more processors coupled to the memory and configured to: determine device position information representing device coordinates of the content consumer device in the acoustic space; select a subset of the plurality of audio streams based on the device position information and the audio position information, the subset of the plurality of audio streams excluding at least one of the plurality of audio streams; and output one or more speaker feeds based on the subset of the plurality of audio streams.

[0007] In another example, various aspects of the present technology may be directed to a method for playing one or more audio streams of a plurality of audio streams, the method comprising: storing, by a memory of a content consumer device, a plurality of audio streams and audio position information associated with the plurality of audio streams, the audio position information representing audio stream coordinates in an acoustic space of captured audio streams or audio stream coordinates in a virtual acoustic space of synthesized audio streams, or both, each audio stream representing a sound field; and determining, by one or more processors of the content consumer device, device position information representing device coordinates of the content consumer device in the acoustic space; selecting, by the one or more processors and based on the device position information and the audio position information, a subset of the plurality of audio streams, the subset of the plurality of audio streams not including at least one audio stream of the plurality of audio streams; and outputting, by the one or more processors and based on the subset of the plurality of audio streams, one or more speaker feeds.

[0008] In another example, various aspects of the present technology may be directed to a content consumer device configured to play one or more audio streams of a plurality of audio streams, the content consumer device comprising: a component for storing a plurality of audio streams and audio position information associated with the plurality of audio streams, the audio position information representing audio stream coordinates in an acoustic space in which the audio streams were captured or audio stream coordinates in a virtual acoustic space in which the audio streams were synthesized, or both, each audio stream representing a sound field; and a component for determining device position information representing device coordinates of the content consumer device in the acoustic space; a component for selecting a subset of the plurality of audio streams based on the device position information and the audio position information, the subset of the plurality of audio streams excluding at least one audio stream of the plurality of audio streams; and a component for outputting one or more speaker feeds based on the subset of the plurality of audio streams.

[0009] In another example, various aspects of the present technology may be directed to a non-transitory computer-readable storage medium having instructions stored thereon, which when executed cause one or more processors of a content consumer device to: store multiple audio streams and audio position information associated with the multiple audio streams, the audio position information representing audio stream coordinates in an acoustic space in which the audio streams were captured or audio stream coordinates in a virtual acoustic space in which the audio streams were synthesized, or both, each audio stream representing a sound field; and determine device position information representing device coordinates of the content consumer device in the acoustic space; based on the device position information and the audio position information, select a subset of the multiple audio streams, the subset of the multiple audio streams excluding at least one audio stream from the multiple audio streams; and output one or more speaker feeds based on the subset of the multiple audio streams.

[0010] The details of one or more examples of the invention are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of various aspects of the technology will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1A-Figure 1C is a schematic diagram illustrating a system that can perform various aspects of the techniques described in this disclosure.

[0012] Figure 2 is a schematic diagram showing an example of a VR device worn by a user.

[0013] Figure 3A-3E is shown in more detail Figure 1A-Figure 1C Schematic diagram of example operations of the stream selection unit shown in the example.

[0014] Figure 4A-4E It is shown Figure 1A-Figure 1C The example operations of the stream selection unit shown in the example perform various aspects of the stream selection technology.

[0015] Figure 5A-5D is a conceptual diagram illustrating an example of alignment according to aspects of the present disclosure.

[0016] Figure 6 is a schematic diagram illustrating an example of a wearable device that can operate according to various aspects of the technology described in this disclosure.

[0017] Fig. 7A and Figure 7B is a schematic diagram illustrating other example systems that can perform various aspects of the techniques described in this disclosure.

[0018] Figure 8 is a block diagram illustrating example components of one or more of the source device and content consumer devices shown in the example of FIG. 1 .

[0019] Fig. 9 An example of a wireless communication system according to aspects of the present disclosure is shown. DETAILED DESCRIPTION

[0020] When rendering an XR scene such as a six degrees of freedom (6DOF) scene, many audio sources available from audio capture devices in the live scene or synthetic sources in the virtual or live scene may need to be balanced between including more or less audio information. This balancing can be performed offline by a mixing engineer, which can be expensive and time consuming. In some cases, the balancing can be performed by a server that communicates with the renderer. In these cases, the balancing may not occur while the renderer is offline, and when the renderer is online, the balancing may cause delays because the renderer may repeatedly communicate with the server to provide information about the orientation of the XR device and receive updated audio information.

[0021] According to the technology of the present disclosure, a content consumer device (such as an XR device) can determine device position information representing device coordinates in an acoustic space. The content consumer device can select a larger or smaller number of audio streams from the multiple audio streams based on whether the proximity distance is less than or greater than a proximity distance threshold, respectively, based on the proximity distance between the device position information and audio position information associated with multiple audio streams, and the audio position information represents the audio stream coordinates in the acoustic space of the captured or synthesized audio streams. The technology of the present disclosure can eliminate the need for balancing by a mixing engineer and repeated communications between the content consumer device and the server.

[0022] In addition, when the user is in an XR scene, the user may want to experience audio from a different listening position than indicated by the device position information. According to the technology of the present disclosure, the user can enter snapping mode. In snapping mode, the user's audio experience can be snapped to the audio stream based on one or more audio source distances and one or more audio source distance thresholds. The audio source distance can be the distance between the device coordinates of the audio stream and the audio stream coordinates. In this way, the user's auditory experience can be improved.

[0023] There are several different ways to represent a sound field. Example formats include channel-based audio formats, object-based audio formats, and scene-based audio formats. Channel-based audio formats refer to 5.1 surround sound formats, 7.1 surround sound formats, 22.2 surround sound formats, or any other channel-based formats that position audio channels to specific locations around the listener in order to recreate the sound field.

[0024] Object-based audio formats may refer to formats in which audio objects (typically encoded using pulse code modulation (PCM) and referred to as PCM audio objects) are specified to represent a sound field. Such audio objects may include metadata identifying the location of the audio object relative to a listener or other reference point in the sound field, so that the audio object can be rendered into one or more speaker channels for playback in an effort to recreate the sound field.

[0025] A scene-based audio format may include a set of hierarchical elements that define a sound field in three dimensions. An example of a set of hierarchical elements is a set of spherical harmonic coefficients (SHCs). The following expression demonstrates the description or representation of a sound field using SHCs:

[0026]

[0027] This expression shows that at time t, any point {r r ,θ r , The pressure p i Can be used Uniquely represents. Here, c is the speed of sound (~343 m / s), is the reference point (or observation point), j n (·) is a spherical Bessel function of order n, and are the n-th and m-th order spherical harmonic basis functions (also called spherical basis functions). It can be appreciated that the terms in square brackets are the frequency domain representations of the signal (e.g., ), which can be approximated by various time-frequency transforms such as discrete Fourier transform (DFT), discrete cosine transform (DCT) or wavelet transform. Other examples of hierarchical sets include wavelet transform coefficient sets and other coefficient sets of multiresolution basis functions.

[0028] They can be physically acquired (e.g., recorded) by various microphone array configurations, or alternatively, they can be derived from a channel-based or object-based description of the sound field. SHCs (also known as ambisonic coefficients) represent scene-based audio, where the SHCs can be input to an audio encoder to obtain encoded SHCs, which can facilitate more efficient transmission or storage. For example, a method involving (1+4) 2 (25, hence fourth order) The fourth order representation of the coefficients.

[0029] As described above, SHC can be derived from microphone recordings using a microphone array. Various examples of how to physically obtain SHC from a microphone array are described in Poletti, M., "Three-Dimensional Surround Sound Systems Based on Spherical Harmonics", J. Audio Eng. Soc., No. 11, Vol. 53, pp. 1004-1025, November 2005.

[0030] The following equation illustrates how the SHC can be derived from an object-based description. The coefficients of the sound field corresponding to an individual audio object are It can be expressed as:

[0031]

[0032] where i is is a spherical Hankel function of order n (second kind), and is the position of the object. Knowing that the object source energy g(ω) is a function of frequency (e.g., using time-frequency analysis techniques such as performing a fast Fourier transform on a pulse code modulation (PCM) stream) allows each PCM object and corresponding position to be converted to Furthermore, it can be shown (because of the linear and orthogonal decomposition above) that for each object The coefficients are additive. In this way, several PCM objects can be represented by The coefficients are represented (e.g. as the sum of coefficient vectors of individual objects). The coefficients may contain information about the sound field (pressure as a function of 3D coordinates), and the above represents a transformation from individual objects to a representation of the overall sound field near the observation point

[0033] The techniques described in this disclosure may be applied to any of the formats discussed herein, including scene-based audio formats, channel-based audio formats, object-based audio formats, or any combination thereof.

[0034] Computer-mediated reality systems (also referred to as "extended reality systems" or "XR systems") are being developed to take advantage of many of the potential benefits provided by ambisonic coefficients. For example, ambisonic coefficients can represent a sound field in three dimensions in a manner that potentially enables accurate 3D positioning of sound sources within the sound field. Thus, an XR device can render the ambisonic coefficients to a speaker feed that, when played back via one or more speakers, accurately reproduces the sound field.

[0035] As another example, the ambisonic coefficients may be translated or rotated to account for user movement without overly complex mathematical operations, thereby potentially accommodating the low latency requirements of an XR device. Additionally, the ambisonic coefficients are layered, thereby naturally accommodating scalability through order reduction (which may eliminate ambisonic coefficients associated with higher orders), and thereby potentially allowing dynamic adaptation of the sound field to accommodate the latency and / or battery requirements of an XR device.

[0036] Using ambisonics coefficients for XR devices may enable the development of some use cases that rely on the more immersive sound field provided by ambisonics coefficients, particularly for computer gaming applications and real-time video streaming applications. In these highly dynamic use cases that rely on low-latency reproduction of the sound field, XR devices may prefer ambisonics coefficients over other representations that are more difficult to manipulate or involve complex rendering. Figures 1A-1C More information about these use cases is available.

[0037] Although described in the present disclosure with respect to VR devices, various aspects of the present technology may be performed in the context of other devices such as mobile devices. In this case, a mobile device (such as a so-called smartphone) may present the displayed world via a screen that may be mounted to the user's head or viewed as when the mobile device is conventionally used. Therefore, any information on the screen may be part of the mobile device. The mobile device is capable of providing tracking information, thereby allowing both a VR experience (when head-mounted) and a conventional experience to view the display world, wherein the conventional experience still allows the user to view the display world, providing a VR-lite type of experience (e.g., lifting the device and rotating or translating the device to view different parts of the display world). In addition, although the display world is mentioned in various examples of the present disclosure, the technology of the present disclosure may also be used with an acoustic space that does not correspond to the display world or does not exist in the display world. For example, the display world does not need to be displayed on the content consumer device, and in this regard, the concept of the display world may be a world with an acoustic space presented to the user, but it does not have to be visually presented on the display.

[0038] Figure 1A-Figure 1C is a schematic diagram showing a system that can perform various aspects of the technology described in this disclosure. Figure 1AAs shown in the example of , system 10 includes a source device 12A and a content consumer device 14A. Although described in the context of source device 12A and content consumer device 14A, the present technology can be implemented in any context in which any representation of a sound field is encoded to form a bitstream representing audio data. In addition, source device 12A can represent any form of computing device capable of generating a sound field representation, and is generally described in this document in the context of being a VR content creator device. Similarly, content consumer device 14A can represent any form of computing device capable of implementing the rendering metadata techniques and audio playback described in this disclosure, and is generally described in this document in the context of being a VR client device.

[0039] Source device 12A may be operated by an entertainment company or other entity that may generate mono and / or multi-channel audio content for consumption by an operator of a content consumer device such as content consumer device 14A. In some VR scenarios, source device 12A generates audio content in conjunction with video content. Source device 12A includes content capture device 20, content editing device 22, and sound field representation generator 24. Content capture device 20 may be configured to connect to or otherwise communicate with microphone 18.

[0040] The microphone 18 may represent a device capable of capturing a sound field and representing it as audio data 19. or other types of 3D audio microphones, the audio data 19 may refer to one or more of the above-mentioned scene-based audio data (e.g., ambisonic reverberation coefficients), object-based audio data, and channel-based audio data. Although described as a 3D audio microphone, the microphone 18 may also represent other types of microphones (e.g., omnidirectional microphones, point microphones, unidirectional microphones, etc.) configured to capture the audio data 19. The audio data 19 may represent an audio stream or include an audio stream.

[0041] In some examples, the content capture device 20 may include an integrated microphone 18 integrated into the housing of the content capture device 20. The content capture device 20 may be connected to the microphone 18 wirelessly or via a wired connection. Rather than capturing the audio data 19 via the microphone 18, or in conjunction with capturing the audio data 19, the content capture device 20 may process the audio data 19 after the audio data 19 is input via some type of removable memory, wirelessly, and / or via a wired input process. Thus, various combinations of content capture devices 20 and microphones 18 are possible in accordance with the present disclosure.

[0042] The content capture device 20 may also be configured to connect to or otherwise communicate with a content editing device 22. In some instances, the content capture device 20 may include the content editing device 22 (in some instances, the content editing device 22 may represent software or a combination of software and hardware, including software executed by the content capture device 20 to configure the content capture device 20 to perform a particular form of content editing). The content editing device 22 may represent a unit configured to edit or otherwise change the content 21 including the audio data 19 received from the content capture device 20. The content editing device 22 may output the edited content 23 and the associated metadata 25 to the sound field representation generator 24.

[0043] The sound field representation generator 24 may include any type of hardware device capable of being connected to the content editing device 22 (or the content capturing device 20). Figure 1A Not shown in the example of , but the sound field representation generator 24 may generate one or more bitstreams 27 using the edited content 23 including the audio data 19 and the metadata 25 provided by the content editing device 22. Figure 1A In the example focusing on the audio data 19 , the sound field representation generator 24 may generate one or more representations of the same sound field represented by the audio data 19 to obtain a bitstream 27 comprising a representation of the sound field and audio metadata 25 .

[0044] For example, to generate different representations of the sound field using the ambisonics coefficients (which is again an example of audio data 19), the sound field representation generator 24 may use an encoding scheme for the ambisonics representation of the sound field, which encoding scheme is called mixed-order ambisonics (MOA), which is discussed in more detail in U.S. patent application No. 15 / 672,058, entitled “MIXED-ORDER AMBISONICS (MOA) AUDIO DATA FOR COMPUTER-MEDIATED REALITY SYSTEMS,” filed on August 8, 2017, and in U.S. patent publication no. 20190007781, published on January 3, 2019.

[0045] To generate a particular MOA representation of a sound field, the sound field representation generator 24 may generate a partial subset of the entire set of ambisonic reverberation coefficients. For example, each MOA representation generated by the sound field representation generator 24 may provide accuracy with respect to certain areas of the sound field, but less accuracy in other areas. In one example, the MOA representation of a sound field may include eight (8) uncompressed ambisonic reverberation coefficients, while a third-order ambisonic reverberation representation of the same sound field may include sixteen (16) uncompressed ambisonic reverberation coefficients. Thus, each MOA representation of a sound field generated as a partial subset of ambisonic reverberation coefficients may not be as storage-intensive and bandwidth-intensive as a corresponding third-order ambisonic reverberation representation of the same sound field generated from the ambisonic reverberation coefficients (if and when transmitted as part of the bitstream 27 over the transmission channel shown).

[0046] Although described with respect to MOA representation, the techniques of the present disclosure may also be performed with respect to first-order ambisonic (FOA) representation, where all ambisonic coefficients associated with first-order spherical basis functions and zero-order spherical basis functions are used to represent the sound field. In other words, the sound field representation generator 24 may use all ambisonic coefficients of a given order N to represent the sound field, rather than using a partial subset of non-zero ambisonic coefficients to represent the sound field, resulting in a total ambisonic coefficient equal to (N+1) 2 .

[0047] In this regard, the ambisonic reverberation audio data (which is another way of referring to ambisonic reverberation coefficients in a MOA representation or a full-order representation such as the first-order representation described above) may include ambisonic reverberation coefficients associated with spherical basis functions having an order of first order or less (which may be referred to as "first-order ambisonic reverberation audio data"), ambisonic reverberation coefficients associated with spherical basis functions having mixed orders and sub-orders (which may be referred to as the "MOA representation" discussed above), or ambisonic reverberation coefficients associated with spherical basis functions having an order greater than first (which is referred to as a "full-order representation").

[0048] In some examples, the content capture device 20 or the content editing device 22 may be configured to wirelessly communicate with the sound field representation generator 24. In some examples, the content capture device 20 or the content editing device 22 may communicate with the sound field representation generator 24 via one or both of a wireless connection or a wired connection. Via the connection between the content capture device 20 or the content editing device 22 and the sound field representation generator 24, the content capture device 20 or the content editing device 22 may provide various forms of content, which for discussion purposes are described herein as part of the audio data 19.

[0049] In some examples, the content capture device 20 may utilize various aspects of the sound field representation generator 24 (in terms of hardware or software capabilities of the sound field representation generator 24). For example, the sound field representation generator 24 may include dedicated hardware configured to perform psychoacoustic audio coding (or dedicated software that, when executed, causes one or more processors to perform psychoacoustic audio coding), such as the Unified Speech and Audio Coder denoted “USAC” proposed by the Moving Picture Experts Group (MPEG), the MPEG-H 3D Audio Coding standard, the MPEG-I immersive audio standard, or a standard such as AptX. TM (including various versions of AptX, such as Enhanced AptX - E-AptX, AptX live, AptX Stereo and AptX High Definition - AptX-HD), Advanced Audio Coding (AAC), Audio Codec 3 (AC-3), Apple Lossless Audio Codec (ALAC), MPEG-4 Audio Lossless Streams (ALS), Enhanced AC-3, Free Lossless Audio Codec (FLAC), Monkey's Audio, MPEG-1 Audio Layer II (MP2), MPEG-1 Audio Layer III (MP3), Opus and proprietary standards such as Windows Media Audio (WMA).

[0050] The content capture device 20 may not include psychoacoustic audio encoder dedicated hardware or dedicated software, but may provide audio aspects of the content 21 in a non-psychoacoustic audio encoded form. The sound field representation generator 24 may assist in the capture of the content 21 by at least partially performing psychoacoustic audio encoding with respect to the audio aspects of the content 21.

[0051] The sound field representation generator 24 may also (in the case where the audio data 19 includes scene-based audio data) assist in content capture and transmission by generating one or more bitstreams 27 based at least in part on audio content (e.g., MOA representations and / or first-order ambisonic representations) generated from the audio data 19. The bitstream 27 may represent a compressed version of the audio data 19 and any other different types of content 21 (such as compressed versions of spherical video data, image data, or text data).

[0052] As an example, the sound field representation generator 24 may generate a bitstream 27 for transmission over a transmission channel, which may be a wired or wireless channel, a data storage device, etc. The bitstream 27 may represent an encoded version of the audio data 19, and may include a main bitstream and another side bitstream, which may be referred to as side channel information or metadata. In some instances, the bitstream 27 representing a compressed version of the audio data 19 (which may also represent scene-based audio data, object-based audio data, channel-based audio data, or a combination thereof) may conform to a bitstream generated according to the MPEG-H 3D audio coding standard and / or the MPEG-I immersive audio standard.

[0053] Content consumer device 14A may be operated by an individual and may represent a VR client device. Although described with respect to a VR client device, content consumer device 14A may represent other types of devices, such as an augmented reality (AR) client device, a mixed reality (MR) client device (or other XR client device), a standard computer, a head mounted device, a headset, a mobile device (including so-called smartphones), or any other device capable of tracking the head movement and / or general translational movement of the individual operating content consumer device 14A. Figure 1A As shown in the example of , content consumer device 14A includes audio playback system 16A, which may refer to any form of audio playback system capable of rendering audio data for playback as mono and / or multi-channel audio content.

[0054] Although Figure 1A 2 is shown as being sent directly to content consumer device 14A, but source device 12A may output bitstream 27 to an intermediate device located between source device 12A and content consumer device 14A. The intermediate device may store bitstream 27 for later transmission to content consumer device 14A, which may request bitstream 27. The intermediate device may include a file server, web server, desktop computer, laptop computer, tablet computer, mobile phone, smart phone, or any other device capable of storing bitstream 27 for later retrieval by an audio decoder. The intermediate device may reside in a content delivery network capable of streaming bitstream 27 (and possibly in conjunction with sending a corresponding video data bitstream) to a subscriber (such as content consumer device 14A) requesting bitstream 27.

[0055] Alternatively, source device 12A may store bitstream 27 to a storage medium such as a compact disk, digital video disk, high-definition video disk, or other storage medium, most of which are capable of being read by a computer and may therefore be referred to as a computer-readable storage medium or a non-transitory computer-readable storage medium. In this context, a transmission channel may refer to a channel used to transmit content stored to the medium (e.g., in the form of one or more bitstreams 27) (and may include retail stores and other store-based delivery mechanisms). In any case, the techniques of this disclosure should therefore not be limited in this regard. Figure 1A .

[0056] As described above, the content consumer device 14A includes an audio playback system 16A. The audio playback system 16A may represent any system capable of playing back mono and / or multi-channel audio data. The audio playback system 16A may include a plurality of different audio renderers 32. The audio renderers 32 may each provide different forms of rendering, wherein the different forms of rendering may include performing one or more of various modes of vector basis amplitude translation (VBAP), and / or performing one or more of various modes of sound field synthesis. As used herein, "A and / or B" means "A or B", or "both A and B".

[0057] The audio playback system 16A may also include an audio decoding device 34. The audio decoding device 34 may represent a device configured to decode the bitstream 27 to output audio data 19' (where the prime notation may represent that the audio data 19' is different from the audio data 19 due to lossy compression (e.g., quantization) of the audio data 19). Again, the audio data 19' may include scene-based audio data, which in some examples may form a complete first (or higher) order ambisonic representation, or a subset of the ambisonic representation that forms a MOA representation of the same sound field, a decomposition of the ambisonic representation (such as a primary audio signal, ambisonic coefficients, and vector-based signals described in the MPEG-H 3D Audio Coding Standard), or other forms of scene-based audio data. The audio data 19' may include an audio stream or a representation of an audio stream.

[0058] Other forms of scene-based audio data include audio data defined according to the HOA (High Order Ambisonics) Transport Format (HTF). More information about HTF can be found in the European Telecommunications Standards Institute (ETSI) Technical Specification (TS), entitled "Higher Order Ambisonics (HOA) Transport Format", ETSI TS 103589 v1.1.1, dated June 2018 (2018-06), and in U.S. Patent Publication No. 2019 / 0918028, entitled "PRIORITY INFORMATION FOR HIGHER ORDER AMBISONIC AUDIO DATA", filed on December 20, 2018. In any case, the audio data 19' can be similar to the full set or a partial subset of the audio data 19, but differ due to lossy operations (e.g., quantization) and / or transmission via a transmission channel.

[0059] As an alternative to scene-based audio data, or in combination with scene-based audio data, the audio data 19' may include channel-based audio data. As an alternative to scene-based audio data, or in combination with scene-based audio data, the audio data 19' may include object-based audio data or channel-based audio data. Thus, the audio data 19' may include any combination of scene-based audio data, object-based audio data, and channel-based audio data.

[0060] The audio renderer 32 of the audio playback system 16A may render the audio data 19' to an output speaker feed 35 after the audio decoding device 34 has decoded the bitstream 27 to obtain the audio data 19'. The speaker feed 35 may drive one or more speakers (for ease of illustration, in FIG. Figure 1A Various audio representations including scene-based audio data (and possibly channel-based audio data and / or object-based audio data) of the sound field may be normalized in a variety of ways including N3D, SN3D, FuMa, N2D, or SN2D.

[0061] In order to select an appropriate renderer or, in some cases, generate an appropriate renderer, the audio playback system 16A may obtain speaker information 37 indicating the number of speakers (e.g., loudspeakers or headphone speakers) and / or the spatial geometry of the speakers. In some cases, the audio playback system 16A may obtain the speaker information 37 using a reference microphone, and may drive the speakers (which may refer to the output of an electrical signal for vibrating a transducer) in a manner that dynamically determines the speaker information 37. In other cases, or in conjunction with the dynamic determination of the speaker information 37, the audio playback system 16A may prompt the user to connect to the audio playback system 16A and enter the speaker information 37.

[0062] The audio playback system 16A may select one of the audio renderers 32 based on the speaker information 37. In some cases, the audio playback system 16A may generate one of the audio renderers 32 based on the speaker information 37 when none of the audio renderers 32 are within a certain threshold similarity metric (in terms of speaker geometry) to the speaker geometry specified in the speaker information 37. In some cases, the audio playback system 16A may generate one of the audio renderers 32 based on the speaker information 37 without first attempting to select an existing one of the audio renderers 32.

[0063] When outputting the speaker feeds 35 to headphones, the audio playback system 16A may utilize one of the renderers 32 that provides binaural rendering using head-related transfer functions (HRTFs) or other functions capable of rendering left and right speaker feeds 35 for headphone speaker playback, such as a binaural room impulse response renderer. The terms "speaker" or "transducer" may generally refer to any speaker, including a loudspeaker, headphone speaker, bone conduction speaker, earbud speaker, wireless headphone speaker, etc. One or more speakers may then replay the rendered speaker feeds 35 to reproduce the sound field.

[0064] Although described as rendering the speaker feeds 35 from the audio data 19', references to rendering of the speaker feeds 35 may refer to other types of rendering, such as rendering incorporated directly into the decoding of the audio data from the bitstream 27. Examples of alternative rendering may be found in Annex G of the MPEG-H 3D Audio standard, where rendering occurs during main and background signal formation prior to sound field synthesis. Applications to rendering of the audio data 19' should therefore be understood to refer to rendering of the actual audio data 19' or a decomposition or representation of the audio data 19', such as the above-mentioned main audio signal, ambisonics coefficients and / or vector-based signals - which may also be referred to as V-vectors or multi-dimensional ambisonics space vectors.

[0065] The audio playback system 16A can also adapt the audio renderer 32 based on the tracking information 41. That is, the audio playback system 16A can be connected to a tracking device 40, which is configured to track the head movement and possible translational movement of the user of the VR device. The tracking device 40 can represent one or more sensors (e.g., cameras - including depth cameras, gyroscopes, magnetometers, accelerometers, light-emitting diodes - LEDs, etc.), which are configured to track the head movement and possible translational movement of the user of the VR device. The audio playback system 16A can adapt the audio renderer 32 based on the tracking information 41 so that the speaker feed 35 reflects the changes in the user's head and possible translational movement to correctly reproduce the sound field in response to such movement.

[0066] Figure 1B is a block diagram illustrating another example system 50 configured to perform various aspects of the techniques described in this disclosure. The system 50 is similar to Figure 1A The system 10 shown, except Figure 1A The audio renderer 32 shown in FIG. 1 is replaced by a binaural renderer 42 (in the audio playback system 16B of the content consumer device 14B) which is capable of performing binaural rendering using one or more head-related transfer functions (HRTFs) or other functions capable of rendering to left and right speaker feeds 43.

[0067] The audio playback system 16B may output the left and right speaker feeds 43 to headphones 48, which may represent another example of a wearable device and may be coupled to an additional wearable device (such as a watch, the VR headset mentioned above, smart glasses, smart clothing, a smart ring, a smart bracelet, or any other type of smart jewelry (including a smart necklace), etc.) to facilitate reproduction of the sound field. The headphones 48 may be coupled to the additional wearable device wirelessly or via a wired connection.

[0068] Additionally, the earphones 48 may be coupled to the audio playback system 16B via a wired connection (such as a standard 3.5 mm audio jack, a universal system bus (USB) connection, an optical audio jack, or other form of wired connection) or wirelessly (such as through a Bluetooth connection, a wireless network connection, etc.). The earphones 48 may recreate the sound field represented by the audio data 19' based on the left and right speaker feeds 43. The earphones 48 may include a left earphone speaker and a right earphone speaker that are powered (or in other words, driven) by the corresponding left and right speaker feeds 43.

[0069] Figure 1C is a block diagram illustrating another example system 60. The example system 60 is similar to Figure 1AExample system 10 of system 60, but source device 12B of system 60 does not include a content capture device. Source device 12B includes synthesis device 29. Content developers can use synthesis device 29 to generate a synthesized audio source. The synthesized audio source can have position information associated with it, which can identify the position of the audio source relative to the listener or other reference point in the sound field, so that the audio source can be rendered to one or more speaker channels for playback in an effort to recreate the sound field. In some examples, synthesis device 29 can also synthesize visual data or video data.

[0070] For example, a content developer could generate a synthetic audio stream for use in a video game. Figure 1C An example is with Figure 1A The example content consumer device 14A is shown together, but Figure 1C The source device 12B of the example may be connected with Figure 1B In some examples, Figure 1C Source device 12B may also include a content capture device, so that bitstream 27 may contain both captured audio streams and synthesized audio streams.

[0071] As described above, content consumer device 14A or 14B (hereinafter referred to as content consumer device 14) may represent a VR device in which a human wearable display (also referred to as a "head-mounted display") is mounted in front of the eyes of a user operating the VR device. Figure 2 is a schematic diagram showing an example of a VR device 400 worn by a user 402. The VR device 400 is coupled to or otherwise includes headphones 404, which can reproduce the sound field represented by the audio data 19' through the playback of a speaker feed 35. The speaker feed 35 can represent an analog or digital signal capable of causing a membrane within a transducer of the headphones 404 to vibrate at various frequencies, where this process is generally referred to as driving the headphones 404.

[0072] Video, audio, and other sensory data may play an important role in a VR experience. To participate in a VR experience, a user 402 may wear a VR device 400 (which may also be referred to as a VR client device 400) or other wearable electronic device. A VR client device (such as VR device 400) may include a tracking device (e.g., tracking device 40) that is configured to track the head movement of the user 402 and adapt the video data displayed via the VR device 400 to account for the head movement, providing an immersive experience in which the user 402 can experience the display world displayed in the video data in visual three dimensions. The display world may refer to a virtual world (in which the entire world is simulated), an augmented world (in which portions of the world are augmented by virtual objects), or a physical world (in which real-world images are virtually navigated).

[0073] While VR (and other forms of AR and / or MR) can allow a user 402 to visually reside in a virtual world, VR devices 400 may typically lack the ability to audibly place the user in the displayed world. In other words, a VR system (which may include a computer responsible for rendering video data and audio data—for ease of illustration purposes only) may be capable of audibly placing the user in the displayed world. Figure 2 , and VR device 400) may not be able to audibly support (and in some cases realistically support in a manner that reflects the display scene presented to the user via VR device 400) full three-dimensional immersion.

[0074] Although described in the present disclosure with respect to VR devices, various aspects of the present technology may be performed in the context of other devices such as mobile devices. In this case, a mobile device (such as a so-called smartphone) may present the displayed world via a screen that may be mounted to the head of a user 402 or viewed as when the mobile device is used conventionally. Thus, any information on the screen may be part of the mobile device. The mobile device may be able to provide tracking information 41, thereby allowing both a VR experience (when head mounted) and a conventional experience to view the displayed world, wherein the conventional experience still allows the user to view the displayed world, providing a VR-lite type of experience (e.g., lifting the device and rotating or translating the device to view different parts of the displayed world).

[0075] In any case, returning to the context of VR devices, the audio aspects of VR have been divided into three separate immersion categories. The first category provides the lowest level of immersion and is known as three degrees of freedom (3DOF). 3DOF refers to audio rendering that takes into account head movement in three degrees of freedom (yaw, pitch, and roll), allowing the user to look around freely in any direction. However, 3DOF does not account for translational head movement (in which the center of the head is not at the optical and acoustic center of the soundstage).

[0076] The second category, called 3DOF plus (3DOF+), provides three degrees of freedom (yaw, pitch, and roll) in addition to the limited spatial translational movement caused by moving the head away from the optical and acoustic centers in the sound field. 3DOF+ can provide support for perceptual effects such as motion parallax, which can enhance immersion.

[0077] The third category, referred to as six degrees of freedom (6DOF), renders audio data in a way that takes into account three degrees of freedom in terms of head movement (yaw, pitch, and roll) but also takes into account the translation of the user in space (x, y, and z translation). Spatial translation can be caused by sensors that track the user's position in the physical world or by input controllers.

[0078] 3DOF rendering is the latest technology in VR audio. As a result, the audio aspects of VR are less immersive than the video aspects, potentially reducing the overall immersion that users experience. However, VR is rapidly transforming and may quickly evolve to support 3DOF+ and 6DOF, which may provide opportunities for additional use cases.

[0079] For example, interactive gaming applications can utilize 6DOF to facilitate fully immersive gaming, where the user moves around the VR world themselves and can interact with virtual objects by walking up to them. Additionally, interactive live streaming applications can utilize 6DOF to allow VR client devices to experience a live stream of a concert or sporting event as if they were at the concert themselves, allowing the user to move around the concert or sporting event.

[0080] The difficulties associated with these use cases are numerous. In the case of fully immersive gaming, latency may need to be kept low so that the game does not cause nausea or motion sickness. Additionally, from an audio perspective, delays in audio playback that result in loss of synchronization with the video data may reduce immersion. Furthermore, for certain types of gaming applications, spatial accuracy may be important to allow accurate responses, including with respect to how the user perceives sound, as this allows the user to anticipate actions that are not currently in view.

[0081] In the context of a live streaming application, a large number of source devices 12A or 12B (hereinafter referred to as source devices 12) can stream content 21, where source devices 12 can have widely varying capabilities. For example, one source device can be a smartphone with a digital fixed-lens camera and one or more microphones, while another source device can be a production-level television device capable of obtaining video at a much higher resolution and quality than a smartphone. However, in the context of a live streaming application, all source devices can provide streams of varying quality, and the VR device can attempt to select an appropriate stream from among them to provide the intended experience.

[0082] Furthermore, similar to gaming applications, latency in the audio data causing a loss of synchronization with the video data may result in less immersion. Additionally, spatial accuracy may also be important so that the user can better understand the context or location of different audio sources. Going further, privacy may become an issue when users are live streaming using a camera and microphone, as users may not want the live stream to be fully open to the public.

[0083] In the context of streaming applications (live or recorded), there may be a large number of audio streams associated with different levels of quality and / or content. The audio streams may represent any type of audio data, including scene-based audio data (e.g., ambisonic audio data, including FOA audio data, MOA audio data, and / or HOA audio data), channel-based audio data, and object-based audio data. Selecting only one of a potentially large number of audio streams to recreate the sound field from may not provide an experience that ensures a sufficient level of immersion. However, due to the different spatial positioning between the multiple audio streams, selecting multiple audio streams may cause interference, thereby potentially reducing the sense of immersion.

[0084] According to the techniques described in this disclosure, audio decoding device 34 may adaptively select between audio streams available via bitstream 27 (which are represented by bitstream 27, so bitstream 27 may be referred to as “audio stream 27”). Audio decoding device 34 may select between audio streams available via bitstream 27 based on audio location information (ALI) (e.g., Figure 1A-Figure 1C The audio decoding device 34 may select between different audio streams of the audio stream 27, the ALI 45A such as capture position information or position information related to a synthesized audio source included as metadata accompanying the audio stream 27, wherein the audio position information may define the coordinates of a microphone capturing the corresponding audio stream 27 in the display world or in the acoustic space. The ALI 45A may represent a capture position (or a synthesis position) at which a corresponding one of the audio streams 27 is captured or synthesized in the display world (or acoustic space). The audio decoding device 34 may select a subset of the audio streams 27 based on the ALI 45A, wherein the subset of the audio streams 27 excludes at least one of the audio streams 27. The audio decoding device 34 may output the subset of the audio stream 27 as audio data 19' (also referred to as "audio stream 19'"). In some examples, the audio decoding device 34 may decode only the subset of the audio stream in response to the selection.

[0085] In addition, the audio decoding device 34 can obtain tracking information 41, which the content consumer device 14 can convert into device location information (DLI) 45B. In some examples, the DLI 45B can represent the virtual position of the content consumer device 14 in the display world, which can be defined as one or more device coordinates in the display world. In some examples, the DLI 45B can represent the location device coordinates in the acoustic space. Although sometimes discussed herein as representing the virtual position of the content consumer device in the display world, the DLI 45B can be device information representing the physical position of the content consumer device 14. For example, the content consumer device 14 may not provide the display world to the user. The content consumer device 14 can provide the DLI 45B to the audio decoding device 34. Then, the audio decoding device 34 can select the audio stream 19' from the audio stream 27 based on the ALI 45A and the DLI 45B. Then, the audio playback system 16A or 16B can reproduce the corresponding sound field based on the audio stream 19'.

[0086] In this regard, the audio decoding device 34 may adaptively select a subset of the audio streams 27 to obtain an audio stream 19' that may result in a more immersive experience (compared to selecting a single audio stream or all audio streams 19'). Thus, various aspects of the techniques described in this disclosure may improve the operation of the audio decoding device 34 (as well as the audio playback system 16A and the content consumer device 14) itself by potentially enabling the audio decoding device 34 to better spatialize sound sources within a sound field and thereby improve immersion.

[0087] In operation, the audio decoding device 34 may be connected to one or more source devices 12 to determine the ALI 45A for each audio stream 27. Figure 1A As shown in the example of , the audio decoding device 34 may include a stream selection unit 44, which may represent a unit configured to perform various aspects of the audio stream selection techniques described in the present disclosure.

[0088] The stream selection unit 44 may generate a constellation map (CM) 47 based on the ALI 45A. The CM 47 may define the ALI 45A for each audio stream 27. The stream selection unit 44 may also perform an energy analysis on each audio stream 27 to determine an energy map for each audio stream 27 and store the energy map with the ALI 45A in the CM 47. The energy maps may collectively define the energy of a common sound field represented by the audio streams 27.

[0089] The stream selection unit 44 may then determine the distance between the virtual position represented by the DLI 45B and the capture position (or synthesized position) represented by the ALI 45A, which is associated with at least one (and possibly each) of the audio streams 27. The stream selection unit 44 may then select the audio stream 19′ from the audio streams 27 based on the distance, as described below with respect to Figure 3A-3D discussed in more detail.

[0090] Further, in some examples, the stream selection unit 44 may also select the audio stream 19' from the audio stream 27 based on the energy map stored in the CM 47, the ALI 45A, and the DLI 45B (where the ALI 45A and the DLI 45B are jointly presented in the form of the above-mentioned distance, which may also be referred to as "relative distance"). For example, the stream selection unit 44 may analyze the energy map presented in the CM 47 to determine the audio source position (ASL) 49 of the audio source in the common sound field, which emits the sound captured by the microphone (such as the microphone 18) and represented by the audio stream 27. Then, the stream selection unit 44 may determine the audio stream 19' from the audio stream 27 based on the ALI 45A, the DLI 45B, and the ASL 49. This will be described below with respect to Figure 3A-3D More information on how the stream selection unit 44 may select a stream is discussed.

[0091] Figure 3A-3D is shown in more detail Figure 1A Schematic diagram of an example operation of the stream selection unit 44 shown in the example of FIG. Figure 3A As shown in the example of , the stream selection unit 44 can determine that the DLI 45B indicates that the content consumer device 14 (shown as VR device 400) is at the virtual location 300A. The stream selection unit 44 can then determine the ALI 45A of one or more audio elements of the audio elements 302A-302J (collectively referred to as audio elements 302), which can represent microphones (such as Figure 1A microphone 18 shown in ), other types of capture devices (including other XR devices, mobile phones - including so-called smartphones - etc.), or synthetic sound fields, etc.

[0092] As described above, the stream selection unit 44 can obtain the audio stream 27. The stream selection unit 44 can be connected to the audio elements 302A-302J to obtain the audio stream 27. In some examples, the stream selection unit 44 can interact with an interface (such as a receiver, a transmitter, and / or a transceiver) to provide a cellular connection according to the fifth generation (5G) cellular standard, a personal area network (PAN) (such as Bluetooth), and a wireless communication interface (such as a wireless communication interface). TM ) or some other open source, proprietary or standardized communication protocol to obtain the audio stream 27. The wireless communication of the audio stream is Figure 3A-3DIn the example of , the selected audio data 19 ′ is represented as lightning, where the selected audio data 19 ′ is shown as being transmitted from one or more audio elements of the selected audio elements 302 to the VR device 400.

[0093] In any case, the stream selection unit 44 may then obtain the energy map in the manner described above and analyze the energy map to determine the audio source location 304, which may represent Figure 1A An example of ASL 49 is shown in the example of FIG. The energy map can represent the audio source location 304 because the energy at the audio source location 304 may be higher than the surrounding area. Assuming that each energy map can represent the higher energy, the stream selection unit 44 can triangulate the audio source location 304 based on the higher energy in the energy map.

[0094] The stream selection unit 44 may determine the audio source distance 306A. In some examples, the stream selection unit 44 may determine the audio source distance 306A as the distance between the audio source position in the acoustic space or virtual acoustic space and the device coordinates. In some examples, the stream selection unit 44 may determine the audio source distance as the distance between the audio stream coordinates and the device coordinates. In some examples, the stream selection unit 44 may determine the audio source distance as the distance between the audio source position (e.g., the audio source position 304) and the device coordinates (e.g., the virtual position 300A of the VR device 400). The stream selection unit 44 may compare the audio source distance 306A with an audio source distance threshold. In some examples, the stream selection unit 44 may derive the audio source distance threshold based on the energy of the audio source 308. That is, when the audio source 308 has higher energy (or, in other words, when the audio source 308 is louder), the stream selection unit 44 may increase the audio source distance threshold. When the audio source 308 has lower energy (or, in other words, when the audio source 308 is quieter), the stream selection unit 44 can reduce the audio source distance threshold. In other examples, the stream selection unit 44 can obtain a statically defined audio source distance threshold, which can be statically defined or specified by the user 402.

[0095] In any case, the stream selection unit 44 may select a single audio stream of the audio stream 27 of the audio elements 302A-302J ("audio elements 302") when the audio source distance 306A is greater than an audio source distance threshold (assumed for illustration purposes in this example). For example, the stream selection unit 44 may select the audio stream of the audio element (e.g., audio element 302A) having the shortest distance to the virtual position 300A. The stream selection unit 44 may output a corresponding one of the audio streams 27, which the audio decoding device 34 may decode and output as audio data 19'.

[0096] Assuming user 402 moves from virtual location 300A to virtual location 300B, stream selection unit 44 may determine audio source distance 306B as the distance between audio source location 304 and virtual location 300B. In some examples, stream selection unit 44 may update only after a certain configurable release time, which may refer to the time after the listener stops moving.

[0097] In any case, the stream selection unit 44 can again compare the audio source distance 306B with the audio source distance threshold. The stream selection unit 44 can select multiple audio streams in the audio stream 27 of the audio element 302A-302J ("audio element 302") when the audio source distance 306B is less than or equal to the audio source distance threshold (assumed for illustration purposes in this example). The stream selection unit 44 can output a corresponding one of the audio streams 27, which the audio decoding device 34 can decode and output as audio data 19'.

[0098] Stream selection unit 44 may also determine one or more proximity distances between virtual position 300A and one or more (and possibly each) of the capture positions (or synthesized positions) represented by ALI 45A to obtain one or more proximity distances. Stream selection unit 44 may then compare the one or more proximity distances to a threshold proximity distance. When one or more proximity distances are greater than the threshold proximity distance, stream selection unit 44 may select a smaller number of audio streams 27 to obtain audio stream 19' than when one or more proximity distances are less than or equal to the threshold proximity distance. However, when one or more proximity distances are less than or equal to the threshold proximity distance, stream selection unit 44 may select a larger number of audio streams 27 to obtain audio stream 19' than when one or more proximity distances are greater than the threshold proximity distance.

[0099] In other words, the stream selection unit 44 may attempt to select those of the audio streams 27 that most closely align the audio stream 19' with and around the virtual location 300B. The proximity distance threshold may define such a threshold, which the user 402 may set, or which the stream selection unit 44 may again dynamically determine based on the quality of the audio elements 302F-302J, the gain or loudness of the audio source 308, the tracking information 41 (e.g., to determine whether the user 402 is facing the audio source 308), or any other factor.

[0100] In this regard, stream selection unit 44 can increase the audio spatialization accuracy when the listener is at position 300B. In addition, when the listener is at position 300A, stream selection unit 44 can reduce the bit rate because only the audio stream of audio element 302A is used to reproduce the sound field instead of using multiple audio streams of audio elements 302B-302J.

[0101] Next reference Figure 3B For example, stream selection unit 44 may reiterate through to determine that the audio stream of audio element 302A is corrupted, noisy, or unavailable. Given that audio source distance 306A is greater than an audio source distance threshold, stream selection unit 44 may remove the audio stream from CM 47 and reiterate through audio streams 27 to select a single audio stream (e.g., Figure 3B 302B in the example of FIG. 304 ).

[0102] Next reference Figure 3C For example, stream selection unit 44 may obtain a new audio stream (the audio stream of audio element 302k) and corresponding new audio information, such as metadata including ALI 45A. Stream selection unit 44 may add the new audio stream to CM 47 representing audio stream 27. Given that audio source distance 306A is greater than an audio source distance threshold, stream selection unit 44 may then repeatedly traverse audio streams 27 according to the techniques described in more detail above to select a single audio stream (e.g., Figure 3C 302B in the example of FIG. 304 ).

[0103] exist Figure 3D In the example of , audio element 302 is replaced by specific example devices 320A-320J ("devices 320"), where device 320A represents a dedicated microphone 320A, and devices 320B, 320C, 320D, 320G, 320H, and 320J represent smartphones. Devices 320E, 320F, and 320I may represent VR devices. Each of devices 320 may include an audio element 302, which may represent an audio stream 27 to be selected, captured, or synthesized according to various aspects of the stream selection techniques described in the present disclosure.

[0104] Figure 3E is a conceptual diagram showing an example concert with three or more audio elements. Figure 3EIn the example of , many musicians are depicted on stage 323. Singer 312 is located behind audio element 310A. String section 314 is depicted behind audio element 310B. Drummer 316 is depicted behind audio element 310C. Other musicians 318 are depicted behind audio element 310D. Audio elements 310A-310D can represent captured audio streams corresponding to the sounds received by the microphone. In some examples, audio elements 310A-310D can represent synthesized audio streams. For example, audio element 310A can represent a captured audio stream mainly associated with singer 312, but the audio stream can also include sounds produced by other band members (such as string section 314, drummer 316 or other musicians 318), and audio element 310B can represent a captured audio stream mainly associated with string section 314, but includes sounds produced by other band members. In this way, each audio element 310A-310D can represent a different audio stream.

[0105] In addition, many devices are described. These devices represent user devices located at multiple different listening positions. Head mounted device 321 is located near audio element 310A, but between audio element 310A and audio element 310B. Therefore, according to the technology of the present disclosure, stream selection unit 44 can select at least one of the audio streams to produce a sound similar to that of the user of head mounted device 321. Figure 3E Similarly, VR goggles 326 are shown positioned behind audio element 310C and between drummer 316 and other musicians 318. Stream selection unit 44 may select at least one audio stream to produce an audio experience for a user of VR goggles 326 similar to that of a user positioned at a location other than the VR goggles 326. Figure 3E The audio experience of a user at the location of the VR goggles 326.

[0106] Smart glasses 324 are shown as being centered between audio elements 310A, 310C, and 310C. Stream selection unit 44 may select at least one audio stream to produce an audio stream similar to that located at the user of smart glasses 324. Figure 3E In addition, device 322 (which may represent any device capable of implementing the techniques of the present disclosure, such as a mobile phone, a speaker array, a head-mounted device, VR goggles, smart glasses, etc.) is shown located in front of audio element 310B. Stream selection unit 44 may select at least one audio stream to produce an audio experience similar to that of a user at the location of smart glasses 324 in FIG. Figure 3E Although specific devices are discussed with respect to specific locations, any of the devices described may provide different audio experiences for users at the location of device 322. Figure 3E An indication of the desired listening position as shown in .

[0107] Figure 4A-4E It is shown Figure 1A and Figure 1B The example operation of the stream selection unit 44 shown in the example of FIG. 4 is performed in various aspects of the stream selection technique. Figure 4A For example, the stream selection unit 44 may obtain audio streams 27 from all enabled audio elements, wherein the audio streams 27 may include corresponding audio metadata, such as ALI 45A (800). The stream selection unit 44 may perform energy analysis on each audio stream 27 to calculate a corresponding energy map (802).

[0108] Next, stream selection unit 44 may iterate through different combinations of audio elements (defined in CM 47) based on proximity to audio source 308 (as defined by audio source distances 306A and / or 306B) and proximity to audio elements (as defined by proximity distances discussed above) (804). Figure 4A , the audio elements may be ordered or otherwise associated with different access permissions as shown in . The stream selection unit 44 may iterate in the manner described above to identify whether a larger subset of the audio streams 27 or a reduced subset of the audio streams 27 is desired based on the listener position represented by the DLI 45B (which is another way of referring to the "virtual position") and the audio element position represented by the ALI 45A (806, 808).

[0109] When a larger subset of the audio stream 27 is needed, the stream selection unit 44 may add audio elements to the audio stream 19', or in other words, add additional audio streams (such as in Figure 3A In the example of , when the user is closer to the audio source) (810). When a simplified subset of the audio stream 27 is needed, the stream selection unit 44 can remove audio elements from the audio stream 19', or in other words, remove existing audio streams (such as in Figure 3A In the example of , when the user is farther away from the audio source)(812).

[0110] In some examples, the stream selection unit 44 may determine that the current constellation of audio elements is the best set (or, in other words, the existing audio streams 19' will remain unchanged because the selection process described herein results in the same audio streams 19') (804), and the process may return to 802. However, when audio streams are added or removed from the audio streams 19', the stream selection unit 44 may update the CM 47 (814), generate a constellation history (815) (including positions, energy maps, etc.).

[0111] In addition, the stream selection unit 44 can determine whether the privacy setting allows or prohibits the addition of audio elements (wherein the privacy setting can refer to digital access permissions that restrict access to one or more audio streams 27, for example, by passwords, authorization levels or grades, time, etc.) (816, 818). When the privacy setting allows the addition of audio elements, the stream selection unit 44 can add the audio elements to the updated CM 47 (this refers to adding the audio stream to the audio stream 19') (820). When the privacy setting prohibits the addition of audio elements, the stream selection unit 44 can remove the audio elements from the updated CM 47 (this refers to removing the audio stream from the audio stream 19') (822). In this way, the stream selection unit 44 can identify a new set of enabled audio elements (824).

[0112] The stream selection unit 44 may iterate in this manner and update the various inputs according to any given frequency. For example, the stream selection unit 44 may update the privacy settings at the user interface rate (meaning that the updates are driven by updates input via the user interface). As another example, the stream selection unit 44 may update the position at the sensor rate (meaning that the position changes through the movement of the audio element). The stream selection unit 44 may further update the energy map at the audio frame rate (meaning that the energy map is updated every frame).

[0113] Next reference Figure 4B For example, the stream selection unit 44 may be Figure 4A The CM 47 may be determined based on the energy map, except that the stream selection unit 44 may not determine the CM 47 based on the energy map. Therefore, the stream selection unit 44 may obtain the audio streams 27 from all enabled audio elements, wherein the audio streams 27 may include corresponding audio metadata, such as the ALI 45A (840). The stream selection unit 44 may determine whether the privacy setting allows or prohibits the addition of the audio element (wherein the privacy setting may refer to a digital access permission that limits access to one or more audio streams 27, for example, by a password, an authorization level or rating, time, etc.) (842, 844).

[0114] When the privacy setting allows the addition of audio elements, the stream selection unit 44 may add the audio elements to the updated CM 47 (which refers to adding the audio stream to the audio stream 19') (846). When the privacy setting prohibits the addition of audio elements, the stream selection unit 44 may remove the audio elements from the updated CM 47 (which refers to removing the audio stream from the audio stream 19') (848). In this way, the stream selection unit 44 may identify a new set of enabled audio elements (850). The stream selection unit 44 may iterate (852) through different combinations of audio elements in the CM 47 to determine a constellation history representing the audio stream 19' (854).

[0115] The stream selection unit 44 may iterate in this manner and update the various inputs according to any given frequency. For example, the stream selection unit 44 may update the privacy settings at the user interface rate (meaning the updates are driven by updates input via the user interface). As another example, the stream selection unit 44 may update the position at the sensor rate (meaning the position changes by the movement of the audio element).

[0116] Next reference Figure 4C For example, the stream selection unit 44 may be Figure 4A The CM 47 may be determined based on the audio elements enabled by the privacy settings, except that the stream selection unit 44 may not determine the CM 47 based on the audio elements enabled by the privacy settings. Therefore, the stream selection unit 44 may obtain the audio streams 27 from all enabled audio elements, wherein the audio streams 27 may include corresponding audio metadata, such as the ALI 45A (860). The stream selection unit 44 may perform energy analysis on each audio stream 27 to calculate a corresponding energy map (862).

[0117] Next, stream selection unit 44 may iterate through different combinations of audio elements (defined in CM 47) based on proximity to audio source 308 (as defined by audio source distances 306A and / or 306B) and proximity to audio elements (as defined by proximity distances discussed above) (864). Figure 4C , the audio elements may be ordered or otherwise associated with different access permissions as shown in . The stream selection unit 44 may iterate in the manner described above to identify whether a larger subset of the audio streams 27 or a reduced subset of the audio streams 27 is desired based on the listener position represented by the DLI 45B (which is also another way to refer to the "virtual position" discussed above) and the audio element position represented by the ALI 45A (866, 868).

[0118] When a larger subset of the audio stream 27 is needed, the stream selection unit 44 may add audio elements to the audio stream 19', or in other words, add additional audio streams (such as in Figure 3A In the example of , when the user is closer to the audio source) (870). When a simplified subset of the audio stream 27 is needed, the stream selection unit 44 can remove audio elements from the audio stream 19', or in other words, remove existing audio streams (such as in Figure 3A In the example of , when the user is farther away from the audio source)(872).

[0119] In some examples, the stream selection unit 44 may determine that the current constellation of audio elements is the best set (or, in other words, the existing audio streams 19' will remain unchanged because the selection process described herein results in the same audio streams 19') (864), and the process may return to 862. However, when audio streams are added or removed from the audio streams 19', the stream selection unit 44 may update the CM 47 (874), generating a constellation history (875).

[0120] The stream selection unit 44 may iterate in this manner and update the various inputs according to any given frequency. For example, the stream selection unit 44 may update the position at the sensor rate (meaning the position changes by the movement of the audio element). The stream selection unit 44 may further update the energy map at the audio frame rate (meaning the energy map is updated every frame).

[0121] Reference now Figure 4D In an example of , one or more processors of content consumer device 14 may store multiple audio streams and audio position information (876). The audio position information may be associated with the multiple audio streams and represent the audio stream coordinates where the audio streams are captured or synthesized in an acoustic space. In some examples, the acoustic space may be a virtual acoustic space in which the audio streams may be synthesized. The one or more processors of content consumer device 14 may determine device position information (878). For example, the device position information may represent the device coordinates of the device in the acoustic space.

[0122] The one or more processors of the content consumer device may select a subset of the plurality of audio streams based on the device location information and the audio location information (880). For example, the one or more processors of the content consumer device 14 may determine a proximity distance for at least one of the plurality of audio streams based on the device location information and the audio location information, and select the subset of the plurality of audio streams based on the proximity distance. In another example, the one or more processors of the content consumer device 14 may compare the proximity distance to a threshold proximity distance, and when the proximity distance is less than or equal to the threshold proximity distance, select a larger number of the plurality of audio streams to obtain the subset of the plurality of audio streams than when the proximity distance is greater than the threshold proximity distance. In another example, the one or more processors of the content consumer device 14 may compare the proximity distance to a threshold proximity distance, and when the proximity distance is greater than the threshold proximity distance, select a smaller number of the plurality of audio streams to obtain the subset of the plurality of audio streams than when the proximity distance is less than or equal to the threshold proximity distance.

[0123] The one or more processors of content consumer device 14 may output one or more speaker feeds based on the subset of the plurality of audio streams (882). In some examples, the one or more processors of content consumer device 14 may obtain a new audio stream and corresponding new audio position information, and update the subset of the plurality of audio streams to include the new audio stream. For example, user 402 may move to a new audio stream, and the one or more processors of content consumer device 14 may obtain the new audio stream and update the subset of the plurality of audio streams to include the new audio stream.

[0124] In the context of 6DoF audio, alignment or alignment modes may be defined as a transformation of spatial coordinates used for audio rendering. In some examples, this transformation of spatial coordinates may be real-time or substantially real-time (less than 2 seconds of latency). In this way, alignment may destroy the relationship between the listener coordinates (e.g., the virtual position of the device or listener in the display world) used to render the visual scene (or the audio scene in the absence of visual content) and the coordinates used for audio rendering. There may be two alignment modes - hard alignment and soft alignment.

[0125] Figure 4E is a flow chart illustrating an example of alignment according to the techniques of this disclosure. Figure 4E The example can be compared with Figure 4D The example of can be used together or separately. One or more processors of the content consumer device 14 can determine a first audio source distance (884). For example, the one or more processors of the content consumer device 14 can determine the distance between the audio stream coordinates of the first audio stream in the acoustic space and the device coordinates in the acoustic space. The one or more processors of the content consumer device 14 can compare the first audio source distance with the first audio source distance threshold (886) to determine whether the first audio source distance is less than or equal to the first audio source distance threshold. When the first audio source distance is less than or equal to the first audio source distance threshold (the "yes" path from box 886), then the one or more processors of the content consumer device 14 select the first audio stream (888). In some examples, the first audio stream is the only audio stream selected. The one or more processors of the content consumer device 14 can output one or more speaker feeds based on the first audio stream (898). For example, the one or more processors of the content consumer device 14 align the user's audio experience to the first audio stream.

[0126] When the first audio source distance is greater than the first audio source distance threshold (“No” path from block 886), in some examples (such as in soft alignment), the one or more processors of the content consumer device 14 may determine a second audio source distance (890). For example, the one or more processors of the content consumer device 14 may determine a distance between the audio stream coordinates of the second audio stream in the acoustic space and the device coordinates in the acoustic space. The one or more processors of the content consumer device 14 may compare the second audio source distance to the second audio source distance threshold (892) to determine whether the second audio source distance is greater than the second audio source distance threshold. When the second audio source distance is greater than the second audio source distance threshold (“Yes” path from block 892), the one or more processors of the content consumer device 14 select the first audio stream and the second audio stream (894) because the first audio source distance is greater than the first audio source distance threshold and the second audio source distance is greater than the second audio source distance threshold. For example, the one or more processors of the content consumer device may combine the first audio stream and the second audio stream by at least one of adaptively mixing the first audio stream and the second audio stream or interpolating a third audio stream based on the first audio stream and the second audio stream. In some examples, one or more processors of content consumer device 14 may combine the first audio stream and the second audio stream by applying a function F(x) to the first audio stream and the second audio stream. Figure 5A-Figure 5C The one or more processors of content consumer device 14 may output one or more speaker feeds based on the first audio stream and the second audio stream (898).

[0127] When the second audio source distance is less than or equal to the second audio source distance threshold ("No" path from block 892), the one or more processors select the second audio source (896). In some examples, the second audio stream is the only audio stream selected. The one or more processors of content consumer device 14 may output one or more speaker feeds based on the second audio stream (898).

[0128] In some examples, the one or more processors of the content consumer device 14 may not immediately select the first audio stream and / or the second audio stream. For example, the one or more processors of the content consumer device 14 may determine whether the device coordinates are stable relative to the first audio source distance threshold and the second audio source distance threshold over a predetermined time period. For example, remaining stable means that the first audio source distance has not changed relative to the first audio source distance threshold (e.g., has not changed from less than or equal to greater than or vice versa) and the second audio source distance has not changed relative to the second audio source distance threshold (e.g., has not changed from less than or equal to greater than or vice versa) over a predetermined time period. Based on the device coordinates being stable relative to the first audio source distance threshold and the second audio source distance threshold over a predetermined time period, the one or more processors of the content consumer device 14 may select the first audio stream, the first audio stream and the second audio stream, or the second audio stream. Since the user may be moving along the boundary of the first audio source distance threshold or the second audio source distance threshold and may be receiving an audio experience that may not be desirable because the one or more processors of the content consumer device 14 may repeatedly change the selected stream, it may be desirable to utilize the present technology.

[0129] In some examples, one or more processors of content consumer device 14 may select a different audio stream based on a change in device coordinates, such as when the user moves. For example, the user may move away from one audio source and toward another audio source. In this case, one or more processors of content consumer device 14 may select the audio source that the user is moving toward. In some examples, one or more processors of content consumer device 14 may provide an alert to the user based on the first audio source distance being equal to the audio source distance threshold, as this may indicate that the audio experience the user is having is changing. In some examples, the alert may be at least one of a visual alert, an audible alert, or a tactile alert.

[0130] Alignment will now be discussed in more detail. Figure 5A is a conceptual diagram illustrating an example of hard alignment. In hard alignment, the threshold (w) used to weight the audio streams may be the distance from the position 416 of the listener 410 (also referred to as the virtual position of the device or user) to the nearest capture or synthesis position A or B. The audio element 434 at position A and the audio element 436 at position B may represent audio signals such as from an audio capture device or a predefined listening position.

[0131] The first audio source distance threshold, such as alignment position S 435, may be set anywhere between position A and position B. In some examples, the listener 410 may set or move the alignment position S 435, for example, through a user interface (such as the user interface disclosed herein). In some examples, the alignment position S 435 may be preset by the source device 12, provided in the bitstream 27, or generated by the content consumer device 14. The alignment position S 435 may represent a value for the alignment position S 435. Figure 4E An example of a first audio source distance threshold is discussed.

[0132] When the audio elements 434 and 436 located at positions A and B, respectively, are the same type of audio capture devices, the midpoint between A and B may be set as the default position of position S 435. In the case where the midpoint is the default position of position S 435 (e.g., Figure 5A 435 ). In some examples, the listener 410 may override the selection of position S 435 to bias the alignment toward an audio element or set of audio elements. For example, the listener 410 may move position S 435 closer to position A. When the listener 410 moves position S 435 closer to position A, the listener 410 may hear the audio signal represented by the audio element 436 when the listener 410 is located at the midpoint between position A and position B. When the position 416 of the listener 410 changes to be closer to the position A than the position S 435, the audio provided to the listener 410 may be aligned with the position A, so that the audio signal presented to the listener 416 is the audio signal represented by the audio element 434. When the position 416 of the listener 410 is closer to the position B than the position S 435, the audio provided to the listener 410 may be aligned with the position B, and the audio provided to the listener 410 may be the audio signal represented by the audio element 436.

[0133] For example, a device such as VR device 400 may determine virtual position information representing a virtual position of the device in a displayed world (e.g., position 416 of listener 410). VR device 400 may compare the virtual position information to a first audio source distance threshold (e.g., position S 435) and select at least one audio stream (e.g., audio element 434 or 436) that may be provided to a user (e.g., a listener audio element) based on the comparison.

[0134] Visual or auditory feedback may be provided to alert the listener 410 that they have snapped to a listening position (e.g., position A or position B). For example, the VR device 400 may provide a visual alert (such as a fade to black, a lightning bolt, and / or a visual marker at the new snapped position (e.g., position A or position B)). In the event that the snapped position may be outside the field of view of the listener 410 (e.g., behind the listener), the visual alert may indicate that the snapped position is outside the field of view, such as a fade to black, a lightning bolt, or some other technique that may be associated with the snapped position being outside the field of view of the listener 410.

[0135] In some examples, the listener 410 may hear a fade in / out during the alignment or some other audio alert indicating that the alignment has occurred. For example, the earphones 404 may provide the listener 410 with a fade in / out, a ring tone, a jingle, a camera sound, a word (e.g., "aligned"), or some other indication that the alignment has occurred.

[0136] In some examples, the listener 410 may feel a tactile alert (such as a vibration) during the alignment, or some other tactile alert indicating that the alignment has occurred. For example, the content consumer device 14 may vibrate to notify the listener 410 that the alignment has occurred.

[0137] Figure 5B is a conceptual diagram showing an example of soft alignment. Figure 5B When the listener 410 is between a first audio source distance threshold (e.g., alignment position x1) and a second audio source distance threshold (e.g., alignment position x2) (soft alignment region), soft alignment may occur when the alignment is softened by the function F(x). Figure 5B In , F(x) is shown as a linear function, but F(x) can be any function.

[0138] In this way, audio rendering of an audio element 438 at position A or an audio element 440 at position B may be prioritized by a device such as headphones 404. In some examples, the region may be an asymmetric region and include hysteresis behavior. For example, a device such as VR device 400 may determine whether a position 416 of a listener 410 remains stable relative to a first audio source distance threshold (e.g., alignment position x1 439) and a second audio source distance threshold (e.g., alignment position x2 441) for a predetermined period of time. Based on the device determining that the position 416 of the listener 410 has been stable for a predetermined period of time, an alignment or transition of the audio presented to the listener 410 may occur. For example, the device may determine that the user's virtual position has moved from between alignment position x1 439 and alignment position x2 441 to between position x2 441 and position B (e.g., the second audio source distance is less than the second audio source distance threshold). Once the predetermined time has passed, the audio presented to the listener 410 may be aligned to the audio element 440 at position B.

[0139] In other respects, the functionality of soft alignment may be similar to that of hard alignment. When the position 416 of the listener 410 is between the alignment position x1 439 and the position A (w = A when A ≤ x ≤ x1, for example, the first audio source distance is less than or equal to the first audio source distance threshold), the listener 410 may align to the position A and hear the audio represented by the audio component 438 at the position A. When the position 416 of the listener 410 is between the alignment position x1 439 and the alignment position x2 441 (w = F(x) when x1 < x < x2, for example, both the first audio source distance is greater than the first audio source distance threshold and the second audio source distance is greater than the second audio source distance threshold), the listener 410 may hear the audio represented by both the audio component 438 and the audio component 440 according to the function F(x). For example, when the position 416 of the listener 410 is between the alignment position x1 439 and the alignment position x2 441 (in the soft alignment region), the device may adaptively mix the audio represented by both the audio component 438 and the audio component 440 according to F(x), or interpolate a third audio stream based on the audio represented by the audio component 438, the audio component 440, and F(x). When the position 416 of the listener 410 is between the position B and the alignment position x2 441 (w = B when x2 ≤ x ≤ B, for example, the second audio source distance is less than or equal to the second audio source distance threshold), the listener 410 may align to the position B and hear the audio represented by the audio component 440 at the position B. As with hard alignment, a visual alert or an auditory alert may be presented to the listener 410. In some examples, a visual alert or an auditory alert may be presented when the alignment occurs and when the position 416 of the listener 410 enters the region between the alignment position x1 and the alignment position x2. For example, when the position 416 of the listener 410 is between the position A and the alignment position x1, the audio source w is at the position A (e.g., the audio component 438). In some examples, a cross-fade may occur and depends on both time and the movement of the listener.

[0140] Figure 5C is a conceptual schematic diagram showing another example of soft alignment. In Figure 5C the example, the threshold region between the position x1 and the position x2 extends to the entire region between the position A and the position B. In this example, when the position 416 of the listener 410 is at the position A, the listener 410 may only hear the audio from the position A, when the position 416 of the listener 410 is between the position A and the position B, may hear a mixture of the audio from the position A and the position B, and when the position 416 of the listener 410 is at the position B, only hear the audio from the position B.

[0141] The example mentioned above ( Figure 5A-Figure 5C) may be a position-dependent alignment. In other examples, the alignment may not be position-dependent. For example, the listener 410 may be able to Figure 4C 's user interface) rather than being aligned to a position by the movement position 416 of the listener 410.

[0142] Figure 5D is a conceptual diagram showing an example having three or more audio elements. Figure 5D The layout of a concert is depicted. In this example, as a listener moves between multiple audio elements (S1 446, S2 448, S3 450, S4 452) close to the stage and audio elements (R1 454, R2 456, R3 458, R4 460) away from the stage, the listener may wish to align with the audio elements that are farther away from the listener's position.

[0143] For example, if the listener wants to hear the performers up close (e.g., close to the stage 444), but wants to see more of the stage (e.g., a wider view than a position close to the stage) such that the listener's position 430 is further away from the stage, the listener may bias the aligned audio source distance threshold toward the stage audio element S2 448 instead of the closest audio element R2 456 (distance a > distance b). In some examples, with this bias toward the stage audio element, as the listener moves toward position 432, they remain aligned to S2 448. At position 432, the listener may align with audio element S3 450 because the listener's distance (c) to audio element S3 450 is less than the listener's distance (d) to audio element S2 448. Without the bias, as the listener moves from position 430 to position 432, the listener would align to audio elements R2 456 and R3 458.

[0144] Figure 6is a schematic diagram showing an example of a wearable device 500 that can operate according to various aspects of the technology described in the present disclosure. In various examples, the wearable device 500 can represent a VR headset (such as the VR device 400 described above), an AR headset, an MR headset, or any other type of extended reality (XR) headset. Augmented reality "AR" can refer to computer-rendered images or data that are overlaid on the real world where the user is actually located. Mixed reality "MR" can refer to computer-rendered images or data that are world-locked to a specific location in the real world, or can refer to a variation on VR in which partly computer-rendered 3D elements and partly photographed real elements are combined into an immersive experience that simulates the user's physical presence in the environment. Extended reality "XR" can be an umbrella term for VR, AR, and MR. More information about XR terminology can be found in the document entitled "Virtual Reality, Augmented Reality, and Mixed Reality Definitions" by Jason Peterson dated July 7, 2017.

[0145] Wearable device 500 may represent other types of devices, such as watches (including so-called “smart watches”), glasses (including so-called “smart glasses”), head-mounted devices (including so-called “wireless head-mounted devices” and “smart head-mounted devices”), smart clothing, smart jewelry, etc. Regardless of representing a VR device, a watch, glasses, and / or a head-mounted device, wearable device 500 may communicate with a computing device that supports wearable device 500 via a wired connection or a wireless connection.

[0146] In some instances, a computing device supporting wearable device 500 may be integrated within wearable device 500, and thus, wearable device 500 may be considered to be the same device as the computing device supporting wearable device 500. In other instances, wearable device 500 may communicate with a separate computing device supporting wearable device 500. In this regard, the term "supporting" should not be understood as requiring a separate dedicated device, but rather one or more processors configured to perform various aspects of the techniques described in this disclosure may be integrated within wearable device 500 or within a computing device separate from wearable device 500.

[0147] For example, when wearable device 500 represents VR device 400, a separate dedicated computing device (such as a personal computer including one or more processors) can present audio and visual content, while wearable device 500 can determine translational head motion according to various aspects of the technology described in this disclosure, and the dedicated computing device can present audio content thereon (as a speaker feed) based on the translational head motion. As another example, when wearable device 500 represents smart glasses, wearable device 500 can include one or more processors that determine translational head motion (through connections within one or more sensors of wearable device 500) and present a speaker feed based on the determined translational head motion.

[0148] As shown, the wearable device 500 includes one or more directional speakers, and one or more tracking and / or recording cameras. In addition, the wearable device 500 includes one or more inertial, tactile and / or health sensors, one or more eye tracking cameras, one or more high-sensitivity audio microphones, and optical / projection hardware. The optical / projection hardware of the wearable device 500 may include durable translucent display technology and hardware.

[0149] The wearable device 500 also includes connection hardware, which may represent one or more network interfaces that support multi-mode connections (such as 4G communication, 5G communication, Bluetooth, Wi-Fi, etc.). The wearable device 500 also includes one or more ambient light sensors, one or more cameras and night vision sensors, and one or more bone conduction transducers. In some cases, the wearable device 500 may also include one or more passive and / or active cameras with fisheye lenses and / or telephoto lenses. Although Figure 6 Not shown, but wearable device 500 may also include one or more light emitting diode (LED) lights. In some examples, the LED lights may be referred to as "super bright" LED lights. In some implementations, wearable device 500 may also include one or more rear cameras. It will be appreciated that wearable device 500 may exhibit a variety of different form factors.

[0150] Additionally, tracking and recording cameras and other sensors can aid in the determination of translation distance. Figure 6 Not shown in the example of , the wearable device 500 may include other types of sensors for detecting the translation distance.

[0151] Although described with respect to specific examples of wearable devices, such as those described above with respect to Figure 6 The VR device 400 discussed in the example Figure 1A and Figure 1B Other devices described in the examples, but ordinary technicians in this field should understand that Figure 1A , Figure 1Band Figure 2 The description can be applied to other examples of wearable devices. For example, other wearable devices such as smart glasses can include sensors through which translational head motion is obtained. As another example, other wearable devices such as smart watches can include sensors through which translational motion is obtained. Therefore, the techniques described in this disclosure should not be limited to a particular type of wearable device, but rather any wearable device can be configured to perform the techniques described in this disclosure.

[0152] Fig. 7A and Figure 7B is a schematic diagram illustrating an example system that can perform various aspects of the techniques described in this disclosure. Fig. 7A An example is shown in which source device 12C also includes camera 600. Camera 600 may be configured to capture video data and provide the captured raw video data to content capture device 20. Content capture device 20 may provide the video data to another component of source device 12C for further processing into viewport-divided portions.

[0153] exist Fig. 7A In the example of , content consumer device 14C also includes VR device 400. It will be appreciated that in various implementations, VR device 400 may be included in content consumer device 14C or externally coupled to content consumer device 14C. VR device 400 includes display hardware and speaker hardware for outputting video data (e.g., associated with various viewports) and for rendering audio data.

[0154] Figure 7B Shows Fig. 7A 3 is replaced by a binaural renderer 42 that is capable of performing binaural rendering using one or more HRTFs or other functions that are capable of rendering to left and right speaker feeds 43. The audio playback system 16C of the content consumer device 14D may output the left and right speaker feeds 43 to headphones 48.

[0155] The headphones 48 may be coupled to the audio playback system 16C via a wired connection (such as a standard 3.5 mm audio jack, a universal system bus (USB) connection, an optical audio jack, or other form of wired connection) or wirelessly (such as through a Bluetooth connection, a wireless network connection, etc.). The headphones 48 may recreate the sound field represented by the audio data 19' based on the left and right speaker feeds 43. The headphones 48 may include a left headphone speaker and a right headphone speaker that are powered (or in other words, driven) by the corresponding left and right speaker feeds 43.

[0156] Figure 8 It is shown Figure 1A-Figure 1CA block diagram of example components of one or more of a source device and a content consumer device as shown in the example of FIG. Figure 8 In an example where device 710 is a mobile device, device 710 includes processor 712 (which may be referred to as "one or more processors" or "processor(s)"), graphics processing unit (GPU) 714, system memory 716, display processor 718, one or more integrated speakers 740, display 703, user interface 720, antenna 721, and transceiver module 722. In an example where device 710 is a mobile device, display processor 718 is a mobile display processor (MDP). In some examples, such as examples where device 710 is a mobile device, processor 712, GPU 714, and display processor 718 may be formed as an integrated circuit (IC).

[0157] For example, an IC can be considered a processing chip within a chip package and can be a system on a chip (SoC). In some examples, two of processor 712, GPU 714, and display processor 718 can be housed together in the same IC, while the other can be housed in a different integrated circuit (e.g., a different chip package), or all three can be housed in different ICs or on the same IC. However, in an example where device 710 is a mobile device, processor 712, GPU 714, and display processor 718 can all be housed in different integrated circuits.

[0158] Examples of processor 712, GPU 714, and display processor 718 include, but are not limited to, one or more digital signal processors, general-purpose microprocessors, application-specific integrated circuits, field programmable logic arrays, or other equivalent integrated or discrete logic circuits. Processor 712 may be a central processing unit (CPU) of device 710. In some examples, GPU 714 may be specialized hardware that includes integrated and / or discrete logic circuits that provide GPU 714 with massively parallel processing capabilities suitable for graphics processing. In some instances, GPU 714 may also include general-purpose processing capabilities and may be referred to as a general-purpose GPU (GPGPU) when implementing general-purpose processing tasks (e.g., non-graphics-related tasks). Display processor 718 may also be application-specific integrated circuit hardware that is designed to retrieve image content from system memory 716, synthesize image content into image frames, and output image frames to display 703.

[0159] Processor 712 can execute various types of applications. Examples of applications include web browsers, email applications, spreadsheets, video games, other applications that generate visual objects for display, or any of the types of applications listed in detail above. System memory 716 can store instructions for executing applications. Executing one of the applications on processor 712 causes processor 712 to generate graphics data of image content to be displayed and audio data 19 to be played (possibly via integrated speakers 740). Processor 712 can send graphics data of image content to GPU 714 for further processing based on instructions or commands sent by processor 712 to GPU 714.

[0160] Processor 712 may communicate with GPU 714 according to an application specific processing interface (API). Examples of such APIs include of API, Khronos group or However, aspects of the present disclosure are not limited to DirectX, OpenGL or OpenCL APIs and can be extended to other types of APIs. In addition, the techniques described in the present disclosure do not need to work according to an API, and the processor 712 and the GPU 714 can communicate using any process.

[0161] The system memory 716 may be a memory of the device 710. The system memory 716 may include one or more computer-readable storage media. Examples of the system memory 716 include, but are not limited to, random access memory (RAM), electrically erasable programmable read-only memory (EEPROM), flash memory, or other media that can be used to carry or store desired program code in the form of instructions and / or data structures and can be accessed by a computer or processor.

[0162] In some examples, system memory 716 may include instructions that cause processor 712, GPU 714, and / or display processor 718 to perform the functions assigned to processor 712, GPU 714, and / or display processor 718 in the present disclosure. Thus, system memory 716 may be a computer-readable storage medium having instructions stored thereon that, when executed, cause one or more processors (e.g., processor 712, GPU 714, and / or display processor 718) to perform various functions.

[0163] System memory 716 may include non-transitory storage media. The term "non-transitory" indicates that the storage media is not contained in a carrier wave or propagating signal. However, the term "non-transitory" should not be interpreted to mean that system memory 716 is non-removable or that its contents are static. As an example, system memory 716 can be removed from device 710 and moved to another device. As another example, a memory substantially similar to system memory 716 can be inserted into device 710. In some examples, non-transitory storage media can store data that can change over time (e.g., in RAM).

[0164] The user interface 720 may represent one or more hardware or virtual (meaning a combination of hardware and software) user interfaces through which a user may interact with the device 710. The user interface 720 may include physical buttons, switches, toggles, lights, or virtual versions thereof. The user interface 720 may also include a physical or virtual keyboard, a touch interface such as a touch screen, haptic feedback, and the like.

[0165] The processor 712 may include one or more hardware units (including so-called "processing cores") configured to perform all or some of the operations described above with respect to one or more of any modules, units, or other functional components of the content creator device and / or the content consumer device. The antenna 721 and the transceiver module 722 may represent units configured to establish and maintain a connection between the source device 12 and the content consumer device 14. The antenna 721 and the transceiver module 722 may represent one or more receivers and / or one or more transmitters capable of wireless communication according to one or more wireless communication protocols, such as fifth generation (5G) cellular standards, such as Bluetooth, etc. TM The antenna 721 and the transceiver module 722 may be configured to receive encoded audio data. Similarly, the antenna 721 and the transceiver module 722 may be configured to send encoded audio data.

[0166] Fig. 9An example of a wireless communication system 100 according to aspects of the present disclosure is shown. The wireless communication system 100 includes a base station 105, a UE 115, and a core network 130. In some examples, the wireless communication system 100 can be a long term evolution (LTE) network, an advanced (LTE-A) network, an LTE-A Pro network, or a new radio (NR) network. In some cases, the wireless communication system 100 can support enhanced broadband communications, ultra-reliable (e.g., mission-critical) communications, low-latency communications, or communications with low-cost and low-complexity devices.

[0167] The base station 105 may communicate wirelessly with the UE 115 via one or more base station antennas. The base station 105 described herein may include or may be referred to by those skilled in the art as a basic transceiver, a radio base station, an access point, a radio transceiver, a NodeB, an eNodeB (eNB), a next generation NodeB or a giga NodeB (any of which may be referred to as a gNB), a home NodeB, a home eNodeB, or some other suitable terminology. The wireless communication system 100 may include different types of base stations 105 (e.g., macro cell base stations or small cell base stations). The UE 115 described herein may be able to communicate with various types of base stations 105 and network devices, including macro eNBs, small cell eNBs, gNBs, relay base stations, etc.

[0168] Each base station 105 may be associated with a particular geographic coverage area 110 in which it supports communications with various UEs 115. Each base station 105 may provide communications coverage for a respective geographic coverage area 110 via a communication link 125, and the communication link 125 between the base station 105 and the UE 115 may utilize one or more carriers. The communication link 125 shown in the wireless communication system 100 may include an uplink transmission from the UE 115 to the base station 105, or a downlink transmission from the base station 105 to the UE 115. Downlink transmissions may also be referred to as forward link transmissions, and uplink transmissions may also be referred to as reverse link transmissions.

[0169] The geographic coverage area 110 of the base station 105 can be divided into sectors that constitute a portion of the geographic coverage area 110, and each sector can be associated with a cell. For example, each base station 105 can provide communication coverage for a macro cell, a small cell, a hotspot, or other types of cells, or various combinations thereof. In some examples, the base station 105 can be mobile and thus provide communication coverage for a mobile geographic coverage area 110. In some examples, different geographic coverage areas 110 associated with different technologies can overlap, and overlapping geographic coverage areas 110 associated with different technologies can be supported by the same base station 105 or different base stations 105. The wireless communication system 100 may include, for example, a heterogeneous LTE / LTE-A / LTE-A Pro or NR network in which different types of base stations 105 provide coverage for various geographic coverage areas 110.

[0170] UE 115 can be dispersed throughout the wireless communication system 100, and each UE 115 can be fixed or mobile.UE 115 can also be referred to as a mobile device, a wireless device, a remote device, a handheld device, or a subscriber device, or some other suitable terminology, where "device" can also be referred to as a unit, a station, a terminal, or a client.UE 115 can also be a personal electronic device, such as a cellular phone, a personal digital assistant (PDA), a tablet computer, a laptop computer, or a personal computer. In an example of the present disclosure, UE 115 can be any of the audio sources described in the present disclosure, including a VR headset, an XR headset, an AR headset, a vehicle, a smart phone, a microphone, a microphone array, or any other device including a microphone or capable of sending a captured and / or synthesized audio stream. In some examples, the synthesized audio stream can be an audio stream stored in a memory or previously created or synthesized. In some examples, UE 115 can also refer to a wireless local loop (WLL) station, an Internet of Things (IoT) device, an Internet of Everything (IoE) device, or an MTC device, etc., which can be implemented in various items such as electrical appliances, vehicles, meters, etc.

[0171] Some UEs 115, such as MTC or IoT devices, may be low-cost or low-complexity devices and may provide automatic communication between machines (e.g., via machine-to-machine (M2M) communication). M2M communication or MTC may refer to data communication technology that allows devices to communicate with each other or with base station 105 without human intervention. In some examples, M2M communication or MTC may include communications from devices that exchange and / or use audio metadata indicating privacy restrictions and / or password-based privacy data to switch, mask, and / or nullify various audio streams and / or audio sources.

[0172] In some cases, UE 115 is also able to communicate directly with other UE 115 (e.g., using peer-to-peer (P2P) or device-to-device (D2D) protocols). One or more UEs in a group of UEs 115 utilizing D2D communication may be within the geographic coverage area 110 of the base station 105. Other UEs 115 in the group may be outside the geographic coverage area 110 of the base station 105, or otherwise unable to receive transmissions from the base station 105. In some cases, groups of UEs 115 communicating via D2D communication may utilize a one-to-many (1:M) system in which each UE 115 transmits to each other UE 115 in the group. In some cases, the base station 105 facilitates resource scheduling for D2D communication. In other cases, D2D communication is performed between UEs 115 without the involvement of the base station 105.

[0173] The base stations 105 may communicate with the core network 130 and with each other. For example, the base stations 105 may interface with the core network 130 via a backhaul link 132 (e.g., via S1, N2, N3, or other interfaces). The base stations 105 may communicate with each other directly (e.g., directly between the base stations 105) or indirectly (e.g., via the core network 130) via a backhaul link 134 (e.g., via X2, Xn, or other interfaces).

[0174] In some cases, the wireless communication system 100 can utilize both licensed radio bands and unlicensed radio bands. For example, the wireless communication system 100 can adopt license assisted access (LAA), LTE unlicensed (LTE-U) radio access technology, or NR technology in unlicensed bands such as the 5GHz Industrial Science Medical (ISM) band. When operating in an unlicensed radio band, wireless devices such as base stations 105 and UE 115 can adopt a pre-talk listen (LBT) process to ensure that the frequency channel is idle before sending data. In some cases, operations in unlicensed bands can be based on a combination of carrier aggregation configurations and component carriers operating in licensed bands (e.g., LAA). Operations in unlicensed spectrum can include downlink transmissions, uplink transmissions, peer-to-peer transmissions, or a combination of these transmissions. Duplexing in unlicensed spectrum can be based on frequency division duplexing (FDD), time division duplexing (TDD), or a combination of both.

[0175] This disclosure includes the following examples.

[0176] Example 1A. A device configured to play one or more of a plurality of audio streams, the device comprising: a memory configured to store a plurality of audio streams and corresponding audio metadata, each audio stream representing a sound field, and the audio metadata comprising capture location information representing capture coordinates at which a corresponding one of the plurality of audio streams was captured in a display world; and one or more processors coupled to the memory and configured to: determine location information representing device coordinates of the device in the display world; select a subset of the plurality of audio streams based on the location information and the capture location information, the subset of the plurality of audio streams excluding at least one of the plurality of audio streams; and output one or more speaker feeds based on the subset of the plurality of audio streams.

[0177] Example 2A. A device according to Example 1A, wherein the one or more processors are configured to: determine a proximity distance based on the position information and capture position information included in metadata of at least one of the multiple audio streams; and select a subset of the multiple audio streams based on the proximity distance.

[0178] Example 3A. A device according to any combination of Example 1A and Example 2A, wherein one or more processors are configured to: determine a proximity distance between device coordinates of the device in a display world represented by the position information and each captured coordinate represented by the capture position information to obtain multiple proximity distances; and select a subset of multiple audio streams based on the multiple proximity distances.

[0179] Example 4A. A device according to any combination of Example 2A and Example 3A, wherein one or more processors are configured to: compare the proximity distance to a threshold proximity distance; and when the proximity distance is less than or equal to the threshold proximity distance, select a larger number of the plurality of audio streams to obtain a subset of the plurality of audio streams than when the proximity distance is greater than the threshold proximity distance.

[0180] Example 5A. A device according to any combination of Example 2A and Example 3A, wherein one or more processors are configured to: compare the proximity distance to a threshold proximity distance; and when the proximity distance is greater than the threshold proximity distance, select a smaller number of multiple audio streams to obtain a subset of the multiple audio streams compared to when the proximity distance is less than or equal to the threshold proximity distance.

[0181] Example 6A. A device according to any combination of Examples 1A-5A, wherein the one or more processors are further configured to: obtain a new audio stream and corresponding new audio metadata, the new audio metadata including capture location information representing capture coordinates in the display world where the new audio stream is captured; and update multiple audio streams to include the new audio stream.

[0182] Example 7A. A device according to any combination of Examples 1A-6A, wherein one or more processors are configured to: determine, based on multiple audio streams, an energy map representing energy of a common sound field represented by the multiple audio streams; and determine a subset of the multiple audio streams based on the energy map, virtual position information, and capture position information.

[0183] Example 8A. The device of Example 7A, wherein the one or more processors are configured to: analyze the energy map to determine audio stream coordinates of an audio source in a common sound field; and determine a subset of the plurality of audio streams based on the audio source position, the position information, and the capture position information.

[0184] Example 9A. The device of Example 8A, wherein the one or more processors are configured to: determine a distance between the audio stream coordinates and the device coordinates of the device as an audio source distance; compare the audio source distance to an audio source distance threshold; and when the audio source distance is greater than the audio source distance threshold, select a single audio stream from among the plurality of audio streams as a subset of the plurality of audio streams, the single audio stream being one of the plurality of audio streams having a capture position closest to a virtual position of the device.

[0185] Example 10A. The device of Example 8A, wherein the one or more processors are configured to: determine a distance between an audio stream coordinate and a device coordinate of the device as an audio source distance; compare the audio source distance to an audio source distance threshold; and when the audio source distance is less than or equal to the audio source distance threshold, select more than one audio stream of a plurality of audio streams as a subset of the plurality of audio streams, the more than one audio streams being a subset of the plurality of audio streams having corresponding capture positions surrounding a virtual position of the device.

[0186] Example 11A. A device according to any combination of Examples 1A-10A, wherein the device includes an extended reality head-mounted device, and wherein the displayed world includes a scene represented by video data captured by a camera.

[0187] Example 12A. A device according to any combination of Examples 1A-10A, wherein the device includes an extended reality head-mounted device, and wherein the displayed world includes a virtual world.

[0188] Example 13A. The device of any combination of Example 1A-Example 12A, further comprising a head mounted display configured to present a display world.

[0189] Example 14A. A device according to any combination of Example 1A-Example 12A, wherein the device includes a mobile phone.

[0190] Example 15A. The device of any combination of Example 1A-Example 14A, further comprising a receiver configured to receive a plurality of audio streams.

[0191] Example 16A. The device of Example 15A, wherein the receiver comprises a receiver configured to receive multiple audio streams according to a fifth generation (5G) cellular standard.

[0192] Example 17A. The device of Example 15A, wherein the receiver comprises a receiver configured to receive the plurality of audio streams according to a personal area network standard.

[0193] Example 18A. A device according to any combination of Examples 1A-17A, wherein the device includes one or more speakers configured to reproduce one or more of the sound fields represented by the subset of the audio stream based on the speaker feeds.

[0194] Example 19A. A method for playing one or more audio streams among multiple audio streams, the method comprising: storing the multiple audio streams and corresponding audio metadata by a memory, each audio stream representing a sound field, and the audio metadata including capture position information representing a capture position at which a corresponding one of the multiple audio streams is captured in a display world; and determining, by one or more processors, virtual position information representing a virtual position of the device in the display world; selecting, by the one or more processors and based on the virtual position information and the capture position information, a subset of the multiple audio streams, the subset of the multiple audio streams not including at least one audio stream among the multiple audio streams; and reproducing, by the one or more processors and based on the subset of the multiple audio streams, a corresponding sound field.

[0195] Example 20A. A method according to Example 19A, wherein selecting a subset of the multiple audio streams includes: determining a proximity distance based on position information and capture position information included in metadata of at least one of the multiple audio streams; and selecting the subset of the multiple audio streams based on the proximity distance.

[0196] Example 21A. A method according to any combination of Example 19A and Example 20A, wherein selecting a subset of multiple audio streams includes: determining a proximity distance between a device coordinate of the device in a display world represented by the position information and each capture coordinate represented by the capture position information to obtain a plurality of proximity distances; and selecting a subset of the multiple audio streams based on the plurality of proximity distances.

[0197] Example 22A. A method according to any combination of Example 20A and Example 21A, wherein selecting a subset of the multiple audio streams includes: comparing the proximity distance to a threshold proximity distance; and when the proximity distance is less than or equal to the threshold proximity distance, selecting a larger number of the multiple audio streams to obtain the subset of the multiple audio streams than when the proximity distance is greater than the threshold proximity distance.

[0198] Example 23A. A method according to any combination of Example 20A and Example 21A, wherein selecting a subset of the multiple audio streams includes: comparing the proximity distance to a threshold proximity distance; and when the proximity distance is greater than the threshold proximity distance, selecting a smaller number of the multiple audio streams to obtain the subset of the multiple audio streams than when the proximity distance is less than or equal to the threshold proximity distance.

[0199] Example 24A. A method according to any combination of Examples 19A-23A, further comprising: obtaining a new audio stream and corresponding new audio metadata, the new audio metadata including capture position information representing capture coordinates in a display world where the new audio stream is captured; and updating multiple audio streams to include the new audio stream.

[0200] Example 25A. A method according to any combination of Examples 19A-24A, wherein determining virtual position information includes: determining, based on multiple audio streams, an energy map representing energy of a common sound field represented by the multiple audio streams; and determining a subset of the multiple audio streams based on the energy map, the virtual position information, and the capture position information.

[0201] Example 26A. The method of Example 25A, wherein selecting a subset of the multiple audio streams includes: analyzing an energy map to determine audio stream coordinates of an audio source in a common sound field; and determining the subset of the multiple audio streams based on the audio source position, the position information, and the capture position information.

[0202] Example 27A. The method of Example 26A, wherein selecting a subset of multiple audio streams includes: determining a distance between audio stream coordinates and device coordinates of a device as an audio source distance; comparing the audio source distance to an audio source distance threshold; and when the audio source distance is greater than the audio source distance threshold, selecting a single audio stream from the multiple audio streams as the subset of the multiple audio streams, the single audio stream being one of the multiple audio streams having a capture position closest to a virtual position of the device.

[0203] Example 28A. The method of Example 26A, wherein selecting a subset of multiple audio streams includes: determining a distance between audio stream coordinates and device coordinates of a device as an audio source distance; comparing the audio source distance to an audio source distance threshold; and when the audio source distance is less than or equal to the audio source distance threshold, selecting more than one audio stream from the multiple audio streams as a subset of the multiple audio streams, the more than one audio streams being a subset of the multiple audio streams having a capture position closest to a virtual position of the device.

[0204] Example 29A. A method according to any combination of Examples 19A-28A, wherein the method is performed by an extended reality head-mounted device, and wherein the displayed world includes a scene represented by video data captured by a camera.

[0205] Example 30A. A method according to any combination of Examples 19A-28A, wherein the method is performed by an extended reality head mounted device, and wherein the displayed world includes a virtual world.

[0206] Example 31A. A method according to any combination of Examples 19A-30A, further comprising presenting the display world via a head-mounted display.

[0207] Example 32A. A method according to any combination of Examples 19A-30A, wherein the one or more processors are included in a mobile phone.

[0208] Example 33A. A method according to any combination of Examples 19A-32A, further comprising receiving multiple audio streams.

[0209] Example 34A. The method of Example 33A, wherein receiving multiple audio streams comprises: receiving multiple audio streams according to a fifth generation (5G) cellular standard.

[0210] Example 35A. The method of Example 33A, wherein receiving the plurality of audio streams comprises receiving the plurality of audio streams according to a personal area network standard.

[0211] Example 36A. A method according to any combination of Examples 19A-35A, further comprising reproducing one or more of the sound fields represented by the subset of the audio stream based on speaker feeds.

[0212] Example 37A. A device configured to play one or more audio streams of a plurality of audio streams, the device comprising: a component for storing the plurality of audio streams and corresponding audio metadata, each audio stream representing a sound field, and the audio metadata including capture position information representing a capture position at which a corresponding one of the plurality of audio streams was captured in a display world; and a component for determining virtual position information representing a virtual position of the device in the display world; a component for selecting a subset of the plurality of audio streams based on the virtual position information and the capture position information, the subset of the plurality of audio streams not including at least one of the plurality of audio streams; and a component for reproducing a corresponding sound field based on the subset of the plurality of audio streams.

[0213] Example 38A. An apparatus according to Example 37A, wherein the components for selecting a subset of multiple audio streams include: components for determining a proximity distance based on position information and capture position information included in metadata of at least one of the multiple audio streams; and components for selecting the subset of the multiple audio streams based on the proximity distance.

[0214] Example 39A. A device according to any combination of Example 37A and Example 38A, wherein the component for selecting a subset of multiple audio streams includes: a component for determining a proximity distance between a device coordinate of the device in a display world represented by the position information and each capture coordinate represented by the capture position information to obtain a plurality of proximity distances; and a component for selecting a subset of the multiple audio streams based on the plurality of proximity distances.

[0215] Example 40A. An apparatus according to any combination of Example 38A and Example 39A, wherein the means for selecting a subset of the multiple audio streams includes: means for comparing the proximity distance to a threshold proximity distance; and means for selecting a larger number of the multiple audio streams to obtain the subset of the multiple audio streams when the proximity distance is less than or equal to the threshold proximity distance than when the proximity distance is greater than the threshold proximity distance.

[0216] Example 41A. An apparatus according to any combination of Example 38A and Example 39A, wherein the means for selecting a subset of the multiple audio streams includes: means for comparing the proximity distance to a threshold proximity distance; and means for selecting a smaller number of the multiple audio streams to obtain the subset of the multiple audio streams when the proximity distance is greater than the threshold proximity distance than when the proximity distance is less than or equal to the threshold proximity distance.

[0217] Example 42A. A device according to any combination of Examples 37A-41A, further comprising: a component for obtaining a new audio stream and corresponding new audio metadata, the new audio metadata including capture location information representing capture coordinates in a display world where the new audio stream is captured; and a component for updating multiple audio streams to include the new audio stream.

[0218] Example 43A. A device according to any combination of Examples 37A-42A, wherein the component for determining virtual position information includes: a component for determining, based on multiple audio streams, an energy map representing energy of a common sound field represented by the multiple audio streams; and a component for determining a subset of the multiple audio streams based on the energy map, the virtual position information, and the capture position information.

[0219] Example 44. An apparatus according to Example 43A, wherein the component for selecting a subset of multiple audio streams includes: a component for analyzing an energy map to determine audio stream coordinates of an audio source in a common sound field; and a component for determining a subset of multiple audio streams based on the audio source position, position information, and capture position information.

[0220] Example 45A. The device of Example 44, wherein the means for selecting a subset of the plurality of audio streams comprises: means for determining a distance between audio stream coordinates and device coordinates of the device as an audio source distance; means for comparing the audio source distance to an audio source distance threshold; and means for selecting, when the audio source distance is greater than the audio source distance threshold, a single audio stream from the plurality of audio streams as the subset of the plurality of audio streams, the single audio stream being one of the plurality of audio streams having a capture position closest to a virtual position of the device.

[0221] Example 46A. A device according to Example 44A, wherein the means for selecting a subset of multiple audio streams includes: means for determining a distance between audio stream coordinates and device coordinates of the device as an audio source distance; means for comparing the audio source distance to an audio source distance threshold; and means for selecting more than one audio stream of the multiple audio streams as a subset of the multiple audio streams when the audio source distance is less than or equal to the audio source distance threshold, the more than one audio streams being a subset of the multiple audio streams having corresponding capture positions surrounding a virtual position of the device.

[0222] Example 47A. A device according to any combination of Examples 37A-46A, wherein the device includes an extended reality head-mounted device, and wherein the displayed world includes a scene represented by video data captured by a camera.

[0223] Example 48A. A device according to any combination of Examples 37A-46A, wherein the device includes an extended reality head-mounted device, and wherein the displayed world includes a virtual world.

[0224] Example 49A. The device of any combination of Examples 37A-48A, further comprising means for presenting a display world.

[0225] Example 50A. A device according to any combination of Example 37A-Example 48A, wherein the device includes a mobile phone.

[0226] Example 51A. An apparatus according to any combination of Examples 37A-50A, further comprising means for receiving multiple audio streams.

[0227] Example 52A. The device of Example 51A, wherein the means for receiving multiple audio streams includes: means for receiving multiple audio streams according to a fifth generation (5G) cellular standard.

[0228] Example 53A. The device of Example 51A, wherein the means for receiving the plurality of audio streams comprises: means for receiving the plurality of audio streams according to a personal area network standard.

[0229] Example 54A. An apparatus according to any combination of Examples 37A-53A, further comprising means for reproducing one or more of the sound fields represented by the subset of the audio stream based on speaker feeds.

[0230] Example 55A. A non-transitory computer-readable storage medium having instructions stored thereon that, when executed, cause one or more processors to: store multiple audio streams and corresponding audio metadata, each audio stream representing a sound field, the audio metadata including capture position information indicating a capture position in a display world at which a corresponding one of the multiple audio streams was captured; connect to a head-mounted display to present the display world; determine virtual position information indicating a virtual position of the device in the display world; based on the virtual position information and the capture position information, select a subset of the multiple audio streams, the subset of the multiple audio streams excluding at least one audio stream of the multiple audio streams; and reproduce the corresponding sound field based on the subset of the multiple audio streams.

[0231] Example 1B. A device configured to play one or more of a plurality of audio streams, the device comprising: a memory configured to store a plurality of audio streams and corresponding audio metadata, each audio stream representing a sound field, and the audio metadata comprising capture position information representing capture coordinates at which a corresponding one of the plurality of audio streams was captured in a display world; and one or more processors coupled to the memory and configured to: determine virtual position information representing device coordinates of the device in the display world; determine an audio source distance as a distance between the audio stream coordinates and the device coordinates of the device; compare the audio source distance to an audio source distance threshold; when the audio source distance is greater than the audio source distance threshold, select a single audio stream of the plurality of audio streams; and output one or more speaker feeds based on the single audio stream.

[0232] Example 2B. The device of Example 1B, wherein the audio source distance threshold is provided by a user.

[0233] Example 3B. The device of Example 1B, wherein the audio source distance threshold is predetermined.

[0234] Example 4B. The device of Example 1B, wherein the audio source distance threshold is received in the bitstream.

[0235] Example 5B. A device according to any combination of Examples 1B-4B, wherein the audio source distance is a first audio source distance, the audio stream coordinates are the first audio stream coordinates, the audio source distance threshold is the first audio source distance threshold, and the single audio stream is a first single audio stream, and the one or more processors are further configured to: determine a second audio source distance as a distance between the second audio stream coordinates and the device coordinates of the device; compare the second audio source distance to the second audio source distance threshold; when the audio source distance is between the first audio source distance threshold and the second audio source threshold, select a first single audio stream in the plurality of audio streams and a second single audio stream in the plurality of audio streams; and output one or more speaker feeds based on the first single audio stream and the second single audio stream.

[0236] Example 6B. The device of Example 5B, wherein the one or more processors are further configured to: combine the first single audio stream and the second single audio stream.

[0237] Example 7B. A device according to Example 6B, wherein the one or more processors are configured to: combine the first single audio stream and the second single audio stream by at least one of adaptively mixing the first single audio stream and the second single audio stream or interpolating a third audio stream based on the first single audio stream and the second single audio stream.

[0238] Example 8B. The device of Example 6B or 7B, wherein the one or more processors are configured to combine the first single audio stream and the second single audio stream by applying a function F(x) to the first single audio stream and the second single audio stream.

[0239] Example 9B. A device according to any combination of Examples 6B-8B, wherein the one or more processors are further configured to: determine whether the virtual position information is stable relative to the first audio source distance threshold and the second audio source distance threshold over a predetermined time period; and select the first single audio stream, the first single audio stream and the second single audio stream, or the second single audio stream based on the virtual position being stable relative to the first audio source distance threshold and the second audio source distance threshold over the predetermined time period.

[0240] Example 10B. The device of Example 5B, wherein the one or more processors are further configured to: select a second single audio stream from the plurality of audio streams when the second audio source distance is greater than a second audio source threshold; and output one or more speaker feeds based on the second single audio stream.

[0241] Example 11B. The device of any combination of Examples 1B-10B, wherein the one or more processors are further configured to: select a different audio stream based on a change in the virtual position information.

[0242] Example 12B. The device of any combination of Examples 1B-4B, wherein the one or more processors are further configured to: provide an alert to a user based on the virtual location information indicating a distance equal to the audio source threshold.

[0243] Example 13B. The device of Example 12B, wherein the alarm is at least one of a visual alarm or an audible alarm.

[0244] Example 14B. A device according to any combination of Examples 1B-13B, wherein the device includes an extended reality head-mounted device, and wherein the displayed world includes a virtual world.

[0245] Example 15B. The device of any combination of Example 1B-Example 13B, further comprising a head mounted display configured to present a display world and a graphical user interface.

[0246] Example 16B. A device according to any combination of Example 1B-Example 13B, wherein the device includes a mobile phone.

[0247] Example 17B. The device of any combination of Example 1B-Example 16B, further comprising a receiver configured to receive a plurality of audio streams.

[0248] Example 18B. The device of Example 17B, wherein the receiver comprises a receiver configured to receive multiple audio streams according to a fifth generation (5G) cellular standard.

[0249] Example 19B. The device of Example 17B, wherein the receiver comprises a receiver configured to receive the plurality of audio streams according to a personal area network standard.

[0250] Example 20B. A method comprising: storing, by a memory, a plurality of audio streams and corresponding audio metadata, each audio stream representing a sound field, and the audio metadata including capture position information representing capture coordinates of a corresponding one of the plurality of audio streams captured in a display world; determining, by one or more processors, virtual position information representing device coordinates of a device in the display world; comparing, by the one or more processors, the virtual position information to an audio source distance threshold; selecting, by the one or more processors, a single audio stream from the plurality of audio streams when the audio source distance is greater than the audio source distance threshold; and outputting, by the one or more processors and based on the single audio stream, one or more speaker feeds.

[0251] Example 21B. The method of Example 20B, wherein the audio source distance threshold is provided by a user.

[0252] Example 22B. The method of Example 20B, wherein the audio source distance threshold is predetermined.

[0253] Example 23B. The method of Example 20B, wherein the audio source distance threshold is received in the bitstream.

[0254] Example 24B. A method according to any combination of Examples 20B-23B, wherein the audio source distance is a first audio source distance, the audio stream coordinates are the first audio stream coordinates, the audio source distance threshold is the first audio source distance threshold, and the single audio stream is a first single audio stream, and the method further includes: determining, by the one or more processors, a second audio source distance as a distance between the second audio stream coordinates and the device coordinates of the device; comparing, by the one or more processors, the second audio source distance to the second audio source distance threshold; selecting, by the one or more processors and when the audio source distance is between the first audio source distance threshold and the second audio source threshold, a first single audio stream among the multiple audio streams and a second single audio stream among the multiple audio streams; and outputting, by the one or more processors and based on the first single audio stream and the second single audio stream, one or more speaker feeds.

[0255] Example 25B. The method of Example 24B, further comprising combining, by one or more processors, the first single audio stream and the second single audio stream.

[0256] Example 26B. The method of Example 25B, wherein the combining comprises at least one of: adaptively mixing the first single audio stream and the second single audio stream or interpolating the third audio stream based on the first single audio stream and the second single audio stream.

[0257] Example 27B. The method of Example 25B or 26B, wherein the combining comprises applying a function F(x) to the first single audio stream and the second single audio stream.

[0258] Example 28B. The method of any combination of Examples 24B-27B, further comprising: determining, by one or more processors, whether the virtual position information is stable relative to a first audio source distance threshold and a second audio source distance threshold over a predetermined time period; and selecting, by one or more processors, the first single audio stream, the first single audio stream and the second single audio stream, or the second single audio stream based on the virtual position being stable relative to the first audio source distance threshold and the second audio source distance threshold over a predetermined time period.

[0259] Example 29B. The method of Example 24B, further comprising: selecting a second single audio stream from the plurality of audio streams when the second audio source distance is greater than a second audio source threshold; and outputting one or more speaker feeds based on the second single audio stream.

[0260] Example 30B. The method of any combination of Examples 20B-29B, further comprising selecting, by one or more processors, a different audio stream based on a change in virtual position information.

[0261] Example 31B. The method of any combination of Examples 20B-30B, further comprising providing an alert to a user based on the virtual location information being equal to an audio source distance threshold.

[0262] Example 32B. The method of Example 31B, wherein the alarm is at least one of a visual alarm or an audible alarm.

[0263] Example 33B. A non-transitory computer-readable storage medium having instructions stored thereon that, when executed, cause one or more processors to: store multiple audio streams and corresponding audio metadata, each audio stream representing a sound field, and the audio metadata including capture position information representing capture coordinates at which a corresponding one of the multiple audio streams was captured in a display world; determine virtual position information representing device coordinates of the device in the display world; determine an audio source distance as a distance between the audio stream coordinates and the device coordinates of the device; compare the audio source distance to an audio source distance threshold; when the audio source distance is greater than the audio source distance threshold, select a single audio stream from the multiple audio streams; and output one or more speaker feeds based on the single audio stream.

[0264] Example 34B. A device comprising: means for storing multiple audio streams and corresponding audio metadata, each audio stream representing a sound field, and the audio metadata including capture position information representing capture coordinates at which a corresponding one of the multiple audio streams was captured in a display world; means for determining virtual position information representing device coordinates of the device in the display world; means for comparing the virtual position information to an audio source distance threshold; means for selecting a single audio stream from the multiple audio streams when the audio source distance is greater than the audio source distance threshold; and means for outputting one or more speaker feeds based on the single audio stream.

[0265] Example 1C. A content consumer device configured to play one or more audio streams of a plurality of audio streams, the content consumer device comprising: a memory configured to store a plurality of audio streams and audio position information associated with the plurality of audio streams, the audio position information representing audio stream coordinates in an acoustic space of captured audio streams or audio stream coordinates in a virtual acoustic space of synthesized audio streams, or both, each audio stream representing a sound field; and one or more processors coupled to the memory and configured to: determine device position information representing device coordinates of the content consumer device in the acoustic space; select a subset of the plurality of audio streams based on the device position information and the audio position information, the subset of the plurality of audio streams excluding at least one audio stream of the plurality of audio streams; and output one or more speaker feeds based on the subset of the plurality of audio streams.

[0266] Example 2C. A content consumer device according to Example 1C, wherein the one or more processors are further configured to: determine a proximity distance for at least one of the multiple audio streams based on the device location information and the audio location information; and select a subset of the multiple audio streams based on the proximity distance.

[0267] Example 3C. A content consumer device according to Example 2C, wherein the one or more processors are configured to: compare the proximity distance to a threshold proximity distance; and when the proximity distance is less than or equal to the threshold proximity distance, select a larger number of multiple audio streams to obtain a subset of the multiple audio streams than when the proximity distance is greater than the threshold proximity distance.

[0268] Example 4C. A content consumer device according to any combination of Examples 2C-3C, wherein one or more processors are configured to: compare the proximity distance to a threshold proximity distance; and when the proximity distance is greater than the threshold proximity distance, select a smaller number of multiple audio streams to obtain a subset of the multiple audio streams than when the proximity distance is less than or equal to the threshold proximity distance.

[0269] Example 5C. A content consumer device according to any combination of Examples 1C-4C, wherein the one or more processors are further configured to: obtain a new audio stream and corresponding new audio position information; and update the subset of the plurality of audio streams to include the new audio stream.

[0270] Example 6C. A content consumer device according to any combination of Examples 1C-5C, wherein one or more processors are further configured to: determine, based on multiple audio streams, an energy map representing energy of a common sound field represented by the multiple audio streams; and determine a subset of the multiple audio streams based on the energy map, device position information, and audio position information.

[0271] Example 7C. A content consumer device according to Example 6C, wherein the one or more processors are further configured to: analyze the energy map to determine an audio source position of the audio streams in a common sound field; and determine a subset of the multiple audio streams based on the audio source position, device position information, and audio position information.

[0272] Example 8C. The content consumer device of Example 7C, wherein the one or more processors are further configured to: determine an audio source distance as a distance between an audio source position and the device coordinates; compare the audio source distance to an audio source distance threshold; and when the audio source distance is greater than the audio source distance threshold, select a single audio stream from the plurality of audio streams as a subset of the plurality of audio streams, the single audio stream being associated with the audio stream coordinates having the shortest distance to the device coordinates.

[0273] Example 9C. The content consumer device of Example 7C, wherein the one or more processors are configured to: determine an audio source distance as a distance between an audio source position and a device coordinate; compare the audio source distance to an audio source distance threshold; and when the audio source distance is less than or equal to the audio source distance threshold, select more than one audio stream of a plurality of audio streams as a subset of the plurality of audio streams, the more than one audio stream being a subset of the plurality of audio streams having audio stream coordinates surrounding the device coordinates.

[0274] Example 10C. A content consumer device according to any combination of Examples 1C-9C, wherein the one or more processors are further configured to: determine a first audio source distance as a distance between first audio stream coordinates of a first audio stream and device coordinates; compare the first audio source distance to a first audio source distance threshold; when the first audio source distance is less than or equal to the first audio source distance threshold, select a first audio stream from a plurality of audio streams; and output one or more speaker feeds based on the first audio stream, wherein the first audio stream is the only audio stream selected.

[0275] Example 11C. The content consumer device of Example 10C, wherein the one or more processors are further configured to: determine a second audio source distance as a distance between second audio stream coordinates of a second audio stream and device coordinates; compare the second audio source distance to a second audio source distance threshold; select a first audio stream of the plurality of audio streams and a second audio stream of the plurality of audio streams when the first audio source distance is greater than the first audio source distance threshold and the second audio source distance is greater than the second audio source distance threshold; and output one or more speaker feeds based on the first audio stream and the second audio stream.

[0276] Example 12C. The content consumer device of Example 11C, wherein the one or more processors are configured to combine the first audio stream and the second audio stream by at least one of adaptively mixing the first audio stream and the second audio stream or interpolating a third audio stream based on the first audio stream and the second audio stream.

[0277] Example 13C. The content consumer device of Example 12C, wherein the one or more processors are configured to combine the first audio stream and the second audio stream by applying a function F(x) to the first audio stream and the second audio stream.

[0278] Example 14C. A content consumer device according to any combination of Examples 11C-13C, wherein the one or more processors are further configured to: determine whether device coordinates are stable relative to a first audio source distance threshold and a second audio source distance threshold over a predetermined time period; and select the first audio stream, the first audio stream and the second audio stream, or the second audio stream based on the device coordinates being stable relative to the first audio source distance threshold and the second audio source distance threshold over the predetermined time period.

[0279] Example 15C. A content consumer device according to any combination of Examples 11C-14C, wherein the one or more processors are further configured to: select a second audio stream from the plurality of audio streams when the second audio source distance is less than or equal to a second audio source threshold; and output one or more speaker feeds based on the second audio stream, wherein the second audio stream is the only audio stream selected.

[0280] Example 16C. The content consumer device of any combination of Examples 11C-15C, wherein the one or more processors are further configured to select a different audio stream based on a change in device coordinates.

[0281] Example 17C. A content consumer device according to any combination of Examples 10C-16C, wherein the one or more processors are further configured to: provide an alert to a user based on the first audio source distance being equal to a first audio source distance threshold, wherein the alert is at least one of a visual alert, an audible alert, or a tactile alert.

[0282] Example 18C. A content consumer device according to any combination of Examples 1C-17C, wherein the audio stream coordinates in the acoustic space or the audio stream coordinates in the virtual acoustic space are coordinates in the display world associated with which the corresponding audio stream is captured or synthesized.

[0283] Example 19C. A content consumer device according to Example 18C, wherein the content consumer device comprises an extended reality head-mounted device, and wherein the displayed world comprises a scene represented by video data captured by a camera.

[0284] Example 20C. The content consumer device of Example 19C, wherein the content consumer device comprises an extended reality head mounted device, and wherein the displayed world comprises a virtual world.

[0285] Example 21C. A content consumer device according to any combination of Examples 1C-20C, wherein the content consumer device comprises a mobile phone.

[0286] Example 22C. A content consumer device according to any combination of Examples 1C-21C, further comprising a transceiver configured to wirelessly receive multiple audio streams, wherein the transceiver is configured to wirelessly receive multiple audio streams according to at least one of a fifth generation (5G) cellular standard, a personal area network standard, or a local area network standard.

[0287] Example 23C. The content consumer device of any combination of Examples 1C-22C, wherein the one or more processors are further configured to: decode only a subset of the plurality of audio streams in response to the selection.

[0288] Example 24C. A content consumer device according to any combination of Examples 1C-23C, wherein the one or more processors are further configured to: determine the audio source distance as the distance between the audio source and the device coordinates in the acoustic space; compare the audio source distance to an audio source distance threshold; and when the audio source distance is greater than the audio source distance threshold, select a single audio stream from the multiple audio streams as a subset of the multiple audio streams, the single audio stream having the shortest audio source distance.

[0289] Example 25C. A method for playing one or more audio streams of a plurality of audio streams, the method comprising: storing, by a memory of a content consumer device, a plurality of audio streams and audio position information associated with the plurality of audio streams, the audio position information representing audio stream coordinates in an acoustic space of captured audio streams or audio stream coordinates in a virtual acoustic space of synthesized audio streams, or both, each audio stream representing a sound field; and determining, by one or more processors of the content consumer device, device position information representing device coordinates of the content consumer device in the acoustic space; selecting, by the one or more processors and based on the device position information and the audio position information, a subset of the plurality of audio streams, the subset of the plurality of audio streams not including at least one audio stream of the plurality of audio streams; and outputting, by the one or more processors and based on the subset of the plurality of audio streams, one or more speaker feeds.

[0290] Example 26C. The method of Example 25C, wherein selecting a subset of the multiple audio streams includes: determining a proximity distance for at least one of the multiple audio streams based on device position information and audio position information; and selecting the subset of the multiple audio streams based on the proximity distance.

[0291] Example 27C. A method according to Example 26C, wherein selecting a subset of multiple audio streams includes: comparing the proximity distance to a threshold proximity distance; and when the proximity distance is less than or equal to the threshold proximity distance, selecting a larger number of multiple audio streams to obtain a subset of multiple audio streams than when the proximity distance is greater than the threshold proximity distance.

[0292] Example 28C. A method according to any combination of Examples 25C-27C, wherein selecting a subset of multiple audio streams includes: comparing the proximity distance to a threshold proximity distance; and when the proximity distance is greater than the threshold proximity distance, selecting a smaller number of multiple audio streams to obtain a subset of multiple audio streams than when the proximity distance is less than or equal to the threshold proximity distance.

[0293] Example 29C. A method according to any combination of Examples 25C-28C, further comprising: obtaining a new audio stream and corresponding new audio position information; and updating a subset of the multiple audio streams to include the new audio stream.

[0294] Example 30C. A method according to any combination of Examples 25C-29C, further comprising: determining, by one or more processors and based on multiple audio streams, an energy map representing energy of a common sound field represented by the multiple audio streams; and determining, by one or more processors, a subset of the multiple audio streams based on the energy map, device position information, and audio position information.

[0295] Example 31C. The method of Example 30C, wherein selecting a subset of the multiple audio streams includes: analyzing an energy map to determine audio stream coordinates of audio sources in a common sound field; and determining the subset of the multiple audio streams based on the audio source coordinates, the position information, and the audio position information.

[0296] Example 32C. The method of Example 31C, wherein selecting a subset of multiple audio streams includes: determining an audio source distance as a distance between audio stream coordinates and device coordinates; comparing the audio source distance to an audio source distance threshold; and when the audio source distance is greater than the audio source distance threshold, selecting a single audio stream from the multiple audio streams as a subset of the multiple audio streams, the single audio stream having the shortest audio source distance.

[0297] Example 33C. The method of Example 31C, wherein selecting a subset of the multiple audio streams includes: determining an audio source distance as a distance between an audio stream coordinate and a device coordinate; comparing the audio source distance to an audio source distance threshold; and when the audio source distance is less than or equal to the audio source distance threshold, selecting more than one audio stream of the multiple audio streams as a subset of the multiple audio streams, the more than one audio streams being a subset of the multiple audio streams having audio stream coordinates surrounding the device coordinates.

[0298] Example 34C. The method of any combination of Examples 25C-33C, further comprising: determining, by one or more processors, a first audio source distance as a distance between first audio stream coordinates of a first audio stream and device coordinates; comparing, by one or more processors, the first audio source distance to a first audio source distance threshold; selecting, by the one or more processors, a first audio stream from a plurality of audio streams when the first audio source distance is less than or equal to the first audio source distance threshold; and outputting, by the one or more processors, one or more speaker feeds based on the first audio stream, wherein the first audio stream is the only audio stream selected.

[0299] Example 35C. The method of Example 34C, further comprising: determining, by one or more processors, a second audio source distance as a distance between second audio stream coordinates of a second audio stream and device coordinates; comparing, by one or more processors, the second audio source distance to a second audio source distance threshold; selecting, by the one or more processors, a first audio stream among a plurality of audio streams and a second audio stream among a plurality of audio streams when the first audio source distance is greater than the first audio source distance threshold and the second audio source distance is greater than the second audio source distance threshold; and outputting, by the one or more processors, one or more speaker feeds based on the first audio stream and the second audio stream.

[0300] Example 36C. The method of Example 35C further includes: combining, by one or more processors, the first audio stream and the second audio stream by at least one of adaptively mixing the first audio stream and the second audio stream or interpolating a third audio stream based on the first audio stream and the second audio stream.

[0301] Example 37C. The method of Example 35C, wherein the combining comprises: applying a function F(x) to the first audio stream and the second audio stream.

[0302] Example 38C. The method of any combination of Examples 35C-37C, further comprising: determining, by one or more processors, whether the device coordinate information is stable relative to a first audio source distance threshold and a second audio source distance threshold over a predetermined time period; and selecting, by the one or more processors, the first single audio stream, the first single audio stream and the second single audio stream, or the second single audio stream based on the device coordinates being stable relative to the first audio source distance threshold and the second audio source distance threshold over the predetermined time period.

[0303] Example 39C. A method according to any combination of Examples 35C-38C, further comprising: selecting, by one or more processors and when the second audio source distance is less than or equal to a second audio source threshold, a second audio stream from a plurality of audio streams; and outputting one or more speaker feeds based on the second audio stream, wherein the second audio stream is the only audio stream selected.

[0304] Example 40C. The method of any combination of Examples 35C-39C, further comprising: selecting, by one or more processors, a different audio stream based on a change in device coordinates.

[0305] Example 41C. The method of Examples 34C-40C, further comprising providing an alert to a user based on the first audio source distance being equal to a first audio source distance threshold, wherein the alert is at least one of a visual alert, an audible alert, or a tactile alert.

[0306] Example 42C. A method according to any combination of Examples 25C-41C, wherein the content consumer device includes an extended reality head-mounted device, and wherein the displayed world includes a scene represented by video data captured by a camera.

[0307] Example 43C. A method according to any combination of Examples 25C-41C, wherein the content consumer device includes an extended reality head-mounted device, and wherein the displayed world includes a virtual world.

[0308] Example 44C. A method according to any combination of Examples 25C-43C, wherein the content consumer device includes a mobile phone.

[0309] Example 45C. A method according to any combination of Examples 25C-44C, further comprising: wirelessly receiving multiple audio streams through a transceiver module of a content consumer device, wherein wirelessly receiving multiple audio streams comprises wirelessly receiving multiple audio streams according to a fifth generation (5G) cellular standard, a personal area network standard, or a local area network standard.

[0310] Example 46C. The method of any combination of Examples 25C-45C, further comprising: in response to the selection, decoding, by one or more processors, only a subset of the multiple audio streams.

[0311] Example 47C. The method of any combination of Examples 25C-46C, further comprising: determining, by one or more processors, the audio source distance as the distance between the audio source and the device coordinates in the acoustic space; comparing, by one or more processors, the audio source distance to an audio source distance threshold; and selecting, by one or more processors, a single audio stream from the multiple audio streams as a subset of the multiple audio streams when the audio source distance is greater than the audio source distance threshold, the single audio stream having the shortest audio source distance.

[0312] Example 48C. A content consumer device configured to play one or more of a plurality of audio streams, the content consumer device comprising: components for storing a plurality of audio streams and audio position information associated with the plurality of audio streams, the audio position information representing audio stream coordinates in an acoustic space of captured audio streams or audio stream coordinates in a virtual acoustic space of synthesized audio streams, or both, each audio stream representing a sound field; and components for determining device position information representing device coordinates of the content consumer device in the acoustic space; components for selecting a subset of the plurality of audio streams based on the device position information and the audio position information, the subset of the plurality of audio streams excluding at least one of the plurality of audio streams; and components for outputting one or more speaker feeds based on the subset of the plurality of audio streams.

[0313] Example 49C. A non-transitory computer-readable storage medium having instructions stored thereon that, when executed, cause one or more processors of a content consumer device to: store multiple audio streams and audio position information associated with the multiple audio streams, the audio position information representing audio stream coordinates in an acoustic space of captured audio streams or audio stream coordinates in a virtual acoustic space of synthesized audio streams, or both, each audio stream representing a sound field; and determine device position information representing device coordinates of the content consumer device in the acoustic space; based on the device position information and the audio position information, select a subset of the multiple audio streams, the subset of the multiple audio streams excluding at least one audio stream from the multiple audio streams; and output one or more speaker feeds based on the subset of the multiple audio streams.

[0314] It should be appreciated that, depending on the example, certain actions or events of any technique described in the present invention may be performed in a different order, may be added, combined, or omitted entirely (e.g., not all described actions or events are required to implement the technique). In addition, in some examples, actions or events may be performed concurrently rather than sequentially, such as through multithreading, interrupt handling, or multiple processors.

[0315] In some examples, a VR device (or streaming device) can communicate using a network interface coupled to a memory of the VR / streaming device to exchange messages to an external device, wherein the exchanged messages are associated with representations of multiple available sound fields. In some embodiments, the VR device can receive wireless signals using an antenna coupled to the network interface, including data packets, audio packets, video packets, or transport protocol data associated with representations of multiple available sound fields. In some examples, one or more microphone arrays can capture the sound field.

[0316] In some examples, the multiple available representations of the sound field stored on the storage device may include multiple object-based representations of the sound field, a high-order ambisonic representation of the sound field, a mixed-order ambisonic representation of the sound field, a combination of the object-based representation of the sound field and a high-order ambisonic representation of the sound field, a combination of the object-based representation of the sound field and a mixed-order ambisonic representation of the sound field, or a combination of the mixed-order representation of the sound field and a high-order ambisonic representation of the sound field.

[0317] In some examples, one or more of the multiple available sound field representations may include at least one high-resolution region and at least one lower-resolution region, and wherein the presentation selected based on the steering angle provides higher spatial accuracy relative to the at least one high-resolution region and less spatial accuracy relative to the lower-resolution region.

[0318] In one or more examples, the described functions may be implemented in hardware, software, firmware, or any combination thereof. When implemented in software, the functions may be stored on or sent via a computer-readable medium as one or more instructions or codes and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media, which corresponds to tangible media such as data storage media, or communication media, including, for example, any media that facilitates the transfer of a computer program from one place to another according to a communication protocol. In this manner, computer-readable media may generally correspond to (1) a non-temporary tangible computer-readable storage medium, or (2) a communication medium such as a signal or carrier wave. Data storage media may be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, codes, and / or data structures to implement the techniques described in the present invention. A computer program product may include a computer-readable medium.

[0319] As an example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage devices, flash memory or any other medium that can be used to store the required program code in the form of instructions or data structures and can be accessed by a computer. Moreover, any connection is appropriately referred to as a computer-readable medium. For example, if a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wireless technology (such as infrared, radio and microwave) is used to send instructions from a website, server or other remote source, the definition of the medium includes coaxial cable, fiber optic cable, twisted pair, DSL or wireless technology such as infrared, radio and microwave. However, it should be understood that computer-readable storage media and data storage media do not include connections, carriers, signals or other temporary media, but are directed to non-temporary tangible storage media. The disks and optical disks used in the present invention include compact disks (CDs), laser optical disks, optical disks, digital versatile disks (DVDs), floppy disks and blue-ray disks, wherein the disks usually reproduce data magnetically, while the optical disks reproduce data by laser optical methods. The above combination should also be included in the scope of computer-readable media.

[0320] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Therefore, the term "processor" as used in the present invention may refer to any of the aforementioned structures or any other structure suitable for implementing the techniques described in the present invention. In addition, in some aspects, the functions described in the present invention may be provided in dedicated hardware and / or software modules configured for encoding and decoding or incorporated in a combined codec. Similarly, the technology may be fully implemented in one or more circuits or logic elements.

[0321] The techniques of the present disclosure may be implemented in a variety of devices or apparatuses including a wireless handset, an integrated circuit (IC), or a set of ICs (e.g., a chipset). Various components, modules, or units are described in this disclosure to emphasize functional aspects of devices configured to perform the disclosed techniques, but do not necessarily need to be implemented by different hardware units. Rather, as described above, the various units may be combined in a codec hardware unit, or provided by a collection of interoperable hardware units, including one or more processors as described above in combination with applicable software and / or firmware.

[0322] Various examples have been described. These and other examples are within the scope of the following claims.

Claims

1. A content consumer device, configured to play one or more audio streams of a plurality of audio streams, the content consumer device comprising: a memory configured to store the plurality of audio streams and audio position information associated with the plurality of audio streams, the audio position information representing audio stream coordinates in an acoustic space of captured audio streams or audio stream coordinates in a virtual acoustic space of synthesized audio streams, or both, each of the audio streams representing a sound field; as well as One or more processors coupled to the memory and configured to: determining device location information representing device coordinates of the content consumer device in the acoustic space; determining an audio source distance as a distance between an audio source in the acoustic space and the device coordinates based on the audio position information; comparing the audio source distance to an audio source distance threshold, the audio source distance threshold being definable by a user of the content consumer device via a user interface; When the audio source distance is greater than the audio source distance threshold, selecting a single audio stream from the multiple audio streams as a subset of the multiple audio streams; as well as One or more speaker feeds are output based on the subset of the plurality of audio streams.

2. The content consumer device of claim 1 , wherein the one or more processors are further configured to: Obtaining a new audio stream and corresponding new audio position information; and The subset of the plurality of audio streams is updated to include the new audio stream.

3. The content consumer device of claim 1 , wherein the one or more processors are further configured to: determining, based on the plurality of audio streams, an energy map representing energy of a common sound field represented by the plurality of audio streams, in, The selection of the single audio stream is also based on the energy map.

4. The content consumer device of claim 3, wherein the one or more processors are further configured to: The energy map is analyzed to determine the audio source location.

5. The content consumer device of claim 4, wherein the one or more processors are configured to: When the audio source distance is less than or equal to the audio source distance threshold, more than one audio stream of the plurality of audio streams is selected as the subset of the plurality of audio streams, the more than one audio stream being the subset of the plurality of audio streams having the audio stream coordinates surrounding the device coordinates.

6. The content consumer device of claim 1 , wherein the one or more processors are further configured to: Determine a first audio source distance as a distance between a first audio stream coordinate of a first audio stream and the device coordinate; comparing the first audio source distance to a first audio source distance threshold; When the first audio source distance is less than or equal to the first audio source distance threshold, selecting the first audio stream among the multiple audio streams; as well as outputting one or more speaker feeds based on the first audio stream, The first audio stream is the only audio stream selected.

7. The content consumer device of claim 6, wherein the one or more processors are further configured to: Determine a second audio source distance as a distance between a second audio stream coordinate of a second audio stream and the device coordinate; comparing the second audio source distance to a second audio source distance threshold; When the first audio source distance is greater than the first audio source distance threshold and the second audio source distance is greater than the second audio source distance threshold, selecting the first audio stream among the multiple audio streams and the second audio stream among the multiple audio streams; as well as One or more speaker feeds are output based on the first audio stream and the second audio stream.

8. The content consumer device of claim 7, wherein the one or more processors are configured to combine the first audio stream and the second audio stream by at least one of adaptively mixing the first audio stream and the second audio stream or interpolating a third audio stream based on the first audio stream and the second audio stream.

9. The content consumer device of claim 8, wherein the one or more processors are configured to combine the first audio stream and the second audio stream by applying a function F(x) to the first audio stream and the second audio stream.

10. The content consumer device of claim 7, wherein the device location information is first device location information and the device coordinates are first device coordinates, and wherein the one or more processors are further configured to: determining second device position information representing second device coordinates of the content consumer device in the acoustic space, the second device coordinates being different from the first device coordinates due to movement of a user of the content consumer device; determining a third audio source distance as a distance between the first audio stream coordinates and the second device coordinates; determining a fourth audio source distance as a distance between the second audio stream coordinates and the second device coordinates; comparing the third audio source distance to the first audio source distance threshold; comparing the fourth audio source distance to the second audio source distance threshold; determining whether the second device coordinates are stable relative to the first audio source distance threshold and the second audio source distance threshold over a predetermined period of time; as well as The first audio stream or the second audio stream is selected based on the second device coordinates being stable relative to the first audio source distance threshold and the second audio source distance threshold over a predetermined time period, a comparison of the third audio source distance to the first audio source distance threshold, and a comparison of the fourth audio source distance to the second audio source distance threshold.

11. The content consumer device of claim 7, wherein the one or more processors are further configured to: When the second audio source distance is less than or equal to the second audio source distance threshold, selecting the second audio stream among the multiple audio streams; and outputting one or more speaker feeds based on the second audio stream, The second audio stream is the only audio stream selected.

12. The content consumer device of claim 7, wherein the one or more processors are further configured to select a different audio stream based on a change in device coordinates.

13. The content consumer device of claim 6, wherein the one or more processors are further configured to provide an alert to a user based on the first audio source distance being equal to the first audio source distance threshold, wherein the alert is at least one of a visual alert, an audible alert, or a tactile alert.

14. The content consumer device of claim 1, wherein the audio stream coordinates in the acoustic space or the audio stream coordinates in the virtual acoustic space are coordinates in a display world associated with a corresponding audio stream being captured or synthesized.

15. The content consumer device according to claim 14, wherein the content consumer device comprises an extended reality head mounted device, and The displayed world includes a scene represented by video data captured by a camera.

16. The content consumer device of claim 14, wherein the content consumer device comprises an extended reality head mounted device, and The display world includes a virtual world.

17. The content consumer device of claim 1, wherein the content consumer device comprises a mobile phone.

18. The content consumer device of claim 1 , further comprising a transceiver configured to wirelessly receive the plurality of audio streams, wherein the transceiver is configured to wirelessly receive the plurality of audio streams in accordance with at least one of a fifth generation (5G) cellular standard, a personal area network standard, or a local area network standard.

19. The content consumer device of claim 1, wherein the one or more processors are further configured to decode only the subset of the plurality of audio streams in response to the selection.

20. A method for playing one or more audio streams among a plurality of audio streams, the method comprising: storing, by a memory of a content consumer device, the plurality of audio streams and audio position information associated with the plurality of audio streams, the audio position information representing audio stream coordinates in an acoustic space in which the audio streams were captured or in a virtual acoustic space in which the audio streams were synthesized, or both, each of the audio streams representing a sound field; as well as determining, by one or more processors of the content consumer device, device location information representing device coordinates of the content consumer device in the acoustic space; determining, by the one or more processors, an audio source distance as a distance between an audio source in the acoustic space and the device coordinates based on the audio position information; comparing, by the one or more processors, the audio source distance to an audio source distance threshold, the audio source distance threshold being definable by a user of the content consumer device via a user interface; and selecting, by the one or more processors, a single audio stream from the plurality of audio streams as a subset of the plurality of audio streams when the audio source distance is greater than the audio source distance threshold; as well as One or more speaker feeds are output by the one or more processors and based on the subset of the plurality of audio streams.

21. The method according to claim 20, further comprising: Obtain a new audio stream and corresponding new audio position information; as well as The subset of the plurality of audio streams is updated to include the new audio stream.

22. The method according to claim 20, further comprising: An energy map representing energy of a common sound field represented by the plurality of audio streams is determined, by the one or more processors and based on the plurality of audio streams, wherein the selection of the single audio stream is further based on the energy map.

23. The method of claim 22, wherein selecting the subset of the plurality of audio streams comprises: The energy map is analyzed to determine audio source locations.

24. The method of claim 23, wherein selecting the subset of the plurality of audio streams comprises: Selecting, by the one or more processors and when the audio source distance is less than or equal to the audio source distance threshold, more than one audio stream of the plurality of audio streams as the subset of the plurality of audio streams, the more than one audio streams being the subset of the plurality of audio streams having audio stream coordinates surrounding the device coordinates.

25. The method of claim 20, further comprising: determining, by the one or more processors, a first audio source distance as a distance between first audio stream coordinates of a first audio stream and the device coordinates; comparing, by the one or more processors, the first audio source distance to a first audio source distance threshold; selecting, by the one or more processors and when the first audio source distance is less than or equal to the first audio source distance threshold, the first audio stream from the plurality of audio streams; as well as outputting, by the one or more processors, one or more speaker feeds based on the first audio stream, The first audio stream is the only audio stream selected.

26. The method according to claim 25, further comprising: determining, by the one or more processors, a second audio source distance as a distance between second audio stream coordinates of a second audio stream and the device coordinates; comparing, by the one or more processors, the second audio source distance to a second audio source distance threshold; selecting, by the one or more processors and when the first audio source distance is greater than the first audio source distance threshold and the second audio source distance is greater than the second audio source distance threshold, the first audio stream among the plurality of audio streams and the second audio stream among the plurality of audio streams; as well as One or more speaker feeds are output by the one or more processors based on the first audio stream and the second audio stream.

27. The method according to claim 26, further comprising: The first audio stream and the second audio stream are combined, by the one or more processors, by at least one of adaptively mixing the first audio stream and the second audio stream or interpolating a third audio stream based on the first audio stream and the second audio stream.

28. The method of claim 27, wherein the combining comprises applying a function F(x) to the first audio stream and the second audio stream.

29. The method of claim 26, wherein the device location information is first device location information, and the device coordinates are first device coordinates, and the method further comprises: determining, by the one or more processors, second device position information representing second device coordinates of the content consumer device in the acoustic space, the second device coordinates being different from the first device coordinates due to movement of a user of the content consumption device; determining, by the one or more processors, a third audio source distance as a distance between the first audio stream coordinates and the second device coordinates; determining, by the one or more processors, a fourth audio source distance as a distance between the second audio stream coordinates and the second device coordinates; comparing, by the one or more processors, the third audio source distance to the first audio source distance threshold; comparing, by the one or more processors, the fourth audio source distance to the second audio source distance threshold; determining, by the one or more processors, whether the second device coordinates are stable relative to the first audio source distance threshold and the second audio source distance threshold over a predetermined period of time; as well as The one or more processors select the first audio stream or the second audio stream based on the second device coordinates being stable relative to the first audio source distance threshold and the second audio source distance threshold over a predetermined time period, a comparison of the third audio source distance to the first audio source distance threshold, and a comparison of the fourth audio source distance to the second audio source distance threshold.

30. The method of claim 26, further comprising: selecting, by the one or more processors, the second audio stream from the multiple audio streams when the second audio source distance is less than or equal to the second audio source distance threshold; as well as outputting one or more speaker feeds based on the second audio stream, The second audio stream is the only audio stream selected.

31. The method of claim 26, further comprising selecting, by the one or more processors, a different audio stream based on the device coordinate change.

32. The method of claim 25, further comprising: An alert is provided to a user based on the first audio source distance being equal to the first audio source distance threshold, wherein the alert is at least one of a visual alert, an audible alert, or a tactile alert.

33. The method according to claim 20, wherein the content consumer device comprises an extended reality head mounted device, and The displayed world includes a scene represented by video data captured by a camera.

34. The method according to claim 20, wherein the content consumer device comprises an extended reality head mounted device, and The displayed world includes the virtual world.

35. The method of claim 20, wherein the content consumer device comprises a mobile phone.

36. The method of claim 20, further comprising wirelessly receiving the plurality of audio streams via a transceiver module of the content consumer device, wherein wirelessly receiving the plurality of audio streams comprises wirelessly receiving the plurality of audio streams in accordance with a fifth generation (5G) cellular standard, a personal area network standard, or a local area network standard.

37. The method of claim 20, further comprising: In response to the selection, only the subset of the plurality of audio streams is decoded by the one or more processors.

38. A content consumer device configured to play one or more audio streams of a plurality of audio streams, the content consumer device comprising: means for storing the plurality of audio streams and audio position information associated with the plurality of audio streams, the audio position information representing the audio stream coordinates in an acoustic space in which the audio streams were captured or in a virtual acoustic space in which the audio streams were synthesized, or both, each of the audio streams representing a sound field; means for determining device position information representing device coordinates of said content consumer device in said acoustic space; means for determining an audio source distance as a distance between an audio source in the acoustic space and the device coordinates based on the audio position information; means for comparing the audio source distance to an audio source distance threshold, the audio source distance threshold being definable by a user of a content consumer device via a user interface; means for selecting a single audio stream of the plurality of audio streams as a subset of the plurality of audio streams when the audio source distance is greater than the audio source distance threshold; as well as Means for outputting one or more speaker feeds based on the subset of the plurality of audio streams.

39. A non-transitory computer-readable storage medium having stored thereon instructions that, when executed, cause one or more processors of a content consumer device to: storing a plurality of audio streams and audio position information associated with the plurality of audio streams, the audio position information representing the audio stream coordinates in an acoustic space of captured audio streams or the audio stream coordinates in a virtual acoustic space of synthesized audio streams, or both, each of the audio streams representing a sound field; determining device location information representing device coordinates of the content consumer device in the acoustic space; determining an audio source distance as a distance between an audio source in the acoustic space and the device coordinates based on the audio position information; comparing the audio source distance to an audio source distance threshold, the audio source distance threshold being definable by a user of the content consumer device via a user interface; When the audio source distance is greater than the audio source distance threshold, selecting a single audio stream from the multiple audio streams as a subset of the multiple audio streams; as well as One or more speaker feeds are output based on the subset of the plurality of audio streams.

Images