User interface for controlling audio rendering for extended reality experiences

Abstract

The device may be configured to play one or more of a plurality of audio streams. The device may include a memory configured to store a plurality of audio streams, each of the audio streams representing a sound field. The device may also include one or more processors coupled to the memory and configured to present a user interface to a user, obtain an indication from the user indicating a desired listening position via the user interface; and select at least one of the plurality of audio streams based on the indication.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Patent Application No. 16 / 918,492 filed on July 1, 2020, U.S. Provisional Patent Application No. 62 / 870,577 filed on July 3, 2019, and U.S. Provisional Patent Application No. 62 / 992,624 filed on March 20, 2020, the entire contents of each of which are incorporated by reference. Technical Field

[0003] The present disclosure relates to the processing of media data, such as audio data. Background Art

[0004] Computer-mediated reality systems are being developed to allow computing devices to augment or add to, delete or subtract from, or generally modify the existing reality that a user experiences. Computer-mediated reality systems (which may also be referred to as "extended reality systems" or "XR systems") may include, for example, virtual reality (VR) systems, augmented reality (AR) systems, and mixed reality (MR) systems. The perceived success of a computer-mediated reality system is generally related to the ability of such a computer-mediated reality system to provide a realistic immersive experience in terms of both video and audio experience (where the video and audio experience are aligned in a manner that the user expects). Although the human visual system is more sensitive than the human auditory system (e.g., in terms of perceived localization of various objects within a scene), ensuring an adequate auditory experience is an increasingly important factor in ensuring a realistic immersive experience, especially as the video experience improves to allow better localization of video objects so that users can better identify the source of audio content. Summary of the invention

[0005] The present disclosure generally relates to the auditory aspects of user experience of computer-mediated reality systems (including virtual reality (VR), mixed reality (MR), augmented reality (AR), computer vision and graphics systems). Various aspects of the technology can provide user control of audio rendering (rendering) for the acoustic space of the extended reality system. As used herein, the acoustic environment is represented as an indoor environment or an outdoor environment, or both an indoor environment and an outdoor environment. The acoustic environment may include one or more sub-acoustic spaces, which may include various acoustic elements. Examples of outdoor environments may include cars, buildings, walls, forests, etc. The acoustic space may be an example of an acoustic environment and may be an indoor space or an outdoor space. As used herein, an audio element may be a sound captured by a microphone (e.g., captured directly from a near-field source or a reflection from a far-field source (whether real or synthetic)), or a previously synthesized sound field, or a monophonic sound synthesized from text to speech, or a reflection of a virtual sound from an object in an acoustic environment.

[0006] In one example, various aspects of the technology relate to a device configured to play one or more of a plurality of audio streams, the device comprising: a memory configured to store a plurality of audio streams, each of the audio streams representing a sound field; and one or more processors coupled to the memory and configured to: present a user interface to a user; obtain an indication from the user via the user interface indicating a desired listening position; and select at least one of the plurality of audio streams based on the indication.

[0007] In another example, various aspects of the technology relate to a method, including: storing multiple audio streams by a memory, each of the audio streams representing a sound field, the memory being communicatively coupled to one or more processors; presenting a user interface by the one or more processors; obtaining, by the one or more processors via the user interface, an indication representing a desired listening position; and selecting, by the one or more processors and based on the indication, at least one of the multiple audio streams.

[0008] In another example, various aspects of the technology relate to a device configured to play one or more of a plurality of audio streams, the device comprising: a device for storing a plurality of audio streams, each of the audio streams representing a sound field; a device for presenting a user interface; a device for obtaining an indication representing a desired listening position via the user interface; and a device for selecting at least one of the plurality of audio streams based on the indication.

[0009] In another example, various aspects of the technology relate to a non-transitory computer-readable storage medium having instructions stored thereon, which, when executed, cause one or more processors to: store multiple audio streams, each of which represents a sound field; present a user interface; obtain an indication of a desired listening position via the user interface; and select at least one of the multiple audio streams based on the indication.

[0010] The details of one or more examples of the disclosure are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of various aspects of the technology will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1A-Figure 1C is a diagram illustrating a system that can perform various aspects of the techniques described in this disclosure.

[0012] Figure 2 is a diagram illustrating an example of a VR device worn by a user.

[0013] Figure 3A-3D A more detailed map shows Figure 1A-Figure 1C A diagram of an example operation of a stream selection unit shown in an example.

[0014] Figure 4A-4E A more detailed map shows Figure 1A-Figure 1C FIG. 1 is a diagram of example operations of a user interface of a content consumer device shown in an example of FIG.

[0015] Figure 5A-5C is a conceptual diagram illustrating an example of snapping according to the technology of the present disclosure.

[0016] Figure 5D and Figure 5E is a conceptual diagram illustrating an example of a concert according to the technology of the present disclosure.

[0017] Fig. 5F and Figure 5G is a flow diagram illustrating an example user interface technique according to the present disclosure.

[0018] Figure 6 is a diagram illustrating an example of a wearable device that can operate according to various aspects of the techniques described in this disclosure.

[0019] Fig. 7A and Figure 7B is a diagram illustrating other example systems that can perform various aspects of the techniques described in this disclosure.

[0020] Figure 8 The diagram shows Figure 1A-Figure 1CA block diagram of example components of one or more of a source device and a content consumer device as shown in an example of FIG.

[0021] Figures 9A-9C The diagram shows Figure 1A-Figure 1C Flowchart of example operations of the stream selection unit shown in the example of FIG. 1 in performing various aspects of the stream selection technique.

[0022] Fig.10 The figure illustrates an example of a wireless communication system according to aspects of the present disclosure. DETAILED DESCRIPTION

[0023] Users in the AR / VR / MR world cannot select a desired listening location, for example, where to obtain (one or more) audio streams. The technology of the present disclosure can enable a user to indicate a desired listening location to a user interface of a device, and the device can select at least one of multiple audio streams based on the indication. In this way, the technology of the present disclosure can achieve an enhanced audio experience, in which a user can select a desired listening location and the device can provide the user with an audio experience as if the user is located at the desired listening location. For example, the technology of the present disclosure can provide an enhanced audio experience in a concert or sporting event environment, in which a user can select a desired listening location near the stage, away from the stage, on the home team's bench, in the press box, and so on.

[0024] There are many different ways to represent a sound field. Example formats include channel-based audio formats, object-based audio formats, and scene-based audio formats. Channel-based audio formats refer to 5.1 surround sound formats, 7.1 surround sound formats, 22.2 surround sound formats, or any other channel-based formats that localize audio channels to specific locations around the listener in order to recreate the sound field.

[0025] An object-based audio format may refer to a format in which audio objects, typically encoded using pulse codec modulation (PCM) and referred to as PCM audio objects, are designated to represent a sound field. Such audio objects may include metadata identifying the position of the audio object relative to a listener or other reference point in the sound field, so that the audio object can be rendered to one or more speaker channels for playback in an effort to recreate the sound field. The techniques described in the present disclosure may be applied to any of the following formats, including scene-based audio formats, channel-based audio formats, object-based audio formats, or any combination thereof.

[0026] A scene-based audio format may include a hierarchical set of elements that define a sound field in three dimensions. An example of a hierarchical set of elements is a set of spherical harmonic coefficients (SHCs). The following expression shows a description or representation of a sound field using SHCs:

[0027]

[0028] The expression shows that at time t, any point in the sound field The pressure at p i Available through SHC Here, c is the speed of sound (~343 m / s), is the reference point (or observation point), j n (·) is a spherical Bessel function of order n, and is a spherical harmonic basis function (also called a spherical basis function) of order n and suborder m. It can be seen that the terms in the square brackets are signals (e.g., ), which can be approximated by various time-frequency transforms, such as discrete Fourier transform (DFT), discrete cosine transform (DCT), or wavelet transform. Other examples of hierarchical sets include sets of wavelet transform coefficients and other sets of coefficients of multiresolution basis functions.

[0029] SHC They can be physically acquired (e.g., recorded) through various microphone array configurations, or alternatively, they can be derived from channel-based or object-based descriptions of the sound field. SHC (also known as ambisonic coefficients) represent scene-based audio, where the SHC can be input to an audio encoder to obtain encoded SHC that can facilitate more efficient transmission or storage. For example, a method involving (1+4) 2 (25, hence fourth order) The fourth order representation of the coefficients.

[0030] As described above, SHC can be derived from microphone recordings using a microphone array. Various examples of how to physically obtain SHC from a microphone array are described in Poletti, M., "Three-Dimensional Surround Sound Systems Based on Spherical Harmonics", J. Audio Eng. Soc., Vol. 53, No. 11, Nov. 2005, pp. 1004-1025.

[0031] The following equations illustrate how SHC can be derived from an object-based description. The coefficients corresponding to the individual audio objects for the sound field are: can be expressed as:

[0032]

[0033] where i is is a spherical Hankel function of order n (second kind), and is the position of the object. Knowing the source energy of the object as a function of frequency g(ω) (e.g., using time-frequency analysis techniques such as performing a Fast Fourier Transform on a stream of Pulse Coded Modulation—PCM—), it is possible to implement a translation of each PCM object and the corresponding position to the SHC Furthermore, it can be shown (because the above is a linear and orthogonal decomposition) that for each object The coefficients are additive. In this way, multiple PCM objects can be represented by The coefficients are represented (e.g., as the sum of coefficient vectors for individual objects). The coefficients may contain information about the sound field (pressure as a function of three-dimensional (3D) coordinates) and represent the distance from individual objects to the observation point. A transformation of the representation of the overall sound field nearby.

[0034] Computer-mediated reality systems (which may also be referred to as "extended reality systems" or "XR systems") are being developed to take advantage of the many potential benefits provided by ambisonic coefficients. For example, ambisonic coefficients may represent a sound field in three dimensions in a manner that potentially enables accurate 3D positioning of sound sources within the sound field. In this manner, an XR device may render the ambisonic coefficients to a speaker feed that, when played back via one or more speakers, may accurately reproduce the sound field.

[0035] As another example, ambisonic coefficients may be transformed (e.g., rotated) to account for user movement without overly complex mathematical operations, thereby potentially accommodating the low latency requirements of XR. Furthermore, ambisonic coefficients are hierarchical and thus naturally accommodate scalability through order reduction (which may eliminate ambisonic coefficients associated with higher orders), and thus potentially enable dynamic adaptation of the sound field to accommodate the latency and / or battery requirements of an XR device.

[0036] Using ambisonics coefficients for XR devices can enable the development of many use cases that rely on the more immersive sound field provided by ambisonics coefficients, especially for computer gaming applications and live video streaming applications. In these high-dynamic use cases that rely on low-latency reproduction of the sound field, XR devices may prefer ambisonics coefficients over other representations that are more difficult to manipulate or involve complex rendering. Figure 1A-Figure 1CProvides more information about these use cases.

[0037] Although described in the present disclosure with respect to VR devices, various aspects of these techniques may be performed in the context of other devices such as mobile devices. In this case, a mobile device (such as a so-called smart phone) may present a display world via a screen that may be mounted on the user's head or viewed in the manner in which the mobile device is normally used. In this way, any information on the screen may be part of the mobile device. The mobile device may be able to provide tracking information and thus allow both a VR experience (when head-mounted) and a normal experience of viewing the display world, where the normal experience may still allow the user to view a display world that provides a VR-lite-type experience (e.g., lifting the device and rotating or translating the device to view different parts of the display world). In addition, although a display world is mentioned in various examples of the present disclosure, the techniques of the present disclosure may also be used with an acoustic space that does not correspond to a display world or in which there is no display world.

[0038] Figure 1A-Figure 1C is a diagram illustrating a system that can perform various aspects of the techniques described in this disclosure. Figure 1A As shown in the example of , system 10 includes a source device 12A and a content consumer device 14A. Although described in the context of source device 12A and content consumer device 14A, these techniques may be implemented in any context in which any representation of a sound field is encoded to form a bitstream representing audio data. In addition, source device 12A may represent any form of computing device capable of generating a representation of a sound field, and is generally described herein in the context of being a VR content creator device. Similarly, content consumer device 14A may represent any form of computing device capable of implementing the rendering metadata techniques and audio playback described in the present disclosure, and is generally described herein in the context of being a VR client device.

[0039] Source device 12A may be operated by an entertainment company or other entity that may generate mono and / or multi-channel audio content for consumption by an operator of a content consumer device, such as content consumer device 14A. In some VR scenarios, source device 12A generates audio content in conjunction with video content. Source device 12A includes content capture device 20, content editing device 22, and sound field representation generator 24. Content capture device 20 may be configured to interface with microphone 18 or otherwise communicate.

[0040] Microphone 18 can indicate Or other types of 3D audio microphones capable of capturing a sound field and representing the sound field as audio data 19, which may refer to one or more of the above-mentioned scene-based audio data (such as ambisonic reverberation coefficients), object-based audio data, and channel-based audio data. Although described as a 3D audio microphone, microphone 18 may also represent other types of microphones (such as omnidirectional microphones, point microphones, unidirectional microphones, etc.) configured to capture audio data 19. Audio data 19 may represent or include an audio stream.

[0041] In some examples, the content capture device 20 may include an integrated microphone 18 integrated into the housing of the content capture device 20. In other examples, the microphone 18 is a device that is physically separate from the content capture device 20. The content capture device 20 may wirelessly interface with the microphone 18 (e.g., via Wi-Fi, Bluetooth, or the like). TM The content capture device 20 may be connected to a wireless or mobile connection or interfaced with the microphone 18 via a wired connection. Instead of capturing the audio data 19 (via the microphone 18) or in conjunction with capturing the audio data 19, the content capture device 20 may process the audio data 19 after the audio data 19 is input via some type of removable storage, wirelessly, and / or via a wired input process. As such, various combinations of content capture devices 20 and microphones 18 are possible in accordance with the present disclosure.

[0042] The content capture device 20 may also be configured to interface or otherwise communicate with a content editing device 22. In some cases, the content capture device 20 may include a content editing device 22 (which may, in some cases, represent software or a combination of software and hardware, including software executed by the content capture device 20 to configure the content capture device 20 to perform a particular form of content editing). In other examples, the content editing device 22 is a device physically separate from the content capture device 20. The content editing device 22 may represent a unit configured to edit or otherwise change the content 21 (including audio data 19) received from the content capture device 20. The content editing device 22 may output the edited content 23 and associated metadata 25 to the sound field representation generator 24.

[0043] The sound field representation generator 24 may include any type of hardware device capable of interfacing with the content editing device 22 (or the content capturing device 20). Figure 1A In the example shown in FIG. 1 , the sound field representation generator 24 can use the edited content 23 (including audio data 19) and metadata 25 provided by the content editing device 22 to generate one or more bitstreams 27. Figure 1AIn the example of , which focuses on the audio data 19 , the sound field representation generator 24 may generate one or more representations of the same sound field represented by the audio data 19 to obtain a bitstream 27 comprising the representation of the sound field and the audio metadata 25 .

[0044] For example, in order to generate different representations of the sound field using ambisonics coefficients (which is also an example of audio data 19), the sound field representation generator 24 can use a codec scheme for ambisonics representation of the sound field, which is called mixed-order ambisonics (MOA), as discussed in more detail in U.S. application serial number 15 / 672,058, entitled “MIXED-ORDER AMBISONICS (MOA) AUDIO DATA FOR COMPUTER-MEDIATED REALITY SYSTEMS,” filed on August 8, 2017 and published on January 3, 2019 as U.S. Patent Publication No. 20190007781.

[0045] To generate a particular MOA representation of a sound field, the sound field representation generator 24 may generate a partial subset of the complete set of ambisonic reverberation coefficients. For example, each MOA representation generated by the sound field representation generator 24 may provide accuracy for some areas of the sound field, but lower accuracy in other areas. In one example, the MOA representation of a sound field may include eight (8) uncompressed ambisonic reverberation coefficients, while a third-order ambisonic reverberation representation of the same sound field may include sixteen (16) uncompressed ambisonic reverberation coefficients. In this way, each MOA representation of the sound field generated as a partial subset of the ambisonic reverberation coefficients may be less storage intensive and bandwidth intensive than a corresponding third-order ambisonic reverberation representation of the same sound field generated from the ambisonic reverberation coefficients (if and when transmitted as part of the bitstream 27 over the transmission channel shown).

[0046] Although described with respect to MOA representation, the techniques of the present disclosure may also be performed for a first-order ambisonic (FOA) representation, in which all ambisonic coefficients associated with the first-order spherical basis functions and the zero-order spherical basis functions are used to represent the sound field. In other words, the sound field representation generator 302 may use all ambisonic coefficients of a given order N to represent the sound field, rather than using a partial non-zero subset of the ambisonic coefficients to represent the sound field, so that the total ambisonic coefficients are equal to (N+1) 2 .

[0047] In this regard, the ambisonic reverberation audio data (which is another way to refer to ambisonic reverberation coefficients in an MOA representation or a full-order representation, such as the first-order representation mentioned above) may include ambisonic reverberation coefficients associated with spherical basis functions having an order of one or less (which may be referred to as "first-order ambisonic reverberation audio data"), ambisonic reverberation coefficients associated with spherical basis functions having mixed orders and sub-orders (which may be referred to as the "MOA representation" mentioned above), or ambisonic reverberation coefficients associated with spherical basis functions having an order greater than one (which is referred to as the "full-order representation" above).

[0048] In some examples, the content capture device 20 or the content editing device 22 may be configured to wirelessly communicate with the sound field representation generator 24. In some examples, the content capture device 20 or the content editing device 22 may communicate with the sound field representation generator 24 via one or both of a wireless connection or a wired connection. Via the connection between the content capture device 20 or the content editing device 22 and the sound field representation generator 24, the content capture device 20 or the content editing device may provide content in various forms, which for discussion purposes are described herein as part of the audio data 19.

[0049] In some examples, the content capture device 20 may utilize various aspects of the sound field representation generator 24 (in terms of hardware or software capabilities of the sound field representation generator 24). For example, the sound field representation generator 24 may include dedicated hardware that is configured (or dedicated software that, when executed, causes one or more processors) to perform psychoacoustic audio coding (such as the Unified Speech and Audio Coder denoted “USAC” proposed by the Moving Picture Experts Group (MPEG), the MPEG-H 3D Audio Coding standard, the MPEG-I Immersive Audio standard, or a proprietary standard such as AptX TM (including various versions of AptX, such as Enhanced AptX - E-AptX, AptX Live, AptX Stereo and AptX High Definition - AptX-HD), Advanced Audio Coding (AAC), Audio Codec 3 (AC-3), Apple Lossless Audio Codec (ALAC), MPEG-4 Audio Lossless Streaming (ALS), Enhanced AC-3, Free Lossless Audio Codec (FLAC), Monkey's Audio, MPEG-1 Audio Layer II (MP2), MPEG-1 Audio Layer III (MP3), Opus and Windows Media Audio (WMA).

[0050] The content capture device 20 may not include psychoacoustic audio encoder dedicated hardware or specialized software, but may provide audio aspects of the content 21 in a non-psychoacoustic audio encoded form. The sound field representation generator 24 may assist in the capture of the content 21 by performing psychoacoustic audio encoding at least in part for the audio aspects of the content 21.

[0051] The sound field representation generator 24 may also assist in content capture and transmission by generating one or more bitstreams 27 based at least in part on audio content (e.g., MOA representation and / or first-order ambisonic representation) generated from the audio data 19 (where the audio data 19 includes scene-based audio data). The bitstream 27 may represent a compressed version of the audio data 19 and any other different types of content 21 (such as compressed versions of spherical video data, image data, or text data).

[0052] As an example, the sound field representation generator 24 may generate a bitstream 27 for transmission across a transmission channel, which may be a wired or wireless channel, a data storage device, etc. The bitstream 27 may represent an encoded version of the audio data 19, and may include a main bitstream and another side bitstream, which may be referred to as side channel information or metadata. In some cases, the bitstream 27 representing a compressed version of the audio data 19 (which may also represent scene-based audio data, object-based audio data, channel-based audio data, or a combination thereof) may conform to a bitstream generated according to the MPEG-H 3D audio coding standard and / or the MPEG-I immersive audio standard.

[0053] The content consumer device 14A may be operated by an individual and may represent a VR client device. Although described with respect to a VR client device, the content consumer device 14A may represent other types of devices, such as an augmented reality (AR) client device, a mixed reality (MR) client device (or other XR client device), a standard computer, a head mounted device, a headset, a mobile device (including so-called smartphones), or any other device capable of tracking the head movement and / or general translational movement of the individual operating the content consumer device 14A. Figure 1A As shown in the example of , the content consumer device 14A includes an audio playback system 16A, which may refer to any form of audio playback system capable of rendering audio data for playback as single-channel and / or multi-channel audio content.

[0054] Content consumer device 14A may include a user interface (UI) 36. User interface 36 may include one or more input devices and one or more output devices. Output devices may include, for example, one or more speakers, one or more display devices, one or more haptic devices, etc. configured to output information for user perception. Output devices may be integrated with content consumer device 14A or may be separate devices coupled to the content consumer device.

[0055] One or more input devices may include any suitable device with which a user can interact to provide input to content consumer device 14A. For example, one or more input devices may include a microphone, a mouse, a pointer, a game controller, a remote control, a touch screen, a linear slider potentiometer, a rocker switch, a button, a roller, a knob, etc. In examples where one or more user input devices include a touch screen, the touch screen may allow selection of one or more capture device representations based on a single touch input (e.g., touching, sliding, tapping, long pressing, and / or circling an area of ​​a graphical user interface). In some embodiments, the touch screen may allow multi-touch input. In these examples, the touch screen may allow selection of multiple areas of a graphical user interface based on multiple touch inputs.

[0056] Although in Figure 1A 2 is shown as being sent directly to content consumer device 14A, but source device 12A may output bitstream 27 to an intermediate device located between source device 12A and content consumer device 14A. The intermediate device may store bitstream 27 for later delivery to content consumer device 14A, which may request bitstream 27. The intermediate device may include a file server, a network server, a desktop computer, a laptop computer, a tablet computer, a mobile phone, a smart phone, or any other device capable of storing bitstream 27 for later retrieval by an audio decoder. The intermediate device may reside in a content delivery network capable of streaming bitstream 27 (and possibly in conjunction with sending a corresponding video data bitstream) to a subscriber (such as content consumer device 14A) requesting bitstream 27.

[0057] Alternatively, source device 12A may store bitstream 27 to a storage medium such as a compact disk, digital video disk, high-definition video disk, or other storage medium, most of which are capable of being read by a computer and may therefore be referred to as a computer-readable storage medium or a non-transitory computer-readable storage medium. In this context, a transmission channel may refer to a channel through which content stored to the medium (e.g., in the form of one or more bitstreams 27) is sent (and may include retail stores and other store-based delivery mechanisms). In any case, the technology of the present disclosure should not be limited in this regard. Figure 1A .

[0058] As described above, the content consumer device 14A includes an audio playback system 16A. The audio playback system 16A can represent any system capable of playing back mono and / or multi-channel audio data. The audio playback system 16A can include a plurality of different audio renderers 32. The audio renderers 32 can each provide different forms of rendering, where the different forms of rendering can include one or more of various ways of performing vector-base amplitude panning (VBAP), and / or one or more of various ways of performing sound field synthesis. As used herein, "A and / or B" means "A or B", or "both A and B".

[0059] The audio playback system 16A may also include an audio decoding device 34. The audio decoding device 34 may represent a device configured to decode the bitstream 27 to output audio data 19' (where the apostrophe may represent that the audio data 19' is different from the audio data 19 due to lossy compression of the audio data 19 (such as quantization). The audio decoding device 34 may be part of the same physical device as the audio renderer 32, or may be part of a device that is physically separate and configured to communicate with the audio renderer 32 via a wireless or wired connection. Again, the audio data 19' may include: scene-based audio data that in some examples may form a full first-order (or higher) order ambisonic representation or a subset of it that forms a MOA representation of the same sound field, its decomposition (such as the main audio signal, the ambient ambisonic coefficients, and the vector-based signals described in the MPEG-H 3D Audio codec standard), or other forms of scene-based audio data.

[0060] Other forms of scene-based audio data include audio data defined according to the HOA (Higher Order Ambisonics) Transport Format (HTF). More information about HTF can be found in the technical specification (TS) entitled "Higher Order Ambisonics (HOA) Transport Format" in ETSI TS 103 589 V1.1.1 dated June 2018 (2018-06) of the European Telecommunications Standards Institute (ETSI), and in U.S. Patent Application Publication No. 2019 / 0918028 entitled "PRIORITY INFORMATION FOR HIGHERORDER AMBISONIC AUDIO DATA" filed on December 20, 2018. In any case, the audio data 19' may be similar to a complete set or a partial subset of the audio data 19', but may differ due to lossy operations (e.g., quantization) and / or transmission via a transmission channel.

[0061] The audio data 19' may include channel-based audio data as an alternative to or in combination with scene-based audio data. The audio data 19' may include object-based audio data or channel-based audio data as an alternative to or in combination with scene-based audio data. Thus, the audio data 19' may include any combination of scene-based audio data, object-based audio data, and channel-based audio data.

[0062] The audio renderer 32 of the audio playback system 16A may render the audio data 19' to output a speaker feed 35 after the audio decoding device 34 has decoded the bitstream 27 to obtain the audio data 19'. The speaker feed 35 may drive one or more speakers (which are not shown in FIG. 1 for ease of illustration). Figure 1A ). Various audio representations including scene-based audio data (and possibly channel-based audio data and / or object-based audio data) of the sound field may be normalized in a variety of ways, including N3D, SN3D, FuMa, N2D, or SN2D.

[0063] To select an appropriate renderer, or in some cases, generate an appropriate renderer, the audio playback system 16A may obtain speaker information 37 indicating the number of speakers (e.g., loudspeakers or headphone speakers) and / or the spatial geometry of the speakers. In some cases, the audio playback system 16A may obtain the speaker information 37 using a reference microphone, and may drive the speakers (which may refer to the output of electrical signals to cause transducer vibrations) in a manner that dynamically determines the speaker information 37. In other cases, or in conjunction with the dynamic determination of the speaker information 37, the audio playback system 16A may prompt a user to interface with the audio playback system 16A and enter the speaker information 37.

[0064] The audio playback system 16A may select one of the audio renderers 32 based on the speaker information 37. In some cases, the audio playback system 16A may generate one of the audio renderers 32 based on the speaker information 37 when none of the audio renderers 32 is within a certain threshold similarity metric (in terms of speaker geometry) of the speaker geometry specified in the speaker information 37. In some cases, the audio playback system 16A may generate one of the audio renderers 32 based on the speaker information 37 without first attempting to select an existing one of the audio renderers 32.

[0065] When the speaker feeds 35 are output to headphones, the audio playback system 16A may utilize one of the renderers 32 that provides binaural rendering using a head-related transfer function (HRTF) or other functionality capable of rendering to left and right speaker feeds 35 for headphone speaker playback, such as a binaural room impulse response renderer. The term "speaker" or "transducer" may generally refer to any speaker, including a loudspeaker, headphone speaker, bone conduction speaker, earbud speaker, wireless headphone speaker, etc. One or more speakers may then playback the rendered speaker feeds 35 to reproduce the sound field.

[0066] Although described as rendering the speaker feeds 35 from the audio data 19', references to rendering of the speaker feeds may refer to other types of rendering, such as rendering that is incorporated directly into the decoding of the audio data from the bitstream 27. Examples of alternative rendering may be found in Annex G of the MPEG-H 3D Audio standard, where rendering occurs during the formation of the main signal prior to sound field composition and during the formation of the background signals. As such, references to rendering of the audio data 19' should be understood to refer to either a rendering of the actual audio data 19' or a decomposition or representation of the audio data 19', such as the above-mentioned main audio signal, ambisonic coefficients, and / or a vector-based signal - which may also be referred to as a V-vector or ambisonic space vector.

[0067] The audio playback system 16A can also adapt the audio renderer 32 based on the tracking information 41. That is, the audio playback system 16A can interface with a tracking device 40, which is configured to track the head movement and possible translational movement of the user of the VR device. The tracking device 40 can represent one or more sensors (e.g., cameras - including depth cameras, gyroscopes, magnetometers, accelerometers, light emitting diodes - LEDs, etc.), which are configured to track the head movement and possible translational movement of the user of the VR device. The audio playback system 16A can adapt the audio renderer 32 based on the tracking information 41 so that the speaker feed 35 reflects the changes and possible translational movement of the user's head to correctly reproduce the sound field in response to such movement.

[0068] Figure 1B 1 is a block diagram illustrating another example system 50 configured to perform various aspects of the techniques described in this disclosure. System 50 is similar to system 10 shown in FIG. 1 , except that Figure 1A The audio renderer 32 shown in FIG. 3 is replaced by a binaural renderer 42 capable of performing binaural rendering using one or more head-related transfer functions (HRTFs) or other functions capable of rendering to left and right speaker feeds 43 .

[0069] The audio playback system 16B may output left and right speaker feeds 43 to headphones 48, which may represent another example of a wearable device and may be coupled to an additional wearable device to facilitate reproduction of the sound field, such as a watch, the aforementioned VR headset, smart glasses, smart clothing, a smart ring, a smart bracelet, or any other type of smart jewelry (including a smart necklace), etc. The headphones 48 may be coupled to the additional wearable device wirelessly or via a wired connection.

[0070] Additionally, the headset 48 may be connected via a wired connection (such as a standard 3.5 mm audio jack, a universal system bus (USB) connection, an optical audio jack, or other form of wired connection) or wirelessly (such as via Bluetooth TM The headset 48 may be coupled to the audio playback system 16B by a wireless connection, a wireless network connection, etc. The headset 48 may recreate the sound field represented by the audio data 19' based on the left and right speaker feeds 43. The headset 48 may include a left earphone speaker and a right earphone speaker, which are powered (or in other words, driven) by the corresponding left and right speaker feeds 43.

[0071] Figure 1C is a block diagram illustrating another example system 60. The example system 60 is similar to Figure 1AExample system 10 of system 60, but source device 12B of system 60 does not include a content capture device. Source device 12B includes synthesis device 29. Content developers can use synthesis device 29 to generate synthesized audio sources. The synthesized audio source can have position information associated with it, which can identify the position of the audio source relative to the listener or other reference point in the sound field, so that the audio source can be rendered to one or more speaker channels for playback in an attempt to recreate the sound field. In some examples, synthesis device 29 can also synthesize visual or video data.

[0072] For example, a content developer could generate a synthetic audio stream for use in a video game. Figure 1C An example is shown with Figure 1A The example content consumer device 14A, but Figure 1C The source device 12B of the example may also be connected with Figure 1B In some examples, Figure 1C Source device 12B may also include a content capture device such that bitstream 27 may contain both captured audio stream(s) and synthesized audio stream(s).

[0073] As described above, content consumer device 14A or 14B (for simplicity, either may be referred to as content consumer device 14 hereinafter) may represent a VR device in which a human wearable display (which may also be referred to as a “head-mounted display”) is mounted in front of the eyes of a user operating the VR device. Figure 2 1 is a diagram illustrating an example of a VR headset 1100 worn by a user 1102. The VR headset 1100 is coupled to or otherwise includes headphones 1104, which can reproduce the sound field represented by the audio data 19' through the playback of speaker feeds 35. The speaker feeds 35 can represent analog or digital signals that can cause a membrane within a transducer of the headphones 1104 to vibrate at various frequencies, where such a process is generally referred to as driving the headphones 1104.

[0074] Video, audio, and other sensory data may play an important role in a VR experience. To participate in a VR experience, a user 1102 may wear a VR headset 1100 (which may also be referred to as a VR client device 1100) or other wearable electronic device. A VR client device (such as a VR headset 1100) may include a tracking device (e.g., tracking device 40) configured to track the head movements of the user 1102 and adapt the video data displayed via the VR headset 1100 to account for the head movements, providing an immersive experience in which the user 1102 can experience a display world shown in visual three dimensions with the video data. The display world may refer to a virtual world (in which all of the world is simulated), an augmented world (in which portions of the world are augmented by virtual objects), or a physical world (in which real-world images are virtually navigated).

[0075] While VR (and other forms of AR and / or MR) can allow a user 1102 to visually reside in a virtual world, a VR headset 1100 may often lack the ability to audibly place the user in the displayed world. In other words, a VR system (which may include a computer responsible for rendering video data and audio data—not shown here for ease of illustration) may not be able to audibly place the user in the displayed world. Figure 2 As shown in the example of , and the VR headset 1100) may not be able to audibly (and in some cases, realistically in a manner that reflects the display scene presented to the user via the VR headset 1100) support full three-dimensional immersion.

[0076] Although described in the present disclosure with respect to VR devices, various aspects of these techniques may be performed in the context of other devices such as mobile devices. In this case, a mobile device (such as a so-called smart phone) may present a display world via a screen that may be mounted on the head of the user 1102 or viewed in the manner in which the mobile device is normally used. In this manner, any information on the screen may be part of the mobile device. The mobile device may be able to provide tracking information 41 and thus allow both a VR experience (when head mounted) and a normal experience of viewing the display world, wherein the normal experience may still allow the user to view the display world providing a lightweight VR experience (e.g., lifting the device and rotating or translating the device to view different parts of the display world).

[0077] In any case, returning to the context of VR devices, the audio aspects of VR have been divided into three separate immersion categories. The first category provides the lowest level of immersion and is known as three degrees of freedom (3DOF). 3DOF refers to audio rendering that takes into account head movement in three degrees of freedom (yaw, pitch, and roll), allowing the user to look around freely in any direction. However, 3DOF cannot take into account translational head movements where the head is not centered on the optical and acoustic center of the sound field.

[0078] The second category, referred to as 3DOF plus (3DOF+), provides three degrees of freedom (yaw, pitch, and roll) in addition to limited spatial translational movement due to head movement away from the optical and acoustic centers within the sound field. 3DOF+ can provide support for perceptual effects such as motion parallax, which can enhance immersion.

[0079] The third category, referred to as six degrees of freedom (6DOF), renders audio data in a way that takes into account three degrees of freedom in terms of head movement (yaw, pitch, and roll) and also takes into account the translation of the user in space (x, y, and z translation). Spatial translation can be introduced by sensors that track the user's position in the physical world or through input controllers.

[0080] 3DOF rendering is the current technology used for audio in VR. As such, the audio aspect of VR is not as immersive as the video aspect, and therefore potentially reduces the overall immersion of the user experience. However, VR is rapidly transforming and may quickly evolve to support both 3DOF+ and 6DOF, which may provide opportunities for additional use cases.

[0081] For example, an interactive gaming application may utilize 6DOF to facilitate fully immersive gaming in which the user moves within the VR world themselves and can interact with virtual objects by walking up to them. Additionally, an interactive live streaming application may utilize 6DOF to allow a VR client device to experience a live stream of a concert or sporting event as if they were present at the concert, allowing the user to move from one location within the concert or sporting event to another. In some examples, as the user moves from one location to another, the stream selection unit 44 may select at least one audio stream that is different from a previously selected stream.

[0082] There are a number of difficulties associated with these use cases. In the case of fully immersive gaming, latency may need to be kept low to enable gameplay that does not cause nausea or dizziness. Additionally, from an audio perspective, latency resulting in audio playback losing sync with the video data may reduce immersion. Furthermore, for certain types of gaming applications, spatial accuracy may be important to allow accurate responses, including with respect to how the user perceives sound, as this allows the user to anticipate actions that are not currently in view.

[0083] In the context of a live streaming application, a large number of source devices 12A or 12B (both will be referred to as source devices 12 hereinafter for simplicity) may stream content 21, where source devices 12 may have widely varying capabilities. For example, one source device may be a smartphone with a digital fixed-lens camera and one or more microphones, while another source device may be a production-level television rig capable of obtaining higher resolution and quality video than a smartphone. However, in the context of a live streaming application, all source devices may provide streams of varying quality, and the VR device may attempt to select an appropriate stream from among them to provide the intended experience.

[0084] Furthermore, similar to gaming applications, latency in audio data that causes it to lose synchronization with video data may result in a lower sense of immersion. Additionally, spatial accuracy may also be important so that the user can better understand the context or location of different audio sources. Additionally, privacy may become an issue when a user is live streaming using a camera and microphone, as the user may not want the live stream to be fully available to the public.

[0085] In the context of streaming applications (live or recorded), there may be a large number of audio streams associated with different levels of quality and / or content. The audio streams may represent any type of audio data, including scene-based audio data (e.g., ambisonic audio data, including FOA audio data, MOA audio data, and / or HOA audio data), channel-based audio data, and object-based audio data. Selecting only one of the potentially large number of audio streams to recreate the sound field may not provide an experience that ensures a sufficient level of immersion. However, due to the different spatial positioning between the multiple audio streams, selecting multiple audio streams may cause interference, thereby potentially reducing immersion.

[0086] According to certain techniques described in this disclosure, the audio decoding device 34 may adaptively select between audio streams available via the bitstream 27 (which are represented by the bitstream 27 and thus the bitstream 27 may also be referred to as “audio stream 27”). The audio decoding device 34 may select between the audio streams available via the bitstream 27 based on audio location information (ALI) (e.g., Figure 1A-Figure 1C The audio decoding device 34 may select a subset of the audio streams 27 based on the ALI 45A, wherein the subset of the audio streams 27 excludes at least one of the audio streams 27. The audio decoding device 34 may output the subset of the audio streams 27 as audio data 19' (which may also be referred to as "audio data 19'").

[0087] Additionally, the audio decoding device 34 may obtain tracking information 41, which the content consumer device 14 may convert into device location information (DLI) (e.g., Figure 1A-Figure 1C 45B in the ALI 45A). The DLI 45B may represent a virtual position or an actual position of the content consumer device 14 in the display world or the auditory space. In some examples, the DLI 45B may be associated with coordinates of the display world or the auditory space. The content consumer device 14 may provide the DLI 45B to the audio decoding device 34. The audio decoding device 34 may then select audio data 19' from the audio stream 27 based on the ALI 45A and the DLI 45B. The audio playback system 16A may then reproduce the corresponding sound field based on the audio data 19'.

[0088] In this regard, the audio decoding device 34 can adaptively select a subset of the audio streams 27 to obtain audio data 19' that can result in a more immersive experience (compared to selecting a single audio stream or all of the audio data 19'). In this way, various aspects of the techniques described in this disclosure can improve the operation of the audio decoding device 34 (as well as the audio playback system 16A or 16B and the content consumer device 14) itself by potentially enabling the audio decoding device 34 to better spatialize sound sources within the sound field, thereby improving immersion.

[0089] In operation, the audio decoding device 34 may interface with one or more source devices 12 to determine the ALI 45A for each of the audio streams 27. Figure 1AAs shown in the example of , the audio decoding device 34 may include a stream selection unit 44, which may represent a unit configured to perform various aspects of the audio stream selection techniques described in this disclosure.

[0090] Stream selection unit 44 may generate a constellation map (CM) 47 based on ALI 45A. CM 47 may define ALI 45A for each of audio streams 27. Stream selection unit 44 may also perform energy analysis for each of audio streams 27 to determine an energy map for each of audio streams 27, and store the energy map in CM 47 together with ALI 45A. The energy map may collectively define the energy of a common sound field represented by audio streams 27.

[0091] The stream selection unit 44 may then determine the distance between the position represented by the DLI 45B and the capture position or synthesis position represented by the ALI 45A associated with at least one and possibly each of the audio streams 27. The stream selection unit 44 may then select the audio data 19′ from the audio stream 27 based on at least one and possibly each of the distance(s), as described below with respect to Figure 3A-3D As discussed in more detail.

[0092] In addition, in some examples, the stream selection unit 44 may also select audio data 19' from the audio stream 27 based on the energy map, ALI 45A, and DLI 45B stored in the CM 47 (where ALI 45A and DLI 45B are jointly presented in the form of the distance mentioned above, which may also be referred to as a "relative distance"). For example, the stream selection unit 44 may analyze the energy map presented in the CM 47 to determine the audio source location (ASL) 49 of the audio source in the common sound field that emits the sound captured by the microphone (such as the microphone 18) and represented by the audio stream 27. The stream selection unit 44 may then determine the audio data 19' from the audio stream 27 based on the ALI 45A, the DLI 45B, and the ASL 49. More information on how the stream selection unit 44 may select a stream is provided below with respect to Figure 3A-3D to discuss.

[0093] According to certain techniques described in the present disclosure, content consumer device 14 may be configured to output via user interface 36 to display a graphical user interface (GUI). The GUI may include a visual representation of CM 47. User interface 36 may enable a user to select and control various aspects or parameters of the sound field, and stream selection unit 44 may be configured to select at least one of audio streams 27 based on user input. For example, user interface 36 may display a GUI that includes audio element representations associated with coordinates of a display world or auditory space. The audio element representation may be an indication of content capture device 20, microphone 18, a synthesized sound field, or a combination thereof, and may be based on ALI 45A. For example, the audio element representation may include an icon, an image, an alphanumeric representation, or a combination thereof representing an audio element.

[0094] The user interface 36 may also be configured to obtain or detect an indication of a user input representing the selection of one or more of the audio element representations. In some examples, the user input may indicate a direct selection of one or more audio element representations. In other examples, the user input may indicate an indirect selection of one or more audio element representations, for example, by a user input defining an area on a GUI that surrounds one or more of the audio element representations. The user may define an arbitrary area (e.g., the area may have any shape and / or size) using user input, or may manipulate a predetermined area (e.g., by adjusting the size of an area with a predetermined and fixed shape). Any audio element representation surrounded by an area may be interpreted as being selected and any audio element representation outside the area may be interpreted as not being selected.

[0095] The stream selection unit 44 may be configured to select at least one of the audio streams 27 based on an indication representing a user input. In some examples, the stream selection unit 44 may select at least two audio streams and may be configured to combine the at least two audio streams to form a third audio stream. The stream selection unit 44 may combine the at least two audio streams using mixing, interpolation, etc., and the relative contribution of each of the at least two audio streams may be based on, for example, the distance between the position represented by the ALI 45A, DLI 45B, DLI 45B associated with the respective audio stream 27 and the capture position or synthesis position represented by the ALI 45A associated with at least one and possibly each of the audio streams 27, and / or the metadata 25.

[0096] The audio decoding device 34 may be configured to then output the audio data 19' based on the selection. The audio decoding device 34 may output the audio data 19' to an integrated audio renderer 32, an audio renderer 32 that is part of a physically separate device, or a storage device. In this way, various aspects of the techniques described in the present disclosure may improve the operation of the audio decoding device 34 (as well as the audio playback system 16A and the content consumer device 14) itself by potentially enabling a user to control aspects of the sound field by selecting content devices and associated audio streams that contribute to the sound field. More information on how the stream selection unit 44 may select a stream based on user input is provided below with respect to Figure 4A-4E to discuss.

[0097] Figure 3A-3D A more detailed map shows Figure 1A FIG. 4 is a diagram of an example operation of the stream selection unit 44 shown in the example of FIG. Figure 3A As shown in the example of , the stream selection unit 44 may determine that the DLI 45B indicates that the content consumer device 14 (shown as the VR headset 1100) is at the virtual location 300A. The stream selection unit 44 may then determine the ALI 45A for one or more of the audio elements 302A-302J (collectively referred to as the audio elements 302), which may not only represent microphones, such as Figure 1A The microphone 18 shown in FIG. 1 is also representative of other types of capture devices, including other XR devices, mobile phones - including so-called smartphones - etc., or synthetic sound fields).

[0098] The stream selection unit 44 may then obtain an energy map in the manner described above and analyze the energy map to determine the audio source location 304, which may represent Figure 1A An example of ASL 49 is shown in the example of FIG. The energy map can denote the audio source location 304 because the energy at the audio source location 304 may be higher than the surrounding area. Given that each of the energy maps can denote this higher energy, the stream selection unit 44 can triangulate the audio source location 304 based on the higher energy in the energy map.

[0099] Next, the stream selection unit 44 may determine the audio source distance 306A as the distance between the audio source position 304 and the virtual position 300A of the VR headset 1100. The stream selection unit 44 may compare the audio source distance 306A with an audio source distance threshold. In some examples, the stream selection unit 44 may derive the audio source distance threshold based on the energy of the audio source 308. That is, when the audio source 308 has a higher energy (or in other words, when the sound of the audio source 308 is louder), the stream selection unit 44 may increase the audio source distance threshold. When the audio source 308 has a lower energy (or in other words, when the audio source 308 is quieter), the stream selection unit 44 may reduce the audio source distance threshold. In other examples, the stream selection unit 44 may obtain a statically defined audio source distance threshold, which may be statically defined or specified by the user 1102.

[0100] In any case, when the audio source distance 306A is greater than an audio source distance threshold (assumed for purposes of illustration in this example), the stream selection unit 44 may select a single audio stream (e.g., Figure 3A The stream selection unit 44 may output a corresponding one of the audio streams 27, which the audio decoding device 34 may decode and output as audio data 19'.

[0101] Assuming that the user moves from virtual position 300A to virtual position 300B, stream selection unit 44 may determine audio source distance 306B as the distance between audio source position 304 and virtual position 300B. Stream selection unit 44 may again compare audio source distance 306B with an audio source distance threshold. When audio source distance 306 is less than or equal to the audio source distance threshold (assumed for purposes of illustration in this example), stream selection unit 44 may select multiple audio streams in audio stream 27 of audio elements 302A-302J ("audio elements 302"). Stream selection unit 44 may output the corresponding audio stream 27, which audio decoding device 34 may decode and output as audio data 19'.

[0102] Stream selection unit 44 may also determine a proximity distance between virtual position 300A and one or more (and possibly each) of the capture locations represented by ALI 45A to obtain one or more proximity distances. Stream selection unit 44 may then compare the one or more proximity distances to a threshold proximity distance. When one or more proximity distances are greater than the threshold proximity distance, stream selection unit 44 may select a fewer number of audio streams 27 to obtain audio data 19' than when the proximity distance is less than or equal to the threshold proximity distance. However, when one or more of the proximity distances are less than or equal to the threshold proximity distance, stream selection unit 44 may select a greater number of audio streams 27 to obtain audio data 19' than when the proximity distance is less than or equal to the threshold proximity distance.

[0103] In other words, the stream selection unit 44 may attempt to select those of the audio streams 27 that most closely align the audio data 19' to and around the virtual location 300B. The proximity distance threshold may define a threshold that the user 1100 may set or the stream selection unit 44 may dynamically determine, again based on the quality of the audio elements 302F-302J, the gain or loudness of the audio source 308, the tracking information 41 (e.g., to determine whether the user is facing the audio source 308), or any other factor.

[0104] Next reference Figure 3B For example, stream selection unit 44 may determine that the audio stream of audio element 302A is corrupted, noisy, or unavailable. Given that audio source distance 306A is greater than an audio source distance threshold, stream selection unit 44 may remove the audio stream from CM 47 and iterate through audio streams 27 to select a single audio stream (e.g., Figure 3B 302B in the example of FIG. 304 ).

[0105] Next reference Figure 3C , stream selection unit 44 may obtain a new audio stream (the audio stream of audio element 302K) and corresponding new audio metadata including ALI 45A. Stream selection unit 44 may add the new audio stream to CM 47 representing audio stream 27. Given that audio source distance 306A is greater than an audio source distance threshold, stream selection unit 44 may then iterate through audio streams 27 to select a single audio stream (e.g., Figure 3C 302B in the example of FIG. 304 ).

[0106] exist Figure 3DIn the example of , audio element 302 is replaced by specific example devices 320A-320J ("devices 320"), where device 320A represents a dedicated microphone 320A, and devices 320B, 320C, 320D, 320G, 320H, and 320J represent smart phones. Devices 320E, 320F, and 320I may represent VR devices. Each of devices 320 may include a microphone that captures an audio stream 27 selected according to various aspects of the stream selection techniques described in the present disclosure.

[0107] Figure 4A-4E A more detailed map shows Figure 1A 4. In some examples, user interface 36 may include a graphical user interface (GUI) 402A for display at display device 404. Although user interface 36 is primarily discussed herein as a GUI, any type of user interface may be used in accordance with the techniques of this disclosure. For example, user interface 36 may include a touch-based user interface, a voice command-based user interface, a gesture-based user interface, a mouse, a keyboard, a joystick, or any other type of user interface. Display device 404 may be part of user interface 36 of content consumer device 14 (e.g., display device 404 may be Figure 2 The display device 404 may be part of the VR headset 1100 or mobile computing device) or may be a separate device (e.g., the display device 404 may be a monitor connected to the content consumer device 14 via a wired or wireless connection).

[0108] In some examples, display device 404 may include a touch sensor that may detect touch input (e.g., single-touch input or multi-touch input). In some configurations, the touch sensor may be integrated with display device 404. In other configurations, the touch sensor may be implemented separately in content consumer device 14 or may be coupled to content consumer device 14 and / or display device 404.

[0109] GUI 402A may include a two-dimensional or three-dimensional representation of CM 47, which may include multiple audio element representations 406A-406H (collectively, "audio element representations 406"). In some examples, one or more of audio element representations 406 may represent a synthesized audio stream rather than a captured audio stream. Audio element representation 406 may be an indication of content capture device 20, microphone 18, a synthesized sound field, or a combination thereof, and may be based on ALI 45A. For example, audio element representation 406 may include an icon, an image, an alphanumeric representation, or a combination thereof representing an audio element. In some examples, GUI 402A includes a legend 408 that associates an icon or image with a label to assist a user in understanding the meaning of the icon or image of audio element representation 406. Figure 4A As shown, the audio elements associated with the audio element representation 406 may include a microphone, a mobile device with a microphone, an XR headset with a microphone, a connected video camera with a microphone, a synthetic sound field, etc.

[0110] The audio element representation 406 may be associated with coordinates (e.g., (x, y, z) coordinate tuples) of the display world or auditory space via the ALI 45A. In some examples, such as Figure 4A As shown, audio element representations 406 may be arranged in GUI 402A in a spatial relationship that represents the relative positioning of the respective audio elements within the display world or auditory space. In some examples, GUI 402A may include a representation of audio element representations 406 displayed within the display world or auditory space based on ALI 45A.

[0111] In some examples, user interface 36 can align all or part of GUI 402A with a reference plane. In some embodiments, the reference plane can be horizontal (e.g., parallel to the ground or floor). For example, user interface 36 can align the portion of the coordinate system that displays in GUI 402A and is associated with the display world or auditory space. In some embodiments, user interface 36 can align all or part of GUI 402A in real time.

[0112] In some configurations, the content consumer device 14 (e.g., the user interface 36) may include at least one image sensor. For example, several image sensors may be included within the content consumer device 14. The at least one image sensor may collect data (e.g., image data) related to the content consumer device 14 and / or the environment in which the content consumer device is located. For example, a camera (e.g., an image sensor) may generate an image. In some implementations, the at least one image sensor may provide the image data to the display device 404.

[0113] In some examples, the GUI 402A may include an energy map and / or a representation of the location of the audio source 415. In some examples, the energy map may be a component determined based on energy maps associated with multiple and possibly each of the audio streams 27. The energy map may indicate the audio source location 414 because the energy at the audio source location 414 may be higher than the surrounding area. Given that each of the energy maps may indicate this higher energy, the stream selection unit 44 may triangulate the audio source location 414 based on the higher energy in the energy map.

[0114] The user interface 36 may include one or more elements configured to perform the functions described herein. For example, the GUI 402A may include an indicator of the selected audio element representation 406 and / or may display an icon for editing the selected audio element representation 406.

[0115] GUI 402A may include an audio element representation selection feature that may allow selection of at least one audio element representation in audio element representation 406. In some embodiments, the audio element representation selection feature may be operated based on touch input. For example, the audio element representation selection feature may allow selection of an audio element representation based on a single touch input (e.g., touching, sliding, and / or circling an area of ​​GUI 402A corresponding to an audio element representation). Examples of touch input include one or more taps, slides, patterns (e.g., symbols, shapes, etc.), pinches, stretches, multi-touch rotations, etc. In some configurations, when one or more taps, slides, patterns, gestures, etc. intersect with the displayed audio element representation, one or more processors of content consumer device 14 may select the displayed audio element representation. In some embodiments, the audio element representation selection feature may allow selection of multiple sectors simultaneously. For example, the audio element representation selection feature may allow selection of multiple audio element representations 406 based on multiple touch inputs. As another example, the audio element representation selection feature may allow selection of multiple audio element representations 406 based on listener 410 via user input definition area 412.

[0116] Region 412 may correspond to a set of coordinates in the display world defined, for example, by listener 410 via user input. In some examples, region 412 may have a predefined shape (e.g., a circle) and listener 410 may manipulate the size of region 412 by touch input (e.g., touch and drag input, pinch, or touch input corresponding to an icon or slider position for adjusting the size of the region). In other examples, region 412 may be completely defined by listener 410, including the shape and size of region 412. When a pattern (e.g., a circular area, a rectangular area, or an area within a pattern), etc., completely or partially surrounds or includes a displayed audio element representation, user interface 36 may select the displayed audio element representation. User interface 36 may be configured to interpret the audio element representation within region 412 (e.g., audio element representations 406F-406H) as being selected and interpret the audio element representation outside region 412 (e.g., audio element representations 406A-406E) as not being selected.

[0117] In some configurations, the audio decoding device 34 can edit one or more audio streams 27 associated with the selected audio element representation based on one or more user inputs. For example, the user interface 36 can present one or more options (e.g., one or more buttons, one or more sliders, a drop-down menu, etc.) that provide options for editing the audio stream associated with the selected audio element representation (selected audio signal indicator). The listener 410 can edit the selected audio stream by, for example, assigning more importance to the audio element representation or region 412 (compared to another audio element representation or region 412). For example, the content consumer device 14 can obtain an importance indication from the listener 410. The audio decoding device 34 (e.g., the stream selection unit 44) can assign a higher weight (e.g., gain) to the audio stream indicated by the listener 410 as being more important and can assign a lower weight (e.g., gain) to the audio stream indicated by the listener 410 as being less important or not associated with importance. Additionally or alternatively, the user interface 36 may present one or more options (e.g., one or more buttons, drop-down menus, etc.) that provide options for editing the sectors (e.g., selecting or changing a color, pattern, and / or image for the sectors, setting whether audio signals in the sectors should be filtered (e.g., blocked or passed), enlarging or reducing the sectors, adjusting the size of the sectors (e.g., by expanding or contracting the sectors), etc.). For example, a pinch touch input may correspond to reducing or reducing the size of the sectors, while a stretch may correspond to enlarging or expanding the size of the sectors.

[0118] The stream selection unit 44 may then select at least one of the audio streams 27 based on the selected audio element representation 406. For example, the stream selection unit 44 may select the audio stream associated with the selected audio element representation 406 and may omit or not select the audio stream associated with the unselected audio element representation 406. In some examples, the stream selection unit 44 may combine two or more of the selected audio streams to generate the audio data 19'. For example, the stream selection unit 44 may combine two or more of the selected audio streams using mixing, interpolation, etc., and the stream selection unit 44 may base the relative contribution of each of the at least two audio streams on, for example, the distance between the position represented by the ALI 45A, DLI 45B, DLI 45B associated with the respective audio stream 27 and the capture position or synthesis position represented by the ALI 45A associated with at least one and possibly each of the audio streams 27, and / or the metadata 25.

[0119] In some examples, listener 410 may use user interface 36 to indicate a listening location that is different from the listening location indicated by DLI 45B. Figure 4B A GUI 402B is illustrated in which a listener 410 indicates a desired listening position 413 that is different from the position 416 of the listener 410 within the display world or auditory space as indicated by the DLI 45B. The position of the listener 410 may continue to be tracked in 3DoF, 3DoF+, or 6DoF, but their audio angle may be disassociated from their tracked position. The listener 410 may indicate the desired listening position 413 using any of the user inputs described herein, including single-touch inputs, multi-touch inputs, or other non-touch user inputs. Alternatively, the desired listening position 413 may be a single audio element representation in the audio element representations 406, and the one or more processors may lock the listening position to the position of the selected audio element, moving the listening position as the selected audio element moves. The stream selection unit 44 may then select a desired listening position based on the coordinates associated with the desired listening position 413 and the coordinates described herein, for example with reference to FIG. Figure 1A-Figure 1C and Figure 3A-3D A technique is described for selecting an audio stream 27.

[0120] In some embodiments, the user interface 36 is configured to, in response to obtaining an indication of a desired listening location, possibly suggest an alternative listening location. Figure 4CGUI 402C is illustrated in which a listener 410 indicates a desired listening position 413. One or more processors of the content consumer device 14 obtain an indication of the desired listening position 413, for example from a touch sensor, and determine that the desired listening position 413 is in an undesirable position. For example, the desired listening position 413 may be close to an audio element representation 406E that is muted or whose audio stream is otherwise unavailable. In response, the user interface 36 may be configured to output a user interface element that suggests an alternative listening position 418. The audio decoding device 34 (e.g., the stream selection unit 44) may determine the alternative listening position 418 based on the available audio streams 27, metadata 25 associated with the audio streams, etc. For example, the metadata may indicate a brand (make), model, or other information for the audio element, and the audio decoding device 34 may select the alternative listening position 418 taking into account the quality metric associated with the audio element. In this way, in response to determining that the desired listening position 413 is not preferred, the one or more processors of the content consumer device 14 may "snap" the desired listening position 413 to the preferred listening position. The listener 410 may indicate acceptance of the alternate listening position 418 (eg, through an affirmative user input or by not entering a subsequent desired listening position 413 ) or may enter a subsequent desired listening position 413 .

[0121] In some embodiments, the user interface 36 is configured to allow the user to control or modify the relative angle of the listening experience. Figure 4D GUI 402D is illustrated in which a listener 410 indicates a desired listening angle. GUI 402D illustrates the position 416 of the listener 410 within the display world or auditory space as indicated by DLI 45B. GUI 402D also illustrates a first listening angle 420. The first listening angle 420 is relatively focused or narrow, as indicated by the relatively small size of the circle indicating the first listening angle 420. Since the first listening angle 420 is relatively small, the stream selection unit 44 may select only the audio streams associated with the audio element representations 406 within the first listening angle 420. This includes Figure 4D 406D and 406E in the example of . However, as indicated by the slash through audio element representation 406E, the audio element associated with audio element representation 406E is muted or otherwise unavailable. Therefore, stream selection unit 44 may use only the audio stream associated with audio element representation 406D.

[0122] The listener 410 can modify the relative angle of the listening experience to be wider or farther. For example, the listener 410 can make a gesture 426 or otherwise manipulate a graphical user interface element representing the listening angle to move the listening angle from a first listening angle 420 to a second listening angle 422. Alternatively, the user can interact with another graphical user interface element 424 to move a slider 425 along a continuum between near and far to define the listening angle. As another example, the user can interact with a mechanical input device (such as a linear potentiometer, a knob, a dial, etc.) to control the relative listening angle.

[0123] In response to detecting a user input that controls or modifies the relative listening angle, the stream selection unit 44 may select at least one audio stream from the audio streams 27. For example, the stream selection unit 44 may select an audio stream associated with the audio element representation 406 that is surrounded or otherwise indicated by the second listening angle 422. Figure 4D In the example of , the audio element representations 406 surrounded or otherwise indicated by the second listening angle 422 include audio element representations 406A-406E, although the stream selection unit 44 may not select the audio stream associated with the audio element representation 406E (as described above). In other examples, instead of selecting an audio stream based on the audio element representations 406 surrounded or otherwise indicated by the second listening angle 422, the stream selection unit 44 may select an audio stream based on the second listening angle 422 and, for example, the audio element representations 406A-406E described herein. Figure 1A-Figure 1C and Figure 3A-3D The described techniques are used to select the audio stream 27. For example, the second listening angle may define a threshold proximity distance, where a "far" angle corresponds to a larger threshold proximity distance and a "near" angle corresponds to a smaller threshold proximity distance.

[0124] In some embodiments, the user interface 36 is configured to allow a user to control or modify the presentation of information (such as metadata) associated with an audio element. Figure 4E GUI 402E is illustrated, which includes a display of metadata 428 associated with one or more (e.g., each) audio element representations 406. The metadata 428 may include, for example, indications of whether the associated audio element is public or private, whether the associated audio element includes a calibrated or uncalibrated microphone, whether the associated audio element is static or dynamic, etc. The user interface 36 may enable the listener 410 to toggle the display of the metadata 428, for example, by presenting a UI element configured to control the display of the metadata 428. The UI element may be a graphical UI element, such as an icon, a radio button, a toggle switch, or a menu selection; or may include a dedicated or configurable physical input device, such as a toggle switch, a button, etc.

[0125] As mentioned above about Figure 4C As described, a user (e.g., listener 410) may be able to fit to a preferred listening position. In the context of 6DoF audio, fit or fit mode may be defined as a transformation of spatial coordinates for audio rendering. In some examples, this transformation of spatial coordinates may be real-time or substantially real-time (with a delay of less than 2 seconds). In this way, fit may disrupt the relationship between the listener coordinates (e.g., the virtual position of the device or listener in the display world) used to render the visual scene (or the audio scene in the absence of visual content) and the coordinates used for audio rendering. There may be two fit modes - hard fit and soft fit.

[0126] Figure 5A is a conceptual diagram illustrating an example of hard bonding. Figure 5A One or more elements of may be part of the user interface 36. For example, the location 416 of the listener 410, the snap location S 435, the audio element 434, or the audio element 436, or any combination thereof may be represented in the user interface 36. In hard snapping, the threshold for weighting (w) the audio stream may be the distance of the location 416 of the listener 410 (also referred to as the virtual location of the device or user) to the nearest capture location A or B. The audio element 434 at location A and the audio element 436 at location B may represent audio signals (such as from an audio capture device or a synthesized audio stream), or predefined listening locations.

[0127] The first audio source distance threshold (such as the snap position S 435) can be set anywhere between positions A and B. In some examples, the listener 410 can set or move the snap position S 435, for example, through a user interface (such as the user interface disclosed herein). In some examples, the snap position S 435 can be preset, provided by the source device 12 in the bitstream 27, or generated by the content consumer device 14.

[0128] When the audio elements 434 and 436 at positions A and B are respectively the same type of audio elements, the midpoint of A and B may be set as a default position for positioning S 435. In the case where the midpoint is the default position for positioning P 435 (eg, Figure 5A435 ), when the listener 410's positioning 416 is closer to position A, the audio source w is at position A (e.g., audio element 434) (w=A), and when the listener 410's positioning 416 is closer to position B, the audio source w is at position B (e.g., audio element 436). In some examples, the listener 410 can over-ride the selection of positioning S 435 to bias the fit toward the audio element or collection of audio elements. For example, the listener 410 can move positioning S 435 closer to position A. When the listener 410 moves positioning S 435 closer to position A, the listener 410 can hear the audio signal represented by the audio element 436 when the listener 410 is located at the midpoint between positions A and B. When the position 416 of the listener 410 changes to be closer to the position A than the position S 435, the audio provided to the listener 410 may be snapped to the position A, so that the audio signal presented to the listener 410 is the audio signal represented by the audio element 434. When the position 416 of the listener 410 moves to be closer to the position B than the position S 435, the audio provided to the listener 410 may be snapped to the position B and the audio provided to the listener 410 may be the audio signal represented by the audio element 436.

[0129] For example, a device such as VR headset 1100 may determine virtual position information representing a virtual position of the device in a display world (e.g., position 416 of listener 410). VR headset 1100 may compare the virtual position information with a first audio source distance threshold (e.g., position S 435) and select at least one audio stream (e.g., audio element 434 or 436) based on the comparison, which may be provided to a user (e.g., listener 410).

[0130] Visual or auditory feedback may be provided to alert the listener 410 that they have been snapped into a listening position (e.g., position A or position B). For example, the VR headset 1100 may provide a visual warning, such as a fade to black, a flash, and / or a visual marker, at the new snapped position (e.g., position A or position B). In the case where the snapped position may be outside the field of view of the listener 410 (e.g., behind the listener), the visual warning may indicate that the snapped position is outside the field of view, such as a fade to black, a flash, or some other technique that may be associated with the snapped position being outside the field of view of the listener 410.

[0131] In some examples, the listener 410 may hear a fade in / fade out during the fit, or some other audio alert indicating that the fit has occurred. For example, the earphones 1104 may provide the listener 410 with a fade in / fade out, a tone, a jingle, a fit sound, a word (such as "fit"), or some other indication that the fit has occurred.

[0132] Figure 5B is a conceptual diagram illustrating an example of soft bonding. Figure 5B One or more elements of may be part of user interface 36. For example, position 416 of listener 410, fit position X1 429, fit position X2 442, audio element 438, or audio element 440, or any combination thereof may be represented in user interface 36. The second fit mode is a soft fit. Figure 5B As shown, when the fit is softened by the function F(x), soft fit can occur when the listener's position 416 is between a first audio source distance threshold (e.g., fit position X1) and a second audio source distance threshold (e.g., fit position X2) (soft fit region). Figure 5B It is shown as a linear function in , but F(x) can be any function.

[0133] In this way, a preference for audio rendering of an audio element 438 at position A or an audio element 440 at position B can be given to a device such as headphones 1104. In some examples, the region can be an asymmetric region and include hysteresis behavior. For example, a device such as a VR headset 1100 can determine whether the positioning 416 of the listener 410 remains stable relative to a first audio source distance threshold (e.g., fit positioning X1 439) and a second audio source distance threshold (e.g., fit positioning X2 441) within a predetermined time period. Based on the device determining that the positioning 416 of the listener 410 has been stable within a predetermined time period, a fit or transition of the audio presented to the listener 410 may occur. For example, the device can determine that the user's virtual position has moved from between fit positioning X1 439 and fit positioning X2 441 to between positioning X2 441 and position B. Once the predetermined time has passed, the audio presented to the listener 410 can be fit to the audio element 440 at position B.

[0134] In other respects, the soft snap works similarly to the hard snap. When the location 416 of the listener 410 is between the snap location X1 439 and the location A, the listener 410 can snap to the location A and hear the audio represented by the audio component 438 at the location A (when A ≤ x ≤ X1, w = A). When the location 416 of the listener 410 is between the snap location X1 439 and the snap location X2 441, the listener 410 can hear the audio represented by both the audio component 438 and the audio component 440 according to the function F(x) (when X1 < x < X2, w = F(x)). For example, when the location 416 of the listener 410 is between the snap location X1 439 and the snap location X1 441 (in the soft snap region), the device can adaptively mix the audio represented by both the audio component 438 and the audio component 440 according to F(x), or interpolate a third audio stream based on the audio represented by the audio component 438 and the audio component 440 and F(x). When the location 416 of the listener 410 is between the location B and the snap location X2 441, the listener 410 can snap to the location B and can hear the audio represented by the audio component 440 at the location B (when X2 < x ≤ B, w = B). As with the hard snap, a visual or auditory warning can be presented to the listener 410. In some examples, a visual or auditory warning can be presented when the snap occurs and when the location 416 of the listener 410 enters the region between the snap location X1 and the snap location X2. For example, when the location 416 of the listener 410 is between the locations A, the audio source w is at the location A (e.g., the audio component 438), and in some examples, crossfading can be presented and depends on both time and the movement of the listener.

[0135] Figure 5C is a conceptual diagram illustrating another example of a soft snap. Figure 5C One or more elements of can be part of the user interface 36. For example, a formula or a graph representing a formula or a combination thereof can be presented in the user interface 36. In Figure 5C example, the threshold region between the location X1 and the location X2 is extended to the entire region between the location A and the location B. In this example, the listener 410 may only hear the audio from the location A when the location 416 of the listener 410 is at the location A, may hear a mixture of the audio from the location A and the location B when the location 416 of the listener 410 is between the location A and the location B, and may only hear the audio from the location B when the location 416 of the listener 410 is at the location B.

[0136] The above example ( Figure 5A-Figure 5C ) can be a location-related snap. In other examples, the snap may not be location-related. For example, the listener 410 can through a user interface (such as Figure 4C) rather than by moving the position 416 of the listener 410 to fit to the location.

[0137] Figure 5D is a conceptual diagram illustrating an example concert having three or more audio elements. Figure 5D One or more elements may be part of user interface 36. For example, audio element 310A, audio element 310B, audio element 310C, audio element 310D, singer 312, string section 314, drummer 316, other musicians 318, headphones 321, smart glasses 324, VR goggles 322, or any combination thereof may be represented in user interface 36. Figure 5D In the example of , multiple musicians are depicted on stage 323. Singer 312 is located behind audio element 310A. String group 314 is depicted behind audio element 310B. Drummer 316 is depicted behind audio element 310C. Other musicians 318 are depicted behind audio element 310D. Audio elements 310A-310D can capture an audio stream corresponding to the sound received by the microphone. In some examples, audio elements 310A-310D can represent a synthesized audio stream. For example, audio element 310A can represent (one or more) captured audio streams associated mainly with singer 312, but this (one or more) audio stream can also include sounds produced by other band members (such as string group 314, drummer 316 or other musicians 318), while audio element 310B can represent (one or more) captured audio streams associated mainly with string group 314, but include sounds produced by other band members. In this manner, each of the audio elements 310A-310D may represent a different audio stream(s).

[0138] Likewise, multiple devices are depicted. These devices represent user devices located at multiple different desired listening locations. Headphone 320 is located near audio element 310A, but between audio element 310A and audio element 310B. As such, stream selection unit 44 may select at least one of the audio streams to produce for the user of headset 320 a sound similar to that of the user located at the desired listening location, according to the techniques of the present disclosure. Figure 5D Similarly, VR goggles 322 are shown as being located behind audio element 310C and between drummer 316 and other musicians 318. Stream selection unit 44 may select at least one audio stream to produce an audio experience for the user of VR goggles 322 similar to that of the user located at the position of headset 320. Figure 5D The audio experience of a user at the location of the VR goggles 322.

[0139] Smart glasses 324 are shown as being relatively centrally located between audio elements 310A, 310C, and 310D. Stream selection unit 44 may select at least one audio stream to produce for the user of smart glasses 324 a sound stream similar to that located at Figure 5D In addition, device 326 (which may represent any device capable of implementing the techniques of this disclosure, such as a mobile handset, a speaker array, headphones, VR goggles, smart glasses, etc.) is shown as being located in front of audio element 310B. Stream selection unit 44 may select at least one audio stream to produce for the user of device 326 an audio experience similar to that of the device located at the position of smart glasses 324. Figure 5D Although specific devices are discussed with respect to specific locations, use of any of the depicted devices may provide for an audio experience for a user at a location where device 325 is located. Figure 5D An indication of different desired listening locations depicted in FIG.

[0140] Figure 5E is a conceptual diagram illustrating an example concert having three or more audio elements. Figure 5E One or more elements may be part of user interface 36. For example, audio element S1 446, audio element S2 448, audio element S3 450, audio element S4 452, audio element R1 454, audio element R2 456, audio element R3 458, audio element R4 460, stage 444, listener's location 430, or listener's location 432, or any combination thereof may be represented in user interface 36. Figure 5E A concert layout is depicted. In this example, as a listener moves between multiple audio elements near the stage (S1 446, S2 448, S3 450, S4 452) and audio elements away from the stage (R1 454, R2 456, R3 458, R4 460), the listener may want to stick to the audio elements that are positioned further away from the listener.

[0141] For example, when the listener wants to hear the performer closer (e.g., closer to the stage 444), but wants to see more of the stage (e.g., a wider view than a position closer to the stage), so that the listener's position 430 is farther away from the stage, the listener can bias the fit audio source distance threshold toward the stage audio element S2 448 instead of the closest audio element R2 456 (distance a>distance b). In some examples, with this bias toward the stage audio element, the listener remains fit to S2 448 as the listener moves toward position 432. At position 432, the listener can fit to audio element S3 450 because the distance (c) from the listener to audio element S3 450 is less than the distance from the listener to audio element S2 448. Without the bias, the listener would fit to audio elements R2 456 and R3 458 as the listener moves from position 430 to position 432.

[0142] Fig. 5F 4 is a flowchart illustrating an example user interface technique according to the present disclosure. A content consumer device 14 may store multiple audio streams (462). For example, a content consumer device 14 may receive a bitstream 27 from a source device 12 and the bitstream 27 may include representations of multiple audio streams, one or more audio streams may be recreated from these representations and multiple audio streams may be stored. One or more processors of the content consumer device 14 may present a user interface 36 (464). In some examples, the user interface 36 may be a graphical user interface. In some examples, the user interface 36 may be a gesture-based user interface, such as a user interface configured to recognize a user's gesture. In some examples, the user interface 36 may be a voice-command-based user interface, such as a voice-command-recognizing user interface. In some examples, the user interface 36 may be a touch-based user interface, such as a touch-recognizing user interface (e.g., a multi-touch user interface). In other examples, the user interface 36 may be another type of user interface.

[0143] The one or more processors of content consumer device 14 may obtain an indication of a desired listening location via user interface 36 (466). For example, the user may indicate a desired listening location, such as desired listening location 413, via user interface 36. The one or more processors of content consumer device 14 may select at least one audio stream from a plurality of audio streams based on the indication (468). For example, stream selection unit 44 may select at least one audio stream so that the user can experience the audio as if the user were located at the desired listening location. In some examples, content consumer device 14 may output at least one audio stream, for example, to one or more speakers (470).

[0144] Figure 5Gis a flow chart illustrating further examples of user interface techniques according to the present disclosure. In some examples, Figure 5G The technology can be used with Fig. 5F technology together. Figure 5G The one or more processors of content consumer device 14 may obtain an indication that the user desires to activate the snap mode (472). The snap mode may be as described in relation to Figure 5A-5E 14. The present invention may be described in detail. For example, user 1102 may provide an indication to user interface 36 that she desires to activate the fit mode. One or more processors of content consumer device 14 may determine a first audio source distance threshold (474). In some examples, the first audio source distance threshold may be based on an indication from the user. In other examples, the first audio source distance threshold may be preset or communicated in the bitstream 27 from source device 12. In other examples, the first audio source distance threshold may be generated by content consumer device 14. For example, in an example of hard fit, one or more processors of content consumer device 14 may determine the first audio source distance threshold as fit point S 435. In other examples, in an example of soft fit, one or more processors of content consumer device 14 may determine the first audio source distance threshold as fit location X1. In an example of soft fit, one or more processors may also determine a second audio source distance threshold (476). For example, one or more processors of content consumer device 14 may determine the second audio source distance threshold as fit location X2.

[0145] The one or more processors of the content consumer device 14 may select at least one audio stream (478) further based on the first audio source distance threshold (e.g., in the case of hard snapping) or further based on the first audio source distance threshold and the second audio source distance threshold (e.g., in the case of soft snapping). For example, in the case of hard snapping, the one or more processors of the content consumer device 14 may compare the position of the listener's position (e.g., position 416) with the first audio source distance threshold (e.g., snap point S 435), and if the listener's position is closer to the given audio stream, e.g., audio element 434, the stream selection unit 44 may select the given audio stream. For example, in the case of soft snapping, the one or more processors of the content consumer device 14 may compare the position of the listener's position (e.g., position 416) with the first audio source distance threshold (e.g., snap position X1 439) to the first audio stream (e.g., audio element 438), and with the second audio stream distance threshold (e.g., snap position X2), and based on the comparison, the stream selection unit 44 may select the first audio stream, the second audio stream, or both. In some examples, content consumer device 14 may output at least one audio stream ( 480 ), eg, fed to one or more speakers.

[0146] Figure 6 is a diagram illustrating an example of a wearable device 500 that can operate according to various aspects of the technology described in the present disclosure. In various examples, the wearable device 500 can represent a VR headset (such as the VR headset 1100 described above), an AR headset, an MR headset, or any other type of extended reality (XR) headset. Augmented reality "AR" can refer to computer-rendered images or data overlaid on the real world where the user is actually located. Mixed reality "MR" can refer to computer-rendered images or data that are world-locked to a specific location in the real world, or can refer to a variant of VR in which partly computer-rendered 3D elements and partly photographed real elements are combined into an immersive experience that simulates the user's physical presence in the environment. Extended reality "XR" can refer to VR, AR, and MR collectively. More information about the terminology used for XR can be found in Jason Peterson's document entitled "Virtual Reality, Augmented Reality, and Mixed Reality Definitions" and dated July 7, 2017.

[0147] Wearable device 500 may represent other types of devices, such as watches (including so-called “smart watches”), glasses (including so-called “smart glasses”), headphones (including so-called “wireless headphones” and “smart headphones”), smart clothing, smart jewelry, etc. Whether representing a VR device, a watch, glasses, and / or headphones, wearable device 500 may communicate with a computing device supporting wearable device 500 via a wired connection or a wireless connection.

[0148] In some cases, a computing device supporting the wearable device 500 may be integrated within the wearable device 500, and thus, the wearable device 500 may be considered the same device as the computing device supporting the wearable device 500. In other cases, the wearable device 500 may communicate with a separate computing device that may support the wearable device 500. In this regard, the term "support" should not be understood as requiring a separate dedicated device, but rather one or more processors configured to perform various aspects of the techniques described in this disclosure may be integrated within the wearable device 500 or integrated within a computing device separate from the wearable device 500.

[0149] For example, when wearable device 500 represents a VR device, a separate dedicated computing device (such as a personal computer including one or more processors) can render audio and visual content, while wearable device 500 can determine translational head movement, and in accordance with various aspects of the technology described in this disclosure, the dedicated computing device can render audio content (as a speaker feed) based on the translational head movement. As another example, when wearable device 500 represents smart glasses, wearable device 500 can include one or more processors that both determine translational head movement (via an interface within one or more sensors of wearable device 500) and render speaker feeds based on the determined translational head movement.

[0150] As shown, the wearable device 500 includes one or more directional speakers, and one or more tracking and / or recording cameras. In addition, the wearable device 500 includes one or more inertial, tactile and / or health sensors, one or more eye tracking cameras, one or more high-sensitivity audio microphones, and optical / projection hardware. The optical / projection hardware of the wearable device 500 may include durable translucent display technology and hardware.

[0151] Wearable device 500 also includes connection hardware, which may represent one or more network interfaces that support multi-mode connections (such as 4G communications, 5G communications, Bluetooth, etc.). Wearable device 500 also includes one or more ambient light sensors, one or more cameras and night vision sensors, and one or more bone conduction transducers. In some cases, wearable device 500 may also include one or more passive and / or active cameras with fisheye lenses and / or telephoto lenses. Although not described in Figure 6 As shown in FIG. 5 , the wearable device 500 may also include one or more light emitting diode (LED) lights. In some examples, the (one or more) LED lights may be referred to as "super bright" LED lights. In some embodiments, the wearable device 500 may also include one or more rear cameras. It should be understood that the wearable device 500 may exhibit a variety of different form factors.

[0152] Additionally, tracking and recording cameras and other sensors can facilitate the determination of translation distance. Figure 6 As shown in the example of FIG. 5 , the wearable device 500 may include other types of sensors for detecting the translation distance.

[0153] Although specific examples of wearable devices (such as those described above with respect to Figure 2 The VR head mounted device 1100 discussed in the example of Figure 1A-Figure 1C The other devices described in the examples are described, but those skilled in the art will understand that Figure 1A-Figure 1C and Figure 2The description can be applied to other examples of wearable devices. For example, other wearable devices (such as smart glasses) can include sensors by which translational head movement is obtained. As another example, other wearable devices (such as smart watches) can include sensors by which translational movement is obtained. As such, the techniques described in this disclosure should not be limited to a particular type of wearable device, but rather any wearable device can be configured to perform the techniques described in this disclosure.

[0154] Fig. 7A and Figure 7B is a diagram illustrating an example system that can perform various aspects of the techniques described in this disclosure. Fig. 7A An example is illustrated in which source device 12C further includes a camera 600. Camera 600 may be configured to capture video data and provide the captured raw video data to content capture device 20. Content capture device 20 may provide the video data to another component of source device 12C for further processing into viewport-divided portions.

[0155] exist Fig. 7A In the example of , the content consumer device 14C also includes a VR headset 1100. It will be appreciated that in various implementations, the VR headset 1100 may be included in the content consumer device 14C or externally coupled to the content consumer device 14C. The VR headset 1100 includes display hardware and speaker hardware for outputting video data (e.g., associated with various viewports) and for rendering audio data.

[0156] Figure 7B The figure shows Fig. 7A The example of the audio renderer 32 shown in FIG. 1 is replaced by a binaural renderer 42 capable of performing binaural rendering using one or more HRTFs or other functions capable of rendering to left and right speaker feeds 43. The audio playback system 16C of the content consumer device 14D may output the left and right speaker feeds 43 to headphones 48.

[0157] The headset 48 may be connected via a wired connection (such as a standard 3.5 mm audio jack, a universal system bus (USB) connection, an optical audio jack, or other form of wired connection) or wirelessly (such as via Bluetooth TM The headset 48 may be coupled to the audio playback system 16C (e.g., wireless network connection, wireless network connection, etc.). The headset 48 may recreate the sound field represented by the audio data 19' based on the left and right speaker feeds 43. The headset 48 may include a left earphone speaker and a right earphone speaker, which are powered (or in other words, driven) by the corresponding left and right speaker feeds 43. It should be noted that the content consumer device 14C and the content consumer device 14D may be connected to Figure 1C for use with source device 12B.

[0158] Figure 8 The diagram shows Figure 1A-Figure 1C A block diagram of example components of one or more of a source device and a content consumer device as shown in the example of FIG. Figure 8 In the example of , device 710 includes a processor 712 (which may be referred to as "one or more processors" or "processor(s)"), a graphics processing unit (GPU) 714, a system memory 716, a display processor 718, one or more integrated speakers 740, a display 703, a user interface 720, an antenna 721, and a transceiver module 722. In the example where device 710 is a mobile device, display processor 718 is a mobile display processor (MDP). In some examples, such as examples where device 710 is a mobile device, processor 712, GPU 714, and display processor 718 may be formed as an integrated circuit (IC).

[0159] For example, an IC may be considered a processing chip within a chip package and may be a system on chip (SoC). In some examples, two of processor 712, GPU 714, and display processor 718 may be housed together in the same IC, while the other may be housed in a different integrated circuit (e.g., a different chip package), or all three may be housed in different ICs or on the same IC. However, in an example where device 710 is a mobile device, processor 712, GPU 714, and display processor 718 may all be housed in different integrated circuits.

[0160] Examples of processor 712, GPU 714, and display processor 718 include, but are not limited to, one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Processor 712 may be a central processing unit (CPU) of device 710. In some examples, GPU 714 may be dedicated hardware that includes integrated and / or discrete logic circuits that provide GPU 714 with large-scale parallel processing capabilities suitable for graphics processing. In some cases, GPU 714 may also include general-purpose processing capabilities and may be referred to as a general-purpose GPU (GPGPU) when performing general-purpose processing tasks (e.g., non-graphics-related tasks). Display processor 718 may also be dedicated integrated circuit hardware that is designed to retrieve image content from system memory 716, assemble the image content into image frames, and output image frames to display 703.

[0161] The processor 712 can execute various types of applications. Examples of applications include web browsers, email applications, spreadsheets, video games, other applications that generate viewable objects for display, or any of the types of applications listed in more detail above. The system memory 716 can store instructions for executing applications. Execution of one of the applications on the processor 712 causes the processor 712 to generate graphics data for image content to be displayed and audio data 19 to be played (possibly via integrated speakers 740). The processor 712 can send the graphics data of the image content to the GPU 714 for further processing based on the instructions or commands sent by the processor 712 to the GPU 714.

[0162] Processor 712 may communicate with GPU 714 according to an application specific processing interface (API). Examples of such APIs include of API, Khronos or OpenGL and OpenCL TM However, aspects of the present disclosure are not limited to DirectX, OpenGL or OpenCL APIs, and can be extended to other types of APIs. In addition, the techniques described in the present disclosure do not need to work according to an API, and the processor 712 and the GPU 714 can utilize any process for communication.

[0163] The system memory 716 may be a memory for the device 710. The system memory 716 may include one or more computer-readable storage media. Examples of the system memory 716 include, but are not limited to, random access memory (RAM), electrically erasable programmable read-only memory (EEPROM), flash memory, and other media that may be used to carry or store desired program code in the form of instructions and / or data structures and that may be accessed by a computer or processor.

[0164] In some examples, system memory 716 may include instructions that cause processor 712, GPU 714, and / or display processor 718 to perform functions attributed in the present disclosure to processor 712, GPU 714, and / or display processor 718. Accordingly, system memory 716 may be a computer-readable storage medium having stored thereon instructions that, when executed, cause one or more processors (e.g., processor 712, GPU 714, and / or display processor 718) to perform various functions.

[0165] System memory 716 may include non-transitory storage media. The term "non-transitory" indicates that the storage media is not embodied in a carrier wave or propagating signal. However, the term "non-transitory" should not be interpreted to mean that system memory 716 is non-removable or that its contents are static. As an example, system memory 716 may be removed from device 710 and moved to another device. As another example, a memory substantially similar to system memory 716 may be inserted into device 710. In some examples, non-transitory storage media may store data that may change over time (e.g., in RAM).

[0166] The user interface 720 may represent one or more hardware or virtual (meaning a combination of hardware and software) user interfaces through which a user may interface with the device 710. The user interface 720 may include physical buttons, switches, toggles, lights, or virtual versions thereof. The user interface 720 may also include a physical or virtual keyboard, a touch interface such as a touch screen, tactile feedback, and the like.

[0167] The processor 712 may include one or more hardware units (including so-called "processing cores") configured to perform all or some portion of the operations discussed above with respect to any of the modules, units, or other functional components of the content creator device and / or the content consumer device. For example, the processor 712 may implement the operations discussed above in Figure 4A-4E 721 and the transceiver module 722 may represent a unit configured to establish and maintain a connection between the source device 12 and the content consumer device 14. The antenna 721 and the transceiver module 722 may represent one or more receivers and / or one or more transmitters capable of wireless communication according to one or more wireless communication protocols. For example, the transceiver module 722 may receive and / or transmit wireless signals. The transceiver module 722 may represent a separate transmitter, a separate receiver, both a separate transmitter and a separate receiver, or a combined transmitter and receiver. The antenna 721 and the transceiver module 722 may be configured to receive encoded audio data. Similarly, the antenna 721 and the transceiver module 722 may be configured to transmit encoded audio data.

[0168] Figure 9A-9C The diagram shows Figure 1A-Figure 1C A flowchart of an example operation of the stream selection unit 44 in performing various aspects of the stream selection technique is shown in the example of FIG. Fig.9AFor example, the stream selection unit 44 may obtain audio streams 27 from all enabled audio elements, wherein the audio streams 27 may include corresponding audio metadata, such as ALI 45A (800). The stream selection unit 44 may perform energy analysis on each of the audio streams 27 to calculate a corresponding energy map (802).

[0169] Stream selection unit 44 may then iterate through different combinations of audio elements (defined in CM 47) based on proximity to audio source 308 (as defined by audio source distances 306A and / or 306B) and proximity to audio elements (as defined by proximity distances discussed above) (804). Fig.9A Stream selection unit 44 may loop based on the listener location represented by DLI 45B (which is another way to refer to "virtual location" or "device location") and the audio element location represented by ALI 45A in the manner described above to identify whether a larger subset of audio streams 27 or a reduced subset of audio streams 27 is desired (806, 808).

[0170] When a larger subset of the audio stream 27 is needed, the stream selection unit 44 may add (one or more) audio elements to the audio data 19', or in other words, additional (one or more) audio streams (such as in Figure 3A In the example of , when the user is closer to the audio source) (810). When a reduced subset of the audio stream 27 is needed, the stream selection unit 44 can remove audio elements from the audio data 19', or in other words, (one or more) existing audio streams (such as in Figure 3A In the example of , when the user is further away from the audio source)(812).

[0171] In some examples, the stream selection unit 44 may determine that the current clustering of audio elements is the best set (or, in other words, the existing audio data 19' will remain the same because the selection process described herein results in the same audio data 19') (804), and the process may return to 802. However, when audio streams are added to the audio data 19' or removed from the audio data 19', the stream selection unit 44 may update the CM 47 (814), generate a clustering history (815 (including positioning, energy maps, etc.).

[0172] In addition, the stream selection unit 44 can determine whether the privacy setting enables or disables the addition of audio elements (wherein the privacy setting can refer to digital access permissions that restrict access to one or more of the audio streams 27, for example, by a password, authorization level or order, time, etc.) (816, 818). When the privacy setting enables the addition of audio elements, the stream selection unit 44 can add (one or more) audio elements to the updated CM 47 (which refers to adding (one or more) audio streams to the audio data 19') (820). When the privacy setting disables the addition of audio elements, the stream selection unit 44 can remove (one or more) audio elements from the updated CM 47 (which refers to removal from (one or more) audio streams of the audio data 19') (822). In this way, the stream selection unit 44 can identify a new set of enabled audio elements (824).

[0173] Stream selection unit 44 may loop in this manner and update various inputs according to any given frequency. For example, stream selection unit 44 may update privacy settings at a user interface rate (meaning that the updates are driven by updates input via the user interface). As another example, stream selection unit 44 may update positioning at a sensor rate (meaning that the positioning is changed by movement of an audio element). Stream selection unit 44 may also update the energy map at an audio frame rate (meaning that the energy map is updated once per frame).

[0174] Next reference Fig. 9B For example, the stream selection unit 44 may be as described above with respect to Fig.9A The stream selection unit 44 may operate in the same manner as described, except that the stream selection unit 44 may not base the determination of the CM 47 on the energy map. In this way, the stream selection unit 44 may obtain the audio streams 27 from all enabled audio elements, wherein the audio streams 27 may include corresponding audio metadata, such as the ALI 45A (840). The stream selection unit 44 may determine whether a privacy setting enables or disables the addition of the audio element (wherein a privacy setting may refer to a digital access permission that limits access to one or more of the audio streams 27, e.g., by a password, authorization level or order, time, etc.) (842, 844).

[0175] When the privacy setting enables the addition of audio elements, the stream selection unit 44 may add (one or more) audio elements to the updated CM 47 (which refers to adding audio streams to the audio data 19') (846). When the privacy setting disables the addition of audio elements, the stream selection unit 44 may remove (one or more) audio elements from the updated CM 47 (which refers to removing (one or more) audio streams from the audio data 19') (848). In this way, the stream selection unit 44 may identify a new set of enabled audio elements (850). The stream selection unit 44 may loop (852) through different combinations of audio elements in the CM 47 to determine a cluster history (854) that represents the audio data 19'.

[0176] The stream selection unit 44 may loop in this manner and update the various inputs according to any given frequency. For example, the stream selection unit 44 may update the privacy settings at the user interface rate (meaning that the updates are driven by updates input via the user interface). As another example, the stream selection unit 44 may update the positioning at the sensor rate (meaning that the positioning is changed by the movement of the audio element).

[0177] Next reference Fig. 9C For example, the stream selection unit 44 may be as described above with respect to Fig.9A 45A (860). Stream selection unit 44 may perform energy analysis on each of the audio streams 27 to calculate a corresponding energy map (862).

[0178] Stream selection unit 44 may then cycle through different combinations of audio elements (defined in CM 47) based on proximity to audio source 308 (as defined by audio source distances 306A and / or 306B) and proximity to audio elements (as defined by proximity distances discussed above) (864). Fig. 9C Stream selection unit 44 may loop in the manner described above based on the listener location represented by DLI 45B (which is again another way of referring to the “virtual location” or “device location” discussed above) and the audio element location represented by ALI 45A to identify whether a larger subset of audio streams 27 or a reduced subset of audio streams 27 is desired (866, 868).

[0179] When a larger subset of the audio stream 27 is needed, the stream selection unit 44 may add (one or more) audio elements to the audio data 19', or in other words, additional (one or more) audio streams (such as in Figure 3A In the example of , when the user is closer to the audio source) (870). When a reduced subset of the audio stream 27 is needed, the stream selection unit 44 can remove (one or more) audio elements from the audio data 19', or in other words, (one or more) existing audio streams (such as in Figure 3A In the example of , when the user is further away from the audio source)(872).

[0180] In some examples, the stream selection unit 44 may determine that the current clustering of audio elements is the best set (or, in other words, the existing audio data 19' will remain the same because the selection process described herein results in the same audio data 19') (864), and the process may return to 862. However, as audio streams are added to or removed from the audio data 19', the stream selection unit 44 may update the CM 47 (874), generating a clustering history (875).

[0181] The stream selection unit 44 may loop in this manner and update the various inputs according to any given frequency. For example, the stream selection unit 44 may update the positioning at the sensor rate (meaning that the positioning is changed by the movement of the audio element). The stream selection unit 44 may also update the energy map at the audio frame rate (meaning that the energy map is updated once per frame).

[0182] Fig.10 The figure illustrates an example of a wireless communication system 100 according to aspects of the present disclosure. The wireless communication system 100 includes a base station 105, a UE 115, and a core network 130. In some examples, the wireless communication system 100 may be a long term evolution (LTE) network, an LTE-Advanced (LTE-A) network, an LTE-A Pro network, or a new radio (NR) network. In some cases, the wireless communication system 100 may support enhanced broadband communications, ultra-reliable (e.g., mission-critical) communications, low-latency communications, or communications with low-cost and low-complexity devices.

[0183] The base station 105 may communicate wirelessly with the UE 115 via one or more base station antennas. The base station 105 described herein may include or may be referred to by those skilled in the art as a base transceiver station, a radio base station, an access point, a radio transceiver, a NodeB, an eNodeB (eNB), a next generation NodeB or a giga-NodeB (any of which may be referred to as a gNB), a home NodeB, a home eNodeB, or some other suitable terminology. The wireless communication system 100 may include different types of base stations 105 (e.g., macro base stations or small cell base stations). The UE 115 described herein may be able to communicate with various types of base stations 105 and network equipment, including macro eNBs, small cell eNBs, gNBs, relay base stations, etc.

[0184] Each base station 105 may be associated with a particular geographic coverage area 110 in which communications with various UEs 115 are supported. Each base station 105 may provide communications coverage for a respective geographic coverage area 110 via a communications link 125, and the communications link 125 between the base station 105 and the UE 115 may utilize one or more carriers. The communications link 125 shown in the wireless communication system 100 may include uplink transmissions from the UE 115 to the base station 105, or downlink transmissions from the base station 105 to the UE 115. Downlink transmissions may also be referred to as forward link transmissions, and uplink transmissions may also be referred to as reverse link transmissions.

[0185] The geographic coverage area 110 for the base station 105 can be divided into sectors that constitute part of the geographic coverage area 110, and each sector can be associated with a cell. For example, each base station 105 can provide communication coverage for a macro cell, a small cell, a hotspot, or other type of cell, or various combinations thereof. In some examples, the base station 105 can be mobile and thus provide communication coverage for a mobile geographic coverage area 110. In some examples, different geographic coverage areas 110 associated with different technologies can overlap, and overlapping geographic coverage areas 110 associated with different technologies can be supported by the same base station 105 or different base stations 105. The wireless communication system 100 may include, for example, a heterogeneous LTE / LTE-A / LTE-A Pro or NR network, in which different types of base stations 105 provide coverage for various geographic coverage areas 110.

[0186] UE 115 can be dispersed throughout the wireless communication system 100, and each UE 115 can be fixed or mobile. UE 115 can also be referred to as a mobile device, a wireless device, a remote device, a handheld device or a subscriber device, or some other suitable term, where "device" can also be referred to as a unit, a station, a terminal or a client. UE 115 can also be a personal electronic device, such as a cellular phone, a personal digital assistant (PDA), a tablet computer, a laptop computer or a personal computer. In the examples of the present disclosure, UE 115 can be any of the audio sources described in the present disclosure, including a VR headset, an XR headset, an AR headset, a vehicle, a smart phone, a microphone, a microphone array, or any other device including a microphone or capable of sending a captured and / or synthesized audio stream. In some examples, the synthesized audio stream can be an audio stream stored in a memory or previously created or synthesized. In some examples, UE 115 may also refer to a wireless local loop (WLL) station, an Internet of Things (IoT) device, an Internet of Everything (IoE) device, or a machine type communication (MTC) device, etc., which may be implemented in various items such as appliances, vehicles, meters, etc.

[0187] Certain UEs 115, such as MTC or IoT devices, may be low-cost or low-complexity devices and may provide automated communication between machines (e.g., via machine-to-machine (M2M) communication). M2M communication or MTC may refer to data communication technology that allows devices to communicate with each other or with base stations 105 without human intervention. In some examples, M2M communication or MTC may include communications from devices that exchange and / or use audio metadata indicating privacy restrictions and / or cryptographic-based privacy data to switch, mute, and / or zero various audio streams and / or audio sources.

[0188] In some cases, the UE 115 may also be able to communicate directly with other UEs 115 (e.g., using a peer-to-peer (P2P) or device-to-device (D2D) protocol). One or more UEs 115 in a group of UEs 115 utilizing D2D communication may be within a geographic coverage area 110 of a base station 105. Other UEs 115 in such a group may be outside the geographic coverage area 110 of the base station 105 or may not be able to receive transmissions from the base station 105. In some cases, a group of UEs 115 communicating via D2D communication may utilize a one-to-many (1:M) system, in which each UE 115 transmits to each other UE 115 in the group. In some cases, the base station 105 facilitates the scheduling of resources for D2D communication. In other cases, D2D communication is performed between UEs 115 without the participation of the base station 105.

[0189] The base stations 105 may communicate with the core network 130 and with each other. For example, the base stations 105 may interface with the core network 130 via a backhaul link 132 (e.g., via S1, N2, N3, or other interface). The base stations 105 may communicate with each other directly (e.g., directly between the base stations 105) or indirectly (e.g., via the core network 130) via a backhaul link 134 (e.g., via X2, Xn, or other interface).

[0190] In some cases, the wireless communication system 100 can utilize licensed and unlicensed radio frequency spectrum bands. For example, the wireless communication system 100 can adopt License Assisted Access (LAA), LTE-unlicensed (LTE-U) radio access technology, or NR technology in an unlicensed band (such as the 5GHz Industrial, Scientific, Medical (ISM) band). When operating in an unlicensed radio frequency spectrum band, wireless devices such as base stations 105 and UEs 115 can adopt a listen-before-talk (LBT) process to ensure that the frequency channel is idle before sending data. In some cases, operations in unlicensed bands can be based on carrier aggregation configurations together with component carriers (e.g., LAA) operating in licensed bands. Operations in unlicensed spectrum can include downlink transmissions, uplink transmissions, peer-to-peer transmissions, or a combination of these. Duplexing in unlicensed spectrum can be based on frequency division duplexing (FDD), time division duplexing (TDD), or a combination of both.

[0191] This disclosure includes the following examples.

[0192] Example 1. A device configured to play one or more of a plurality of audio streams, the device comprising: a memory configured to store a plurality of audio streams and corresponding audio metadata, each of the audio streams representing a sound field, and the audio metadata comprising capture location information representing a capture location in a display world, at which a corresponding one of the plurality of audio streams was captured by a corresponding capture device; and one or more processors coupled to the memory and configured to: output a graphical user interface including a capture device representation associated with coordinates of the display world for display; obtain an indication of a user input representing a selection of one or more of the capture device representations; and select at least one of the plurality of audio streams based on the indication.

[0193] Example 2. The device of Example 1, wherein the one or more processors are further configured to reproduce the corresponding at least one sound field based on the at least one audio stream.

[0194] Example 3. The device of Example 1, wherein the one or more processors are further configured to output at least one audio stream or an indication of at least one audio stream to a reproduction device.

[0195] Example 4. The device of any combination of Examples 1 to 3, wherein the one or more processors are further configured to combine the at least two audio streams based on an indication of a user input representing a selection of one or more of the capture device representations.

[0196] Example 5. The device of Example 4, wherein the one or more processors are configured to combine the at least two audio streams by at least one of: mixing the at least two audio streams, or interpolating a third audio stream based on the at least two audio streams.

[0197] Example 6. The device of any combination of Examples 1 to 5, wherein the capture device representation comprises at least one of an icon, an image, or an alphanumeric representation corresponding to the capture device.

[0198] Example 7. The device of any combination of Examples 1-6, wherein the capture device representation corresponds to a corresponding device for capturing the corresponding audio stream.

[0199] Example 8. The device of Example 7, wherein the capture device representations are arranged in the graphical user interface in a spatial relationship representing the relative positioning of respective devices for capturing corresponding audio streams.

[0200] Example 9. The device of any combination of Examples 1 to 8, wherein the indication representing the user input represents a selection of the user input represented by the plurality of capture devices.

[0201] Example 10. The device of any combination of Examples 8-9, wherein the indication representing the user input represents the user input defining a region surrounding the plurality of capture device representations, wherein the region is associated with coordinates of the display world.

[0202] Example 11. A device as any combination of Examples 7 to 10, wherein the one or more processors are further configured to obtain a user input indication representing importance assigned to a region associated with coordinates of the display world or a corresponding device, and wherein the importance indicates a relative gain to be applied to an audio stream associated with the corresponding device or region.

[0203] Example 12. The device of any combination of Examples 1 to 11, wherein the one or more processors are further configured to obtain an indication of a desired listening position, and wherein the one or more processors reproduce the corresponding at least one sound field based on the listening position.

[0204] Example 13. The device of Example 12, wherein the one or more processors are further configured to output for display a graphical user interface element suggesting an alternative listening location in response to obtaining an indication indicating a desired listening location.

[0205] Example 14. A device as any combination of Examples 1 to 13, wherein the one or more processors are further configured to: obtain an indication indicating a relative angle of a listening experience; determine virtual position information representing a virtual position of the device in a display world; and select at least one audio stream based on the virtual position information, the capture position information, and the indication indicating a relative angle of the listening experience.

[0206] Example 15. The device of any combination of Examples 1 to 14, wherein the one or more processors are further configured to obtain an indication indicating a switch of display of metadata associated with the capture location.

[0207] Example 16. A device as in Example 15, wherein the capture device representation includes an enhanced tag for the metadata when the metadata is output for display, and wherein the indication of the user input representing the selection of one or more of the capture device representations includes an indication representing a single touch impression on a display device communicatively coupled to the one or more processors, a multi-touch impression on the display device, a gesture on the display device, or a voice command.

[0208] Example 17. The device of any combination of Examples 1 to 16, wherein the one or more processors are further configured to obtain an indication indicating a switch of display of the energy map with the capture locations.

[0209] Example 18. A device as in any combination of Examples 1 to 17, wherein the device comprises an extended reality headset, and wherein the displayed world comprises a visual scene represented by video data captured by a camera.

[0210] Example 19. An apparatus as in any combination of Examples 1 to 17, wherein the apparatus comprises an extended reality headset, and wherein the displayed world comprises a virtual world.

[0211] Example 20. The device of any combination of Examples 1 to 19, further comprising a head mounted display configured to present the display world and the graphical user interface.

[0212] Example 21. The apparatus of any combination of Examples 1 to 19, wherein the apparatus comprises a mobile handset.

[0213] Example 22. The device of any combination of Examples 1 to 21, further comprising a receiver configured to receive a plurality of audio streams.

[0214] Example 23. The device of Example 22, wherein the receiver comprises a receiver configured to receive multiple audio streams according to a fifth generation (5G) cellular standard.

[0215] Example 24. The device of Example 22, wherein the receiver comprises a receiver configured to receive the plurality of audio streams according to a personal area network standard.

[0216] Example 25. A method comprising: storing, by a memory, a plurality of audio streams and corresponding audio metadata, each of the audio streams representing a sound field, and the audio metadata including capture location information representing a capture location in a display world at which a corresponding one of the plurality of audio streams is captured; outputting, by one or more processors, a graphical user interface including a capture device representation associated with coordinates of the display world for display; obtaining, by the one or more processors, an indication of user input representing selection of one or more of the capture device representations; and selecting, by the one or more processors and based on the indication, at least one of the plurality of audio streams.

[0217] Example 26. The method of Example 25 further includes reproducing, by one or more processors, at least one corresponding sound field based on at least one audio stream.

[0218] Example 27. The method of Example 25, further comprising outputting, by one or more processors, at least one audio stream or an indication of at least one audio stream to a reproduction device.

[0219] Example 28. The method of any combination of Examples 25 to 27, further comprising combining, by the one or more processors, the at least two audio streams based on an indication of a user input representing a selection of one or more of the capture device representations.

[0220] Example 29. The method of Example 28, wherein combining the at least two audio streams comprises mixing the at least two audio streams or interpolating a third audio stream based on the at least two audio streams.

[0221] Example 30. The method of any combination of Examples 25 to 29, wherein the capture device representation comprises at least one of an icon, an image, or an alphanumeric representation corresponding to the capture device.

[0222] Example 31. The method of any combination of Examples 25 to 30, wherein the capture device representation corresponds to a respective device for capturing the respective audio stream.

[0223] Example 32. The method of Example 31, wherein the capture device representations are arranged in the graphical user interface in a spatial relationship representing the relative positioning of respective devices for capturing corresponding audio streams.

[0224] Example 33. The method of any combination of Examples 25 to 32, wherein the indication representing the user input represents a selection of the user input represented by the plurality of capture devices.

[0225] Example 34. The method of any combination of Examples 32 to 33, wherein the indication representing the user input represents the user input defining an area surrounding the plurality of capture device representations, wherein the area is associated with coordinates of the display world.

[0226] Example 35. A method as any combination of Examples 31 to 34, further comprising: obtaining, by one or more processors, a user input indication representing importance assigned to a region associated with coordinates of the display world or a corresponding device, and wherein the importance indicates a relative gain to be applied to an audio stream associated with the corresponding device or region.

[0227] Example 36. A method as in any combination of Examples 25 to 35, further comprising: obtaining, by one or more processors, an indication of a desired listening position; and reproducing, by one or more processors, at least one corresponding sound field based on the listening position.

[0228] Example 37. The method of Example 36, further comprising: outputting, by one or more processors, a graphical user interface element suggesting an alternative listening location in response to obtaining an indication indicating a desired listening location.

[0229] Example 38. A method as any combination of Examples 25 to 37, further comprising: obtaining, by one or more processors, an indication of a relative angle of a listening experience; determining, by one or more processors, virtual position information representing a virtual position of a device in a display world; and selecting, by one or more processors, at least one audio stream based on the virtual position information, the capture position information, and the indication of the relative angle of the listening experience.

[0230] Example 39. The method of any combination of Examples 25 to 38, further comprising: obtaining, by the one or more processors, an indication of a switch indicative of display of metadata associated with the capture location.

[0231] Example 40. The method of Example 39, wherein the capture device representation includes an enhanced tag for the metadata when the metadata is output for display, and wherein the indication of the user input representing selection of one or more of the capture device representations includes an indication representing a single touch impression on a display device communicatively coupled to the one or more processors, a multi-touch impression on the display device, a gesture on the display device, or a voice command.

[0232] Example 41. The method of any combination of Examples 25 to 40, further comprising: obtaining, by one or more processors, a user input indication indicating a switch of display of the energy map with the capture locations.

[0233] Example 42. A method as in any combination of Examples 25 to 41, wherein the device comprises an extended reality headset, and wherein the displayed world comprises a visual scene represented by video data captured by a camera.

[0234] Example 43. A method as in any combination of Examples 25 to 42, wherein the device comprises an extended reality headset, and wherein the displayed world comprises a virtual world.

[0235] Example 44. A method as in any combination of Examples 25 to 43, wherein the head mounted display is configured to present the display world and the graphical user interface.

[0236] Example 45. The method of any combination of Examples 25 to 43, wherein the mobile handset is configured to present the display world and the graphical user interface.

[0237] Example 46. A method as in any combination of Examples 25 to 45, further comprising: receiving, by a receiver, a plurality of audio streams.

[0238] Example 47. The method of Example 46, wherein the receiver comprises a receiver configured to receive multiple audio streams according to a fifth generation (5G) cellular standard.

[0239] Example 48. The device of Example 46, wherein the receiver comprises a receiver configured to receive the plurality of audio streams according to a personal area network standard.

[0240] Example 49. A device configured to play one or more of a plurality of audio streams, the device comprising: means for storing a plurality of audio streams and corresponding audio metadata, each of the audio streams representing a sound field, and the audio metadata including capture location information representing a capture location in a display world, at which a corresponding one of the plurality of audio streams was captured by a corresponding capture device; means for outputting a graphical user interface for display including a capture device representation associated with coordinates of the display world; means for obtaining an indication of a user input representing a selection of one or more of the capture device representations; and means for selecting at least one of the plurality of audio streams based on the indication.

[0241] Example 50. The apparatus of Example 49 further comprises means for reproducing at least one corresponding sound field based on at least one audio stream.

[0242] Example 51. The apparatus of Example 49, further comprising means for outputting at least one audio stream or an indication of at least one audio stream to a reproduction device.

[0243] Example 52. The device of any combination of Examples 49 to 51, further comprising means for combining the at least two audio streams based on an indication of a user input indicating selection of one or more of the capture device representations.

[0244] Example 53. The apparatus of Example 52, wherein the means for combining the at least two audio streams comprises means for mixing the at least two audio streams or means for interpolating a third audio stream based on the at least two audio streams.

[0245] Example 54. The device of any combination of Examples 49 to 53, wherein the capture device representation comprises at least one of an icon, an image, or an alphanumeric representation corresponding to the capture device.

[0246] Example 55. The device of any combination of Examples 49 to 54, wherein the capture device represents a device corresponding to a corresponding device for capturing a corresponding audio stream.

[0247] Example 56. The device of Example 55, wherein the capture device representations are arranged in the graphical user interface in a spatial relationship representing the relative positioning of respective devices for capturing corresponding audio streams.

[0248] Example 57. The device of any combination of Examples 49 to 56, wherein the indication representing the user input represents a selection of the user input represented by the plurality of capture devices.

[0249] Example 58. The device of any combination of Examples 56 to 57, wherein the indication representing the user input represents the user input defining a region surrounding the plurality of capture device representations, wherein the region is associated with coordinates of the display world.

[0250] Example 59. A device as any combination of Examples 55 to 58, further comprising: a device for obtaining a user input indication representing an importance assigned to a region associated with coordinates of the display world or a corresponding device, and wherein the importance indicates a relative gain to be applied to an audio stream associated with the corresponding device or region.

[0251] Example 60. The apparatus of any combination of Examples 49 to 59, further comprising: means for obtaining an indication of a desired listening location; and means for reproducing the corresponding at least one sound field based on the listening location.

[0252] Example 61. The apparatus of Example 60, further comprising means for outputting a graphical user interface element suggesting an alternative listening location in response to obtaining an indication representing a desired listening location.

[0253] Example 62. A device as any combination of Examples 49 to 61, further comprising: a device for obtaining an indication of a relative angle indicating a listening experience; a device for determining virtual position information representing a virtual position of the device in a display world; and a device for selecting at least one audio stream based on the virtual position information, the capture position information, and the indication of the relative angle indicating the listening experience.

[0254] Example 63. The apparatus of any combination of Examples 49 to 62, further comprising means for obtaining an indication indicating a switch in display of metadata associated with the capture location.

[0255] Example 64. A device as in Example 63, wherein the capture device representation includes an enhanced tag for the metadata when the metadata is output for display, and wherein the indication of the user input representing selection of one or more of the capture device representations includes an indication representing a single touch impression on a display device communicatively coupled to the one or more processors, a multi-touch impression on the display device, a gesture on the display device, or a voice command.

[0256] Example 65. The apparatus of any combination of Examples 49 to 64, further comprising means for obtaining a user input indication indicating a switch of display of the energy map with the capture locations.

[0257] Example 66. A device as any combination of Examples 49 to 65, wherein the device includes an extended reality headset, and wherein the displayed world includes a visual scene represented by video data captured by a camera.

[0258] Example 67. An apparatus as any combination of Examples 49 to 65, wherein the apparatus comprises an extended reality head-mounted device, and wherein the displayed world comprises a virtual world.

[0259] Example 68. The device of any combination of Examples 49 to 65, further comprising a head mounted display configured to present the display world and the graphical user interface.

[0260] Example 69. The apparatus of any combination of Examples 49 to 65, wherein the apparatus comprises a mobile handset.

[0261] Example 70. The apparatus of any combination of Examples 49 to 69, further comprising means for receiving a plurality of audio streams.

[0262] Example 71. The apparatus of Example 70, wherein the means for receiving comprises means for receiving multiple audio streams in accordance with a fifth generation (5G) cellular standard.

[0263] Example 72. The apparatus of Example 70, wherein the means for receiving comprises means for receiving a plurality of audio streams according to a personal area network standard.

[0264] Example 73. A non-transitory computer-readable storage medium having instructions stored thereon that, when executed, cause one or more processors to: store multiple audio streams and corresponding audio metadata, each of the audio streams representing a sound field, and the audio metadata including capture location information representing a capture location in a display world, at which a corresponding one of the multiple audio streams was captured by a corresponding capture device; and output a graphical user interface including a capture device representation associated with coordinates of the display world for display; obtain an indication of user input representing selection of one or more of the capture device representations; and select at least one of the multiple audio streams based on the indication.

[0265] Example 74. A device configured to play one or more of a plurality of audio streams, the device comprising: a memory configured to store a plurality of audio streams and corresponding audio metadata, each of the audio streams representing a sound field, and the audio metadata comprising capture location information representing a capture location in a display world, at which a corresponding one of the plurality of audio streams was captured by a corresponding capture device; and one or more processors coupled to the memory and configured to: output a graphical user interface including a capture device representation associated with coordinates of the display world for display; obtain an indication of user input representing selection of a location within the graphical user interface; set an audio source distance threshold based on the indication; and select at least one of the plurality of audio streams based on the audio source distance threshold.

[0266] Example 75. The device of Example 74, wherein the one or more processors are further configured to reproduce the corresponding at least one sound field based on the at least one audio stream.

[0267] Example 76. The device of Example 74 or 75, wherein the one or more processors are further configured to output the at least one audio stream or an indication of the at least one audio stream to a reproduction device.

[0268] Example 77. The device of any combination of Examples 74-76, wherein the one or more processors are further configured to obtain an indication from the user that the user desires to activate the fit mode.

[0269] Example 78. The device of Example 77, wherein the fit mode is a hard fit mode or a soft fit mode.

[0270] Example 79. A device as in Example 78, wherein the fit mode is a soft fit mode, the audio source distance threshold is a first audio source distance threshold, the indication is a first indication and the position is a first position, and the one or more processors are further configured to: obtain a second indication representing user input selecting a second position within the graphical user interface; and set a second audio source distance threshold based on the second indication; and select at least one audio stream from the multiple audio streams based on the second audio source distance threshold.

[0271] Example 80. The device of Example 79, wherein the at least one audio stream comprises two audio streams and the one or more processors are further configured to combine the two audio streams.

[0272] Example 81. The device of Example 80, wherein the one or more processors are configured to combine the two audio streams by at least one of: adaptively mixing the two audio streams, or interpolating a third audio stream based on the two audio streams.

[0273] Example 82. The device of Example 80 or 81, wherein the one or more processors are configured to combine the two audio streams by applying a function F(x) to the two audio streams.

[0274] Example 83. The device of any combination of Examples 74-79, wherein at least one audio stream is a single audio stream.

[0275] Example 84. The device of any combination of Examples 74-83, wherein the one or more processors are further configured to: change the audio source distance threshold based on the indication.

[0276] Example 85. An apparatus as in any combination of Examples 74-84, wherein the apparatus comprises an extended reality head-mounted device, and wherein the displayed world comprises a virtual world.

[0277] Example 86. The device of any combination of Examples 74-85, further comprising a head mounted display configured to present the display world and the graphical user interface.

[0278] Example 87. An apparatus as in any combination of Examples 74-86, wherein the apparatus comprises a mobile handset.

[0279] Example 88. A device as in any combination of Examples 74-87, further comprising a receiver configured to receive a plurality of audio streams.

[0280] Example 89. The device of Example 88, wherein the receiver comprises a receiver configured to receive multiple audio streams according to a fifth generation (5G) cellular standard.

[0281] Example 90. The device of Example 88, wherein the receiver comprises a receiver configured to receive the plurality of audio streams according to a personal area network standard.

[0282] Example 91. A method comprising: storing, by a memory, a plurality of audio streams and corresponding audio metadata, each of the audio streams representing a sound field, and the audio metadata including capture location information representing a capture location in a display world at which a corresponding one of the plurality of audio streams is captured; outputting, by one or more processors, a graphical user interface including a capture device representation associated with coordinates of the display world for display; obtaining, by the one or more processors, an indication of user input representing selection of a location within the graphical user interface; and setting, by the one or more processors and based on the indication, an audio source distance threshold; and selecting, by the one or more processors and based on the audio source distance threshold, at least one of the plurality of audio streams.

[0283] Example 92. The method of Example 91 further includes reproducing, by one or more processors, at least one corresponding sound field based on at least one audio stream.

[0284] Example 93. The method of Example 91 or 92, further comprising outputting at least one audio stream or an indication of at least one audio stream to a reproduction device.

[0285] Example 94. A method as in any combination of Examples 91-93, further comprising obtaining, by one or more processors, an indication from a user that the user desires to activate the fit mode.

[0286] Example 95. The method of Example 94, wherein the fitting mode is a hard fitting mode or a soft fitting mode.

[0287] Example 96. A method as in Example 95, wherein the fit mode is a soft fit mode, the audio source distance threshold is a first audio source distance threshold, the indication is a first indication and the position is a first position, the method further comprising: obtaining, by one or more processors, a second indication representing user input selecting a second position within a graphical user interface; setting, by one or more processors and based on the second indication, a second audio source distance threshold; and selecting, by one or more processors and based on the second audio source distance threshold, at least one audio stream from a plurality of audio streams.

[0288] Example 97. The method of Example 96, wherein the at least one audio stream comprises two audio streams, and the method further comprises combining the two audio streams by one or more processors.

[0289] Example 98. The method of Example 97, wherein the combining comprises at least one of: adaptively mixing the two audio streams or interpolating a third audio stream based on the two audio streams.

[0290] Example 99. A method as in Example 97 or 98, wherein the combining comprises applying a function F(x) to the two audio streams.

[0291] Example 100. A method as in any combination of Examples 91-96, wherein at least one audio stream is a single audio stream.

[0292] Example 101. A method as in any combination of Examples 91-100, further comprising: changing, by one or more processors, the audio source distance threshold based on the indication.

[0293] Example 102. A non-transitory computer-readable storage medium having instructions stored thereon that, when executed, cause one or more processors to: store a plurality of audio streams and corresponding audio metadata, each of the audio streams representing a sound field, and the audio metadata including capture location information representing a capture location in a display world, at which a corresponding one of the plurality of audio streams was captured by a corresponding capture device; output a graphical user interface including a capture device representation associated with coordinates of the display world for display; obtain an indication of user input representing selection of a location within the graphical user interface; set an audio source distance threshold based on the indication; and select at least one of the plurality of audio streams based on the audio source distance threshold.

[0294] Example 103. A device comprising: means for storing a plurality of audio streams and corresponding audio metadata, each of the audio streams representing a sound field, and the audio metadata including capture location information representing a capture location in a display world at which a corresponding one of the plurality of audio streams was captured; means for outputting a graphical user interface including a capture device representation associated with coordinates of the display world; means for obtaining an indication of a user input representing a selection of a location within the graphical user interface; means for setting an audio source distance threshold; and means for selecting at least one of the plurality of audio streams based on the audio source distance threshold.

[0295] It should be appreciated that, depending on the example, certain actions or events of any of the techniques described herein may be performed in a different sequence, may be added, combined, or omitted entirely (e.g., not all described actions or events are necessary for the practice of the techniques). In addition, in some examples, actions or events may be performed concurrently, such as through multithreading, interrupt handling, or multiple processors, rather than sequentially.

[0296] In some examples, the VR device (or streaming device) can communicate communication messages to an external device using a network interface coupled to a memory of the VR / streaming device, wherein the exchanged messages are associated with multiple available representations of the sound field. In some examples, the VR device can receive wireless signals using an antenna coupled to the network interface, including data packets, audio packets, video protocols, or transport protocol data associated with multiple available representations of the sound field. In some examples, one or more microphone arrays can capture the sound field.

[0297] In some examples, the multiple available representations of the sound field stored to the memory device may include multiple object-based representations of the sound field, a high-order ambisonic representation of the sound field, a mixed-order ambisonic representation of the sound field, a combination of the object-based representation of the sound field and the high-order ambisonic representation of the sound field, a combination of the object-based representation of the sound field and the mixed-order ambisonic representation of the sound field, or a combination of the mixed-order representation of the sound field and the high-order ambisonic representation of the sound field.

[0298] In some examples, one or more of the sound field representations of multiple available representations of the sound field may include at least one high-resolution region and at least one low-resolution region, and wherein the selected presentation based on the steering angle provides greater spatial accuracy with respect to the at least one high-resolution region and less spatial accuracy with respect to the lower-resolution region.

[0299] In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored or sent as one or more instructions or codes on a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media, which corresponds to tangible media such as data storage media; or communication media, including any media that facilitates the transfer of a computer program from one place to another, for example, according to a communication protocol. In this manner, computer-readable media may generally correspond to (1) non-transitory tangible computer-readable storage media or (2) communication media such as signals or carrier waves. Data storage media may be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, codes, and / or data structures for implementing the techniques described in the present disclosure. A computer program product may include a computer-readable medium.

[0300] As an example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage devices, flash memory, or any other medium that can be used to store the required program code in the form of instructions or data structures and can be accessed by a computer. Similarly, any connection is appropriately referred to as a computer-readable medium. For example, if a coaxial cable, optical cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, microwaves are used to send instructions from a website, server or other remote source, then coaxial cable, optical cable, twisted pair, DSL, or wireless technologies such as infrared, radio and microwaves are included in the definition of the medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals or other temporary media, but are directed to non-temporary, tangible storage media. The disks and optical disks used herein include compact disks (CDs), laser disks, optical disks, digital versatile disks (DVDs), floppy disks and blue-ray disks, wherein the disks usually reproduce data magnetically, while the optical disks reproduce data optically with lasers. The combination of the above items should also be included in the scope of computer-readable media.

[0301] Instructions may be executed by one or more processors such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Accordingly, the term "processor" as used herein may refer to any of the aforementioned structures or any other structure suitable for implementing the techniques described herein. In addition, in some aspects, the functionality described herein may be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into a combined codec. Similarly, the technology may be fully implemented in one or more circuits or logic elements.

[0302] The techniques of the present disclosure may be implemented in a variety of devices or apparatuses, including wireless handheld devices, integrated circuits (ICs) or IC sets (e.g., chipsets). Various components, modules, or units are described in the present disclosure to emphasize functional aspects of devices configured to perform the disclosed techniques, but do not necessarily need to be implemented by different hardware units. More specifically, as described above, the various units may be combined in a codec hardware unit or provided by a collection of interoperable hardware units including one or more processors as described above in combination with appropriate software and / or firmware.

[0303] Various examples have been described. These and other examples are within the scope of the following claims.

Claims

1. A device configured to play one or more of a plurality of audio streams, the device comprising: a memory configured to store the plurality of audio streams, each of the audio streams representing a sound field; as well as One or more processors, coupled to the memory and configured to: presenting a user interface to a user; obtaining, via the user interface, a first indication from the user indicating a desired listening location; selecting a first at least one audio stream from the plurality of audio streams based on the first indication; obtaining, from the user via the user interface, a second indication representing an audio source distance threshold, the audio source distance threshold representing a distance of an audio source associated with a desired listening position; determining that the user is moving from one location to another; determining an audio source distance as a distance between the audio source and a virtual location of the user; comparing the audio source distance to the audio source distance threshold; Based on the audio source distance being greater than the audio source distance threshold, a second at least one audio stream among the plurality of audio streams is selected, wherein the second at least one audio stream is different from the first at least one audio stream. 2 . The device of claim 1 , wherein the memory is further configured to store location information associated with coordinates of an acoustic space in which a corresponding audio stream of the plurality of audio streams is captured or synthesized.

3. The device of claim 2, wherein the user interface comprises one or more of: a graphical user interface, a gesture-based user interface, a voice command-based user interface, a touch-based user interface.

4. The device of claim 3, wherein the user interface is configured to obtain user input with at least one of single touch, multi-touch, gesture, voice command, or tapping.

5. The apparatus of claim 3, wherein the user interface comprises the graphical user interface, and the graphical user interface comprises a representation associated with coordinates of an acoustic space in which the plurality of audio streams are captured or synthesized.

6. The apparatus of claim 5, wherein the representation is arranged in the graphical user interface in a spatial relationship representing the relative positioning of coordinates of an acoustic space in which the plurality of audio streams are captured or synthesized.

7. The device of claim 6, wherein the one or more processors are further configured to output for display a graphical user interface element for suggesting an alternative listening location in response to obtaining the first indication representing the desired listening location.

8. The device of claim 1, wherein the one or more processors are further configured to combine at least two audio streams based on the first indication by at least one of: mixing the at least two audio streams, or interpolating a third audio stream based on the at least two audio streams.

9. The device of claim 1, wherein the one or more processors are further configured to obtain, via the user interface, an importance indication representing an importance assigned to an audio stream, and wherein the importance indication is to be applied to a relative gain of the audio stream.

10. The device of claim 1, wherein the one or more processors are further configured to set the audio source distance threshold based on the second indication.

11. A device as described in claim 1, wherein the one or more processors are further configured to obtain a third indication from the user via the user interface that the user desires to activate a fit mode, the fit mode being configured to transform spatial coordinates for audio presentation based on a current position of the user and at least one audio source distance threshold.

12. The apparatus of claim 11, wherein the fitting mode is a hard fitting mode including a single audio source distance threshold or a soft fitting mode including two audio source distance thresholds.

13. The apparatus of claim 12, wherein the audio source distance threshold is a first audio source distance threshold and wherein the one or more processors are further configured to: determining a second audio source distance threshold, and Wherein the one or more processors are configured to select a second at least one audio stream of the plurality of audio streams further based on the second audio source distance threshold.

14. The apparatus of claim 13, wherein the one or more processors are configured to combine the two audio streams by applying a function F(x) to the two audio streams.

15. The device of claim 1, wherein the device comprises an extended reality head mounted device, and The displayed world includes a visual scene represented by video data captured by a camera.

16. The device according to claim 1, The device includes an extended reality head-mounted device, and The displayed world includes the virtual world.

17. The apparatus of claim 1, wherein the apparatus comprises a mobile handset.

18. The device of claim 1, wherein the device further comprises a wireless transceiver coupled to the one or more processors and configured to receive a wireless signal, wherein the wireless signal comprises at least one of Bluetooth or Wi-Fi, or complies with a fifth generation 5G cellular protocol.

19. A method for playing one or more of a plurality of audio streams, comprising: storing a plurality of audio streams by a memory, each of the audio streams representing a sound field, the memory being communicatively coupled to one or more processors; presenting, by the one or more processors, a user interface to a user; obtaining, by the one or more processors via the user interface, a first indication from the user indicating a desired listening location; selecting a first at least one audio stream from among the plurality of audio streams based on the first indication, obtaining, by the one or more processors, from the user via the user interface, a second indication representing an audio source distance threshold, the audio source distance threshold representing a distance of an audio source associated with a desired listening position; determining that the user is moving from one location to another; determining an audio source distance as a distance between the audio source and a virtual location of the user; comparing the audio source distance to the audio source distance threshold; as well as Based on the audio source distance being greater than the audio source distance threshold, selecting, by the one or more processors, a second at least one audio stream from the plurality of audio streams, wherein the second at least one audio stream is different from the first at least one audio stream.

20. The method of claim 19, further comprising storing position information associated with coordinates of an acoustic space in which a corresponding audio stream of the plurality of audio streams is captured or synthesized.

21. The method of claim 20, wherein the user interface comprises one or more of: a graphical user interface, a gesture-based user interface, a voice command-based user interface, a touch-based user interface.

22. The method of claim 21, further comprising obtaining, by the one or more processors via the user interface, a user input, the user input comprising at least one of a single touch, a multi-touch, a gesture, a voice command, or a tap.

23. The method of claim 21, wherein the user interface comprises the graphical user interface, and the graphical user interface comprises a representation associated with coordinates of an acoustic space in which the plurality of audio streams are captured or synthesized.

24. The method of claim 23, wherein the representation is arranged in the graphical user interface in a spatial relationship representing the relative positioning of coordinates of an acoustic space captured or synthesized in the plurality of audio streams.

25. The method of claim 24, further comprising, in response to obtaining the first indication representing the desired listening location, outputting for display a graphical user interface element for suggesting an alternative listening location.

26. The method of claim 19, further comprising combining, by the one or more processors based on the first indication, at least two audio streams by at least one of: mixing the at least two audio streams, or interpolating a third audio stream based on the at least two audio streams.

27. The method of claim 19, further comprising obtaining, by the one or more processors via the user interface, an importance indication representing an importance assigned to an audio stream, and wherein the importance indication is to be applied to a relative gain of the audio stream.

28. The method of claim 19, further comprising setting, by the one or more processors and based on the second indication, the audio source distance threshold.

29. The method of claim 28 further includes obtaining, by the one or more processors, via the user interface, a third indication from the user that the user desires to activate a fit mode, the fit mode being configured to transform spatial coordinates for audio presentation based on a current position of the user and at least one audio source distance threshold.

30. The method of claim 29, wherein the fitting mode is a hard fitting mode including a single audio source distance threshold or a soft fitting mode including two audio source distance thresholds.

31. The method of claim 30, wherein the audio source distance threshold is a first audio source distance threshold, the method further comprising: determining, by the one or more processors via the user interface, a second audio source distance threshold, Wherein selecting the second at least one audio stream among the plurality of audio streams is further based on the second audio source distance threshold.

32. The method of claim 31 , wherein the one or more processors are configured to combine the two audio streams by applying a function F(x) to the two audio streams.

33. The method of claim 19, wherein the method is performed by an extended reality head mounted device, and wherein the displayed world comprises a visual scene represented by video data captured by a camera.

34. The method of claim 19, wherein the method is performed on an extended reality head mounted device, and wherein the displayed world comprises a virtual world.

35. The method of claim 19, wherein the method is performed on a mobile handset.

36. The method of claim 19, further comprising receiving a wireless signal via a wireless transceiver, wherein the wireless signal comprises at least one of a Bluetooth signal, a Wi-Fi signal, or a signal according to a fifth generation 5G cellular standard.

37. A device configured to play one or more of a plurality of audio streams, the device comprising: means for storing the plurality of audio streams, each of the audio streams representing a sound field; means for presenting a user interface to a user; means for obtaining, via the user interface, a first indication from a user of a desired listening location; means for selecting a first at least one audio stream of the plurality of audio streams based on the first indication; means for obtaining from the user via the user interface a second indication representing an audio source distance threshold, the audio source distance threshold representing a distance of an audio source associated with a desired listening position; as well as means for determining that the user is moving from one location to another; means for determining an audio source distance as a distance between the audio source and a virtual location of the user; means for comparing said audio source distance to said audio source distance threshold; means for selecting a second at least one audio stream of the plurality of audio streams based on the audio source distance being greater than the audio source distance threshold, wherein the second at least one audio stream is different from the first at least one audio stream.

38. A non-transitory computer-readable storage medium having stored thereon instructions that, when executed, cause one or more processors to: storing a plurality of audio streams, each of the audio streams representing a sound field; presenting a user interface to a user; obtaining, via the user interface, a first indication from the user indicating a desired listening location; selecting a first at least one audio stream from the plurality of audio streams based on the first indication; obtaining, from the user via the user interface, a second indication representing an audio source distance threshold, the audio source distance threshold representing a distance of an audio source associated with a desired listening position; determining that the user is moving from one location to another; determining an audio source distance as a distance between the audio source and a virtual location of the user; comparing the audio source distance to the audio source distance threshold; as well as Based on the audio source distance being greater than the audio source distance threshold, a second at least one audio stream among the plurality of audio streams is selected, wherein the second at least one audio stream is different from the first at least one audio stream.

Images