Password-based authorization for audio rendering

Abstract

The present disclosure describes a method and apparatus for processing one or more audio streams according to a privacy restriction based on a password. A device may be configured to: receive an unrestricted audio stream from one or more audio streams based on a privacy restriction associated with a password, wherein the one or more audio streams are from audio elements represented in an acoustic environment, the acoustic environment comprising one or more sub-acoustic spaces, each of the one or more audio streams representing a corresponding sound field; and generate a corresponding sound field of the unrestricted audio stream.

Description

[0001] This application claims priority to U.S. Application No. 16 / 918,338, filed on July 1, 2020, which claims the benefit of U.S. Provisional Application No. 62 / 870,479, filed on July 3, 2019, the entire contents of each of which are incorporated herein by reference Technical Field

[0002] The present disclosure relates to the processing of media data, such as audio data. Background Art

[0003] Computer-mediated reality systems are being developed to allow computing devices to augment or add to, remove or subtract from, or generally modify an existing reality that a user experiences. Computer-mediated reality systems (which may also be referred to as "extended reality systems" or "XR systems") may include, for example, virtual reality (VR) systems, augmented reality (AR) systems, and mixed reality (MR) systems. The perceived success of computer-mediated reality systems is often related to the ability of such computer-mediated reality systems to provide a realistic immersive experience in terms of both the video and audio experience, where the video and audio experience are aligned in a manner that the user desires. Although the human visual system is more sensitive than the human auditory system (e.g., in terms of perceived positioning of various objects in a scene), ensuring an adequate auditory experience is an increasingly important factor in ensuring a realistic immersive experience, particularly as the video experience improves to allow better positioning of video objects, which enables the user to better identify the source of audio content. Summary of the invention

[0004] The present disclosure generally relates to the auditory aspects of user experience of computer-mediated reality systems, including virtual reality (VR), mixed reality (MR), augmented reality (AR), computer vision and graphics systems. Various aspects of the technology can provide user control of audio rendering of an acoustic space for an extended reality system. As used herein, an acoustic environment is represented as an indoor environment or an outdoor environment, or both an indoor environment and an outdoor environment. An acoustic environment can include one or more sub-acoustic spaces, which can include various acoustic elements. A sub-acoustic space can be, for example, a room, or an area within a room or building, a compartment of a vehicle, a classroom, an area within a classroom, or any subset of any type of acoustic space. Examples of outdoor environments can include cars, buildings, walls, forests, etc. An acoustic space can be an example of an acoustic environment and can be an indoor space or an outdoor space. As used herein, an audio element may be sound captured by a microphone (e.g., directly from a near-field source or as a reflection from a far-field source, whether real or synthesized), sound captured by a microphone array, a previously synthesized sound field, mono sound synthesized from text to speech, and / or reflections of virtual sound from objects in the acoustic environment.

[0005] When rendering an XR scene (e.g., a six degrees of freedom (6DOF) XR scene) with many audio sources that can be obtained from audio elements of the real-time scene, certain audio elements or clusters of audio elements may contain sensitive information and / or may be intended for restricted or exclusive access. According to the technology of the present disclosure, an audio playback device (e.g., a VR or XR device) may be configured to receive and / or determine restricted and unrestricted audio streams based on privacy restrictions associated with a password. In the context of the present disclosure, a password can be any information (e.g., a key, an encryption key, an access code, etc.) used to associate a specific holder of a password with a set of privacy restrictions (e.g., restrictions indicating whether a specific audio stream is played back and / or rendered).

[0006] In some examples, the audio playback device can obtain a password from a host that sends one or more audio streams. In one example, the audio playback device can request an audio stream and provide the password back to the host. In this example, the host can send only unrestricted audio streams based on the password. In other examples, the audio playback device can request an audio stream, and provide the password back to the host, and the host can send all audio streams. However, the host can also include data (e.g., audio metadata), which indicates whether a specific stream is restricted based on the password. The audio playback device will then limit playback based on the data (e.g., audio metadata). In other examples, the audio playback device will not provide the password to the host. Instead, the host can send all audio streams and multiple data (e.g., audio metadata), which indicate whether a specific stream is restricted based on a specific type of password. The audio playback device can then associate the password stored at the audio playback device with the audio metadata corresponding to the password of the same type, and then limit playback based on the stream, which is restricted based on the association.

[0007] A password may be generated for one or more zones / groups or audio streams associated with a zone / group, and authentication may be performed based on the listener or user associated with the password. The disclosed techniques may provide rendering-side (e.g., audio playback device) privacy access for situations where audio streams may not be restricted from the capture side. Such techniques provide privacy flexibility when serving dynamic sources of audio data and may add an additional layer of security.

[0008] In one example, the present disclosure describes a device configured to process one or more audio streams, the device comprising a memory configured to store the one or more audio streams; and one or more processors coupled to the memory and configured to: receive an unrestricted audio stream from one or more audio streams based on privacy restrictions associated with a password, wherein the one or more audio streams are from audio elements represented in an acoustic environment, the acoustic environment comprising one or more sub-acoustic spaces, each of the one or more audio streams representing a corresponding sound field; and generate a corresponding sound field for the unrestricted audio stream.

[0009] In another example, the present disclosure describes a method for processing one or more audio streams, the method comprising: receiving an unrestricted audio stream among the one or more audio streams based on a privacy restriction associated with a password, wherein the one or more audio streams are from audio elements represented in an acoustic environment, the acoustic environment comprising one or more sub-acoustic spaces, each of the one or more audio streams representing a corresponding sound field; and generating a corresponding sound field for the unrestricted audio stream.

[0010] In another example, the present disclosure describes a non-transitory computer-readable storage medium storing instructions that, when executed, cause one or more processors of a device configured to process one or more audio streams to: receive an unrestricted audio stream from the one or more audio streams based on privacy restrictions associated with a password, wherein the one or more audio streams are from audio elements represented in an acoustic environment, the acoustic environment comprising one or more sub-acoustic spaces, each of the one or more audio streams representing a corresponding sound field; and generate a corresponding sound field for the unrestricted audio stream.

[0011] In another example, the present disclosure describes a device configured to process one or more audio streams, the device comprising: a device for receiving an unrestricted audio stream of the one or more audio streams based on a privacy restriction associated with a password, wherein the one or more audio streams are from audio elements represented in an acoustic environment, the acoustic environment comprising one or more sub-acoustic spaces, each of the one or more audio streams representing a corresponding sound field; and a device for generating a corresponding sound field for the unrestricted audio stream.

[0012] The details of one or more examples of the present disclosure are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of various aspects of the present technology will be apparent from the detailed description, accompanying drawing description, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1A and Figure 1Bis a diagram illustrating a system that can perform various aspects of the techniques described in this disclosure.

[0014] Figure 2 is a diagram showing an example of a VR device worn by a user.

[0015] FIG. 3A to FIG. 3D is shown in more detail Figure 1A Illustration of an exemplary operation of a stream selection unit as shown in the example of .

[0016] FIG. 4A to FIG. 4D It is shown by Figure 1A and Figure 1B An illustration of example operations of password-based privacy restrictions performed by a source device and / or content consumer device as shown in the examples of FIG.

[0017] Figure 5 is a diagram illustrating an example of a wearable device that can operate according to various aspects of the techniques described in this disclosure.

[0018] Fig. 6A and Figure 6B is a diagram showing other exemplary systems that can perform various aspects of the techniques described in this disclosure.

[0019] Figure 7 is a block diagram illustrating exemplary components of one or more of the source device and the content consumer device shown in the example of FIG. 1 .

[0020] FIG. 8A to FIG. 8C It is shown Figure 1A and Figure 1B Flowchart of exemplary operations of the stream selection unit shown in the example of FIG. 1 when performing various aspects of the stream selection technology.

[0021] 9A to 9D is a flow chart illustrating an exemplary technique for processing one or more audio streams based on a cipher using the techniques of this disclosure.

[0022] Fig.10 An example of a wireless communication system supporting password-based privacy restrictions in accordance with aspects of the present disclosure is shown. DETAILED DESCRIPTION

[0023] There are many different ways to represent a sound field. Exemplary formats include channel-based audio formats, object-based audio formats, and scene-based audio formats. Channel-based audio formats refer to 5.1 surround sound formats, 7.1 surround sound formats, 22.2 surround sound formats, or any other channel-based format that positions audio channels to specific locations around the listener in order to reconstruct the sound field.

[0024] An object-based audio format may refer to a format in which audio objects (typically encoded using pulse code modulation (PCM) and referred to as PCM audio objects) are specified in order to represent a sound field. Such audio objects may include metadata identifying the position of the audio objects relative to a listener or other reference point in the sound field, so that the audio objects can be rendered to one or more speaker channels for playback in an effort to reconstruct the sound field. The techniques described in the present disclosure may be applied to any of the aforementioned formats, including scene-based audio formats, channel-based audio formats, object-based audio formats, or any combination thereof.

[0025] A scene-based audio format may include a set of hierarchical elements that define a three-dimensional sound field. An example of a set of hierarchical elements is a set of spherical harmonic coefficients (SHCs). The following expression shows a description or representation of a sound field using SHCs:

[0026]

[0027] This expression shows that any point in the sound field The pressure p at time t i Can be provided by SHC Here, c is the speed of sound (about 343 m / s), is the reference point (or observation point), j n (·) is a spherical Bessel function of order n, and is a spherical harmonic basis function (also referred to as a spherical basis function) of order n and suborder m. It can be recognized that the term in the square brackets is the signal (i.e., ), which can be approximated by various time-frequency transforms such as discrete Fourier transform (DFT), discrete cosine transform (DCT), or wavelet transform. Other examples of hierarchical sets include wavelet transform coefficient sets and other coefficient sets of multiresolution basis functions.

[0028] SHC The SHCs can be physically acquired (e.g., recorded) through various microphone array configurations, or alternatively, they can be derived from channel-based or object-based sound field descriptions. SHCs (also known as surround sound coefficients) represent scene-based audio, where the SHCs can be input to an audio encoder to obtain encoded SHCs that can facilitate more efficient transmission or storage. For example, a method involving (1+4) 2 (25, hence fourth order) The fourth order representation of the coefficients.

[0029] As described above, SHC can be derived from microphone recordings using a microphone array. Various examples of how SHC can be physically obtained from a microphone array are described in Poletti, M., "Three-Dimensional Surround Sound Systems Based on Spherical Harmonics", J. Audio Eng. Soc., Vol. 53, No. 11, Nov. 2005, pp. 1004-1025.

[0030] The following equations may show how the SHC may be derived from an object-based description. The coefficients of the sound field corresponding to each audio object can be expressed as:

[0031]

[0032] where i is is a spherical Hankel function (of the second kind) of order n, and is the position of the object. Knowing the object source energy g(ω) as a function of frequency (e.g., using time-frequency analysis techniques such as performing a Fast Fourier Transform on a PCM stream) enables the conversion of each PCM object and corresponding position into SHC Furthermore, it can be shown (since the above is a linear and orthogonal decomposition) that the coefficients of each object are are additive. In this way, the number of PCM objects can be determined by The coefficients are represented (e.g., as the sum of the coefficient vectors for each object). The coefficients may contain information about the sound field (pressure as a function of 3D coordinates), and the above represents the distance from each object to the observation point The entire sound field nearby represents the transform.

[0033] Computer-mediated reality systems (which may also be referred to as "extended reality systems" or "XR systems") are being developed to take advantage of many of the potential benefits provided by surround sound coefficients. For example, the surround sound coefficients may represent a three-dimensional sound field in a manner that potentially enables accurate three-dimensional (3D) positioning of sound sources within the sound field. In doing so, an XR device may render the surround sound coefficients to a speaker feed that, when played through one or more speakers, may accurately reproduce the sound field.

[0034] As another example, the surround sound coefficients may be transformed (e.g., rotated) to account for user movement without overly complex mathematical operations, thereby potentially accommodating the low latency requirements of XR. Additionally, the surround sound coefficients are hierarchical, thereby naturally accommodating scalability by reducing the order (which may eliminate surround sound coefficients associated with higher orders), thereby potentially enabling dynamic adjustment of the sound field to accommodate the latency and / or battery requirements of the XR device.

[0035] Using surround sound coefficients for XR can enable the development of many use cases that rely on the more immersive sound field provided by the surround sound coefficients, especially for computer gaming applications and real-time video streaming applications. In these highly dynamic use cases that rely on low-latency reproduction of the sound field, XR devices may prefer surround sound coefficients over other representations that are more difficult to manipulate or involve complex rendering. Figure 1A and Figure 1B More information about these use cases is available.

[0036] Although described in the present disclosure with respect to VR devices, various aspects of these techniques may be performed in the context of other devices, such as mobile devices. In this instance, a mobile device, such as a so-called smartphone, may present a displayed world via a screen that may be mounted on a user's head or viewed in the manner in which the mobile device is normally used. As such, any information about the screen may be part of the mobile device. The mobile device may be able to provide tracking information 41, thereby allowing both a VR experience (when mounted on the head) and a normal experience to view the displayed world, wherein the normal experience may still allow the user to view the displayed world, providing an acoustic space for a VR-lite-type experience (e.g., lifting the device and rotating or translating the device to view different parts of the displayed world).

[0037] The present disclosure generally relates to the auditory aspects of user experience of computer-mediated reality systems, including virtual reality (VR), mixed reality (MR), augmented reality (AR), computer vision and graphics systems. Various aspects of the technology can provide user control of audio rendering of an acoustic space for an extended reality system. As used herein, an acoustic environment is represented as an indoor environment or an outdoor environment, or both an indoor environment and an outdoor environment. An acoustic environment can include one or more sub-acoustic spaces, which can include various acoustic elements. A sub-acoustic space can be, for example, a room, or an area within a room or building, a compartment of a vehicle, a classroom, an area within a classroom, or any subset of any type of acoustic space. Examples of outdoor environments can include cars, buildings, walls, forests, etc. An acoustic space can be an example of an acoustic environment and can be an indoor space or an outdoor space. As used herein, an audio element may be sound captured by a microphone (e.g., directly from a near-field source or as a reflection from a far-field source, whether real or synthesized), sound captured by a microphone array, a previously synthesized sound field, monophonic sound synthesized from text to speech, and / or reflections of virtual sound from objects in the acoustic environment.

[0038] When rendering an XR scene (e.g., a six degrees of freedom (6DOF) XR scene) with many audio sources that may be obtained from audio elements of a real-time scene, certain audio elements or groups of audio elements may contain sensitive information and / or may be intended for restricted or exclusive access. According to the techniques of the present disclosure, an audio playback device (e.g., a VR or XR device) may be configured to receive and / or determine restricted and unrestricted audio streams from privacy restrictions associated with a password.

[0039] In some examples, the audio playback device can obtain a password from a host that sends one or more audio streams. In one example, the audio playback device can request an audio stream and provide the password back to the host. In this example, the host can send only unrestricted audio streams based on the password. In other examples, the audio playback device can request an audio stream, and provide the password back to the host, and the host can send all audio streams. However, the host can also include data (e.g., audio metadata), which indicates whether a specific stream is restricted based on the password. The audio playback device will then limit playback based on the data (e.g., audio metadata). In other examples, the audio playback device will not provide the password to the host. Instead, the host can send all audio streams and multiple data (e.g., audio metadata), which indicate whether a specific stream is restricted based on a specific type of password. The audio playback device can then associate the password stored at the audio playback device with the audio metadata corresponding to the password of the same type, and then limit playback based on the stream, which is restricted based on the association.

[0040] A password may be generated for one or more zones / groups or audio streams associated with a zone / group, and authentication may be performed based on the listener or user associated with the password. The disclosed techniques may provide rendering-side (e.g., audio playback device) privacy access for situations where audio streams cannot be restricted from the capture side. Such techniques provide privacy flexibility when serving dynamic sources of audio data and may add an additional layer of security.

[0041] Figure 1A and Figure 1B is a diagram illustrating a system that can perform various aspects of the technology described in this disclosure. Figure 1A As shown in the example of , system 10 includes a source device 12 and a content consumer device 14. Although described in the context of source device 12 and content consumer device 14, the technology can be implemented in any context in which any representation of a sound field is encoded to form a bitstream representing audio data. In addition, source device 12 can represent any form of computing device capable of generating a sound field representation, and is generally described herein in the context of being a VR content creator device. Similarly, content consumer device 14 can represent any form of computing device capable of implementing the rendering metadata techniques described in the present disclosure as well as audio playback, and is generally described herein in the context of being a VR client device.

[0042] Source device 12 may be operated by an entertainment company or other entity that may generate multi-channel audio content for consumption by operators of content consumer devices such as content consumer device 14. In some VR scenarios, source device 12 generates audio content in conjunction with video content. Source device 12 includes content capture device 20, content editing device 22, and sound field representation generator 24. Content capture device 20 may be configured to interface with microphone 18 or otherwise communicate.

[0043] Microphone 18 can indicate Or other types of 3D audio microphones that can capture the sound field and represent it as audio data 19, which can refer to one or more of the above-mentioned scene-based audio data (such as surround sound coefficients), object-based audio data, and channel-based audio data. Although described as a 3D audio microphone, the microphone 18 can also represent other types of microphones configured to capture audio data 19 (such as omnidirectional microphones, point microphones, unidirectional microphones, etc.).

[0044] In some examples, the content capture device 20 may include an integrated microphone 18 integrated into the housing of the content capture device 20. The content capture device 20 may interface with the microphone 18 wirelessly or via a wired connection. Rather than capturing or in conjunction with capturing audio data 19 via the microphone 18, the content capture device 20 may process the audio data 19 after the audio data 19 is input via some type of removable storage, wirelessly, and / or via a wired input process. As such, various combinations of content capture devices 20 and microphones 18 are possible in accordance with the present disclosure.

[0045] The content capture device 20 may also be configured to interface or otherwise communicate with a content editing device 22. In some instances, the content capture device 20 may include a content editing device 22 (which, in some instances, may represent software or a combination of software and hardware, including software executed by the content capture device 20 to configure the content capture device 20 to perform a particular form of content editing). The content editing device 22 may represent a unit configured to edit or otherwise alter content 21 (including audio data 19) received from the content capture device 20. The content editing device 22 may output the edited content 23 and associated metadata 25 to the sound field representation generator 24.

[0046] The sound field representation generator 24 may include any type of hardware device capable of interfacing with the content editing device 22 (or the content capturing device 20). Figure 1A , but the sound field representation generator 24 may use the edited content 23, which includes the audio data 19 and the metadata 25 provided by the content editing device 22 to generate one or more bitstreams 27. Figure 1A In the example of , the sound field representation generator 24 may generate one or more representations of the same sound field represented by the audio data 19 to obtain a bitstream 27 comprising a representation of the sound field and audio metadata 25 .

[0047] For example, to generate different representations of the sound field using surround sound coefficients (which is also an example of audio data 19), the sound field representation generator 24 can use a codec scheme for surround sound representation of the sound field, called mixed-order surround (MOA), as discussed in more detail in U.S. application serial number 15 / 672,058, entitled “MIXED-ORDER AMBISONICS (MOA) AUDIO DATA FO COMPUTER-MEDIATED REALITY SYSTEMS,” filed on August 8, 2017 and published as U.S. Patent Publication No. 20190007781 filed on January 3, 2019.

[0048] To generate a particular MOA representation of a sound field, the sound field representation generator 24 may generate a partial subset of the full set of surround sound coefficients. For example, each MOA representation generated by the sound field representation generator 24 may provide accuracy with respect to some areas of the sound field, but provide lower accuracy in other areas. In one example, the MOA representation of the sound field may include eight (8) uncompressed surround sound coefficients, while a third-order surround sound representation of the same sound field may include sixteen (16) uncompressed surround sound coefficients. As such, each MOA representation of the sound field (generated as a partial subset of the surround sound coefficients) may be less storage-intensive and less bandwidth-intensive than a corresponding third-order surround sound representation of the same sound field generated from the surround sound coefficients (if and when transmitted as part of the bitstream 27 over the transmission channel shown).

[0049] Although described with respect to an MOA representation, the techniques of the present disclosure may also be performed for a first-order surround (FOA) representation, in which all surround coefficients associated with the first-order spherical basis functions and the zero-order spherical basis functions are used to represent the sound field. In other words, the sound field representation generator 24 may use all surround coefficients of a given order N to represent the sound field, rather than using a partial non-zero subset of the surround coefficients to represent the sound field, resulting in a total surround coefficient equal to (N+1) 2 .

[0050] In this regard, the surround sound audio data (which is another way of referring to the surround sound coefficients in the MOA representation or the full-order representation, such as the first-order representation mentioned above) may include surround sound coefficients associated with spherical basis functions having the first order or lower order (which may be referred to as "1st-order surround sound audio data"), surround sound coefficients associated with spherical basis functions having mixed orders and sub-orders (which may be referred to as the "MOA representation" discussed above), or surround sound coefficients associated with spherical basis functions having an order greater than the first order (which is referred to as the "full-order representation" above).

[0051] In some examples, the content capture device 20 or the content editing device 22 may be configured to wirelessly communicate with the sound field representation generator 24. In some examples, the content capture device 20 or the content editing device 22 may communicate with the sound field representation generator 24 via one or both of a wireless connection or a wired connection. Via the connection between the content capture device 20 and the sound field representation generator 24, the content capture device 20 may provide content in a variety of content forms, which for discussion purposes are described herein as part of the audio data 19.

[0052] In some examples, the content capture device 20 may utilize various aspects of the sound field representation generator 24 (in terms of hardware or software capabilities of the sound field representation generator 24). For example, the sound field representation generator 24 may include dedicated hardware that is configured to (or dedicated software that, when executed, causes one or more processors to perform) perform psychoacoustic audio coding (such as that provided by the Moving Picture Experts Group (MPEG), the MPEG-H 3D Audio Codec standard, the MPEG-I Immersive Audio standard, or a proprietary standard (such as AptX TM (including various versions of AptX, such as Enhanced AptX – E-AptX, AptX live, AptX Stereo and AptX High Definition – AptX-HD), Advanced Audio Codec (AAC), Audio Codec 3 (AC-3), Apple Lossless Audio Codec (ALAC), MPEG-4 Audio Lossless Streaming (ALS), Enhanced AC-3, Free Lossless Audio Codec (FLAC), Monkey's Audio, MPEG-1 Audio Layer II (MP2), MPEG-1 Audio Layer III (MP3), Opus and Windows Media Audio (WMA).

[0053] The content capture device 20 may not include psychoacoustic audio encoder dedicated hardware or dedicated software, but may provide audio aspects of the content 21 in the form of a non-psychoacoustic audio codec. The sound field representation generator 24 may assist in capturing the content 21 by performing psychoacoustic audio encoding, at least in part, for the audio aspects of the content 21.

[0054] The sound field representation generator 24 may also assist in content capture and transmission by generating one or more bitstreams 27 based at least in part on audio content (e.g., MOA representations and / or third-order surround sound representations) generated from the audio data 19 (where the audio data 19 includes scene-based audio data). The bitstream 27 may represent a compressed version of the audio data 19 and any other different types of content 21 (such as compressed versions of spherical video data, image data, or text data).

[0055] As an example, the sound field representation generator 24 may generate a bitstream 27 for transmission across a transmission channel, which may be a wired or wireless channel, a data storage device, etc. The bitstream 27 may represent an encoded version of the audio data 19, and may include a main bitstream and another side bitstream, which may be referred to as side channel information or metadata. In some cases, the bitstream 27 representing a compressed version of the audio data 19 (which may also represent scene-based audio data, object-based audio data, channel-based audio data, or a combination thereof) may conform to a bitstream generated according to the MPEG-H 3D audio coding standard and / or the MPEG-I immersive audio standard.

[0056] In some examples of the present disclosure, source device 12 may be configured to generate multiple audio streams for transmission to content consumer device 14. Source device 12 may be configured to generate each of the multiple audio streams via a single content capture device 20 and / or a group (e.g., multiple content capture devices) of content capture devices 20. In some use cases, it may be desirable to be able to control which of the multiple audio streams generated by source device 12 are available for playback by content consumer device 14.

[0057] For example, audio from certain capture devices of content capture devices 20 may contain sensitive information and / or audio from certain capture devices of content capture devices 20 may not be meant for exclusive access (e.g., unrestricted access for all users). In some examples, it may be desirable to limit access to audio from certain capture devices of content capture devices 20 based on the type of information captured by the content capture devices 20 and / or based on the location of the physical zone in which the content capture devices 20 are located.

[0058] According to example techniques of the present disclosure, the source device 12 may also include a controller 31 configured to generate a password associated with a privacy setting in a side channel 33 for one or more of the multiple audio streams generated by the source device 12. In the context of the present disclosure, a password may be any information (e.g., a key, an encryption key, an access code, etc.) used to associate a particular holder of the password with a set of privacy restrictions (e.g., restrictions indicating whether a particular audio stream is played back and / or rendered). The controller 31 may send the password directly to the content consumer device 14, or the password may be obtained out-of-band by other means by the content consumer device 14. In some examples, the controller 31 need not be a separate physical unit. Instead, the controller 31 may be integrated into the content editing device 22 or the sound field representation generator 24.

[0059] In one example, the password indicates whether one or more of the plurality of audio streams is restricted or unrestricted. In other examples, the controller 31 may only generate passwords associated with unrestricted audio streams. In this example, the content consumer device 14 may infer that the audio streams that are not associated with the password are restricted, or vice versa.

[0060] The content consumer device 14 provides the password back to the controller 31 when requesting an audio stream for playback. In some examples, the controller 31 will only send audio streams that are unrestricted for that particular password. In other examples, the controller 31 sends all audio streams to the content consumer device 14, where the audio streams are tagged with metadata indicating which streams are unrestricted or restricted based on the password. The content consumer device 14 can then restrict playback of restricted streams based on the password stored at the content consumer device 14. Figure 1A In one example, controller 31 sends the password in side channel 33. In other examples, controller 31 may send the password in bitstream 27. In other examples, content consumer device 14 may obtain the password out-of-band by other means. Additional examples of privacy restrictions, including privacy restrictions based on passwords, are described below with reference to FIG. 4A to FIG. 4D Let's describe it in more detail.

[0061] The content consumer device 14 may be operated by an individual and may represent a VR client device. Although described with respect to a VR client device, the content consumer device 14 may represent other types of devices, such as an augmented reality (AR) client device, a mixed reality (MR) client device (or other XR client device), a standard computer, a headset, a headphone, a mobile device (including so-called smartphones), or any other device capable of tracking the head movements and / or general translational movements of the individual operating the client consumer device 14. Figure 1A As shown in the example of , content consumer device 14 includes audio playback system 16A, which may refer to any form of audio playback system capable of rendering audio data for playback as multi-channel audio content.

[0062] Despite Figure 1A 2 is shown as being sent directly to content consumer device 14, but source device 12 may output bitstream 27 to an intermediate device located between source device 12 and content consumer device 14. The intermediate device may store bitstream 27 for later delivery to content consumer device 14, which may request bitstream 27. The intermediate device may include a file server, a network server, a desktop computer, a laptop computer, a tablet computer, a mobile phone, a smart phone, or any other device capable of storing bitstream 27 for later retrieval by an audio decoder. The intermediate device may reside in a content delivery network capable of streaming bitstream 27 (and possibly in conjunction with sending a corresponding video data bitstream) to a subscriber (such as content consumer device 14) requesting bitstream 27.

[0063] Alternatively, source device 12 may store bitstream 27 to a storage medium, such as an optical disc, digital video disc, high-definition video disc, or other storage medium, most of which are capable of being read by a computer and may therefore be referred to as a computer-readable storage medium or a non-transitory computer-readable storage medium. In this context, a transmission channel may refer to a channel through which content stored to the medium (e.g., in the form of one or more bitstreams 27) is sent (and may include retail stores and other store-based delivery mechanisms). In any case, the technology of the present disclosure should therefore not be limited in this regard. Figure 1A .

[0064] As noted above, the content consumer device 14 includes an audio playback system 16A. The audio playback system 16A may represent any system capable of playing back multi-channel audio data. The audio playback system 16A may include a plurality of different renderers 32. The renderers 32 may each provide different forms of rendering, wherein the different forms of rendering may include one or more of various ways of performing vector base amplitude panning (VBAP) and / or one or more of various ways of performing sound field synthesis. As used herein, "A and / or B" means "A or B", or "both A and B".

[0065] The audio playback system 16A may also include an audio decoding device 34. The audio decoding device 34 may represent a device configured to decode the bitstream 27 to output audio data 19' (where the apostrophe may represent that the audio data 19' is different from the audio data 19 due to lossy compression (such as quantization) of the audio data 19). Moreover, the audio data 19' may include scene-based audio data, which in some examples may form a full first-order (or higher-order) surround sound representation or a subset of a full first-order (or higher-order) surround sound representation that forms a MOA representation of the same sound field, a decomposition of a full first-order (or higher-order) surround sound representation, such as the main audio signal, ambient surround sound coefficients, and vector-based signals described in the MPEG-H 3D Audio codec standard or other forms of scene-based audio data.

[0066] Other forms of scene-based audio data include audio data defined according to the HOA (High Order Ambisonics) Transport Format (HTF). More information about HTF can be found in the technical specification (TS) entitled "Higher Order Ambisonics (HOA) Transport Format" published by the European Telecommunications Standards Institute (ETSI) in ETSI TS 103 589 V1.1.1 in June 2018 (2018-06) and in U.S. Patent Publication No. 2019 / 0918028 entitled "PRIORITY INFORMATION FOR HIGHER ORDER AMBISONIC AUDIO DATA" filed on December 20, 2018. In any case, the audio data 19' may be similar to the entire set or a partial subset of the audio data 19, but may differ due to lossy operations (e.g., quantization) and / or transmission via a transmission channel.

[0067] The audio data 19' may include channel-based audio data as an alternative to scene-based audio data or a combination thereof. The audio data 19' may include object-based audio data as an alternative to scene-based audio data or a combination thereof. As such, the audio data 19' may include any combination of scene-based audio data, object-based audio data, and channel-based audio data.

[0068] The audio renderer 32 of the audio playback system 16A may render the audio data 19' to output a speaker feed 35 after the audio decoding device 34 has decoded the bitstream 27 to obtain the audio data 19'. The speaker feed 35 may drive one or more speakers (for ease of illustration, Figure 1A Various audio representations (including scene-based audio data (and possibly channel-based audio data and / or object-based audio data) of the sound field can be normalized in a variety of ways (including N3D, SN3D, FuMa, N2D or SN2D).

[0069] To select an appropriate renderer or, in some instances, generate an appropriate renderer, the audio playback system 16A may obtain speaker information 37 indicating the number of speakers (e.g., loudspeakers or headphone speakers) and / or the spatial geometry of the speakers. In some instances, the audio playback system 16A may obtain the speaker information 37 using a reference microphone and may drive the speakers (which may refer to the output of electrical signals to cause transducer vibrations) in a manner that dynamically determines the speaker information 37. In other instances or in conjunction with the dynamic determination of the speaker information 37, the audio playback system 16A may prompt the user to interact with the audio playback system 16A and input the speaker information 37.

[0070] The audio playback system 16A may select one of the audio renderers 32 based on the speaker information 37. In some instances, the audio playback system 16A may generate one of the audio renderers 32 based on the speaker information 37 when none of the audio renderers 32 is within a certain threshold similarity metric (in terms of speaker geometry) from the speaker geometry specified in the speaker information 37. In some instances, the audio playback system 16A may generate one of the audio renderers 32 based on the speaker information 37 without first attempting to select an existing one of the audio renderers 32.

[0071] When the speaker feeds 35 are output to the headphones, the audio playback system 16A may utilize one of the renderers 32 that provides binaural rendering using a head-related transfer function (HRTF) or other function that can be rendered to the left and right speaker feeds 35 for playback by the headphone speakers, such as a binaural room impulse response renderer. The term "speaker" or "transducer" may generally refer to any speaker, including an amplifier, a headphone speaker, a bone conduction speaker, an earbud speaker, a wireless headphone speaker, etc. One or more speakers may then play back the rendered speaker feeds 35 to reproduce the sound field.

[0072] Although described as rendering the speaker feeds 35 from the audio data 19', references to rendering of the speaker feeds of the audio data 19' may refer to other types of rendering, such as rendering incorporated directly into the decoding of the speaker feeds 35 from the bitstream 27. Examples of alternative rendering may be found in Annex G of the MPEG-H 3D Audio standard, with rendering occurring during the formation of the main signal prior to sound field composition and during the formation of the background signals. As such, references to rendering of the audio data 19' should be understood to refer to either rendering of the actual audio data 19' or a decomposition of the audio data 19' or a representation thereof (such as the above-mentioned main audio signal, ambient surround coefficients, and / or vector-based signals - which may also be referred to as V-vectors or multi-dimensional surround space vectors).

[0073] The audio playback system 16A can also adapt the audio renderer 32 based on the tracking information 41. That is, the audio playback system 16A can interface with a tracking device 40 that is configured to track the head movement and possible translational movement of a user of the VR device. The tracking device 40 can represent one or more sensors (e.g., cameras—including depth cameras, gyroscopes, magnetometers, accelerometers, light emitting diodes—LEDs, etc.) that are configured to track the head movement and possible translational movement of a user of the VR device. The audio playback system 16A can adapt the audio renderer 32 based on the tracking information 41 so that the speaker feed 35 reflects the changes and possible translational movement of the user's head to correctly reproduce the sound field in response to such movement.

[0074] As described above, the content consumer device 14 may represent a VR device in which a human wearable display (which may also be referred to as a “head-mounted display”) is mounted in front of the eyes of a user operating the VR device. Figure 2 1 is a diagram showing an example of a VR device 1100 worn by a user 1102. The VR device 1100 is coupled to or otherwise includes headphones 1104, which can reproduce the sound field represented by the audio data 19' through the playback of speaker feeds 35. The speaker feeds 35 can represent analog or digital signals that can cause a diaphragm within a transducer of the headset 1104 to vibrate at various frequencies, where this process is generally referred to as driving the headset 1104.

[0075] Video, audio, and other sensory data can play an important role in the VR experience. To participate in the VR experience, the user 1102 can wear a VR device 1100 (which can also be referred to as a VR headset 1100) or other wearable electronic device. A VR client device (such as a VR headset 1100) may include a tracking device (e.g., tracking device 40) configured to track the head movement of the user 1102 and adapt the video data shown via the VR headset 1100 to account for the head movement, providing an immersive experience in which the user 1102 can experience the displayed world shown in the video data in visual three dimensions. The displayed world may refer to a virtual world (in which all of the world is simulated), an augmented world (in which parts of the world are augmented by virtual objects), or a physical world (in which real-world images are virtualized for navigation).

[0076] While VR (and other forms of XR, AR, and / or MR) may allow a user 1102 to visually reside in a virtual world, a VR headset 1100 may generally lack the ability to audibly place the user in the displayed world. In other words, a VR system (which may include a computer responsible for rendering video data and audio data—not shown for ease of illustration) may not be able to audibly place the user in the displayed world. Figure 2As shown in the example of , and VR headset 1100) may not be able to support full three-dimensional immersion audibly (and in some instances, realistically in a manner that reflects the displayed scene presented to the user via VR headset 1100).

[0077] Although described in the present disclosure with respect to VR devices, various aspects of these techniques may be performed in the context of other devices, such as mobile devices. In this instance, a mobile device, such as a so-called smartphone, may present a displayed world via a screen that may be mounted to the head of a user 1102 or viewed as when the mobile device is normally used. As such, any information about the screen may be part of the mobile device. The mobile device may be able to provide tracking information 41, thereby allowing both a VR experience (when mounted on the head) and a normal experience to view the displayed world, wherein the normal experience may still allow the user to view the displayed world, providing an acoustic space for a VR-lite experience (e.g., lifting the device and rotating or translating the device to view different parts of the displayed world).

[0078] In any case, returning to the VR device context, the audio aspects of VR have been categorized into three separate immersion categories. The first category provides the lowest level of immersion and is called three degrees of freedom (3DOF). 3DOF refers to audio rendering that takes into account head movement in three degrees of freedom (yaw, pitch, and roll), thereby allowing the user to look around freely in any direction. However, 3DOF cannot account for translational head movements where the head is not centered on the optical and acoustic center of the sound field.

[0079] The second category, referred to as 3DOF plus (3DOF+), provides three degrees of freedom (yaw, pitch, and roll) in addition to limited spatial translational movement due to head movement away from the optical and acoustic centers within the sound field. 3DOF+ can provide support for perceptual effects such as motion parallax, which can enhance immersion.

[0080] The third category, referred to as six degrees of freedom (6DOF), renders audio data in a way that takes into account three degrees of freedom in terms of head movement (yaw, pitch, and roll) and also takes into account the translation of the user in space (x, y, and z translation). Spatial translation can be introduced by sensors that track the user's position in the physical world or through input controllers.

[0081] 3DOF rendering is the current technology used for audio in VR. As such, the audio aspect of VR is not as immersive as the video aspect and therefore potentially reduces the overall immersion of the user experience. However, VR is rapidly transforming and may quickly evolve to support both 3DOF+ and 6DOF, which may provide opportunities for additional use cases.

[0082] For example, interactive gaming applications can utilize 6DOF to facilitate fully immersive gaming, where the user moves around the VR world and can interact with virtual objects by walking towards them. Additionally, interactive live streaming applications can utilize 6DOF to allow VR client devices to experience a live stream of a concert or sporting event as if they were there, allowing the user to move around the concert or sporting event.

[0083] There are a number of difficulties associated with these use cases. In the instance of fully immersive gaming, latency may need to be kept low to enable gameplay that does not cause nausea or motion sickness. Additionally, from an audio perspective, latency that causes audio playback to lose sync with the video data may reduce immersion. Furthermore, for certain types of gaming applications, spatial accuracy may be important to allow accurate responses, including to how the user perceives sound, as this allows the user to anticipate actions that are not currently in view.

[0084] In the context of a live streaming application, a large number of source devices 12A or 12B (both will be referred to as source devices 12 hereinafter for simplicity) may stream content 21, where source devices 12 may have widely varying capabilities. For example, one source device may be a smartphone with a digital fixed-lens camera and one or more microphones, while another source device may be a production-level television rig capable of obtaining higher resolution and quality video than a smartphone. However, in the context of a live streaming application, all source devices may provide streams of varying quality, and the VR device may attempt to select an appropriate stream from among them to provide the intended experience.

[0085] Furthermore, similar to gaming applications, latency in audio data that causes loss of synchronization with video data may result in a lower sense of immersion. Additionally, spatial accuracy may also be important so that the user can better understand the context or location of different audio sources. Additionally, privacy may become an issue when a user is live streaming using a camera and microphone, as the user may not want the live stream to be fully available to the public.

[0086] In the context of streaming applications (live or recorded), there may be a large number of audio streams associated with different levels of quality and / or content. The audio streams may represent any type of audio data, including scene-based audio data (e.g., ambisonic audio data, including FOA audio data, MOA audio data, and / or HOA audio data), channel-based audio data, and object-based audio data. Selecting only one of the potentially large number of audio streams to reconstruct the sound field may not provide an experience that ensures a sufficient level of immersion. However, due to the different spatial positioning between the multiple audio streams, selecting multiple audio streams may cause interference, thereby potentially reducing immersion.

[0087] According to the exemplary techniques described in the present disclosure, the audio decoding device 34 may adaptively select between audio streams available via the bitstream 27 (which are represented by the bitstream 27 and thus the bitstream 27 may be referred to as “audio stream 27”). The audio decoding device 34 may select between different audio streams of the audio stream 27 based on capture location information (CLI) 45A included as metadata accompanying the audio stream 27, wherein the capture location information may define capture coordinates in the displayed world for a microphone capturing the corresponding audio stream 27. The CLI 45A may represent a capture location in the displayed world at which a corresponding one of the audio streams 27 is captured. The audio decoding device 34 may select a subset of the audio stream 27 based on the CLI 45A, wherein the subset of the audio stream 27 excludes at least one of the audio streams 27. The audio decoding device 34 may output the subset of the audio stream 27 as audio data 19 ' (which may also be referred to as “audio stream 19 '”). The adaptive audio stream selection techniques of the present disclosure may be used with any of the cryptographic-based techniques described below.

[0088] In addition, the audio decoding device 34 may obtain tracking information 41, which the content consumer device 14 may convert into virtual position information (VLI) 45B. The VLI 45B may represent a virtual position of the content consumer device 14 in the displayed world, which may be defined as one or more device coordinates in the displayed world. The content consumer device 15 may provide the VLI 45B to the audio decoding device 34. The audio decoding device 34 may then select an audio stream 19' from the audio stream 27 based on the CLI 45A and the VLI 45B. The audio playback system 16A may then reproduce the corresponding sound field based on the audio stream 19'.

[0089] In this regard, the audio decoding device 34 may adaptively select a subset of the audio streams 27 to obtain an audio stream 19' that may result in a more immersive experience (compared to selecting a single audio stream or all audio streams 19'). In this way, various aspects of the techniques described in this disclosure may improve the operation of the audio decoding device 34 (and the audio playback system 16A and the content consumer device 14) itself by potentially enabling the audio decoding device 34 to better spatialize sound sources within a sound field, and thereby improve immersion.

[0090] In operation, the audio decoding device 34 may interface with one or more source devices 12 to determine the CLI 45A for each of the audio streams 27. Figure 1A As shown in the example of , the audio decoding device 34 may include a stream selection unit 44, which may represent a unit configured to perform various aspects of the audio stream selection techniques described in this disclosure.

[0091] The stream selection unit 44 may generate a constellation map (CM) 47 based on the CLI 45A. The CM 47 may define the CLI 45A for each of the audio streams 27. The stream selection unit 44 may also perform energy analysis on each of the audio streams 27 to determine an energy map for each of the audio streams 27, thereby storing the energy map together with the CLI 45A in the CM 47. The energy map may collectively define the energy of a common sound field represented by the audio streams 27.

[0092] The stream selection unit 44 may then determine a distance between the virtual position represented by the VLI 45B and the capture position represented by the CLI 45A, the CLI being associated with at least one and possibly each of the audio streams 27. The stream selection unit 44 may then select the audio data 19′ from the audio stream 27 based on at least one and possibly each of the distance(s), as described below with respect to Figure 3A-3D As discussed in more detail.

[0093] Furthermore, in some examples, the stream selection unit 44 may also select the audio stream 19' from the audio stream 27 based on the energy map stored in the CM 47, the CLI 45A, and the VLI 45B (collectively, when the CLI 45A and the VLI 45B are presented in the form of the distance mentioned above, the distance may also be referred to as a "relative distance"). For example, the stream selection unit 44 may analyze the energy map presented in the CM 47 to determine an audio source location (ASL) 49 of an audio source in the common sound field that emits sound captured by a microphone (such as the microphone 18) and represented by the audio stream 27. The stream selection unit 44 may then determine the audio stream 19' from the audio stream 27 based on the CLI 45A, the VLI 45B, and the ASL 49. FIG. 3A to FIG. 3D More information about the manner in which the stream selection unit 44 may select a stream is discussed.

[0094] As explained in more detail below, in one example of the present disclosure, the content consumer device 14 may be configured to: receive an unrestricted audio stream from the one or more audio streams based on a privacy restriction associated with a password, wherein the one or more audio streams are from audio elements represented in an acoustic environment comprising one or more sub-acoustic spaces, each of the one or more audio streams representing a corresponding sound field; and generate a corresponding sound field for the unrestricted audio stream.

[0095] In one example, when requesting an audio stream from source device 12, content consumer device 14 may send a password to a host device (e.g., source device 12), and based on the privacy restrictions associated with the password, receive and / or obtain only an unrestricted audio stream from the host device of one or more audio streams. The content consumer device may be configured to communicate with source device 12 via a modem and / or transceiver (e.g., over a wireless link (e.g., Bluetooth, WiFi, 5G, LTE, etc.)). In some examples, the modem / transceiver may be a separate chip within content consumer device 14. In other examples, the modem / transceiver may be integrated with one or more processors on an integrated circuit, ASIC, and / or SoC (system on chip).

[0096] In another example, when requesting an audio stream from source device 12, content consumer device 14 may send a password to a host device (e.g., source device 12), receive an unrestricted audio stream from one or more audio streams and a restricted audio stream from one or more audio streams from the host device, receive data indicating restricted playback of the restricted audio stream based on a privacy restriction associated with the password, and limit playback of a corresponding sound field of the restricted audio stream based on the data indicating the restricted playback.

[0097] In another example, content consumer device 14 may not send the password to a host device (e.g., source device 12) when requesting an audio stream from source device 12. Instead, content consumer device 14 may receive an unrestricted audio stream from one or more audio streams and a restricted audio stream from one or more audio streams from the host device, receive data indicating restricted playback of the restricted audio stream, associate the password with the data indicating restricted playback, and restrict playback of a corresponding sound field of the restricted audio stream based on the data indicating restricted playback associated with the password.

[0098] Figure 1B is a block diagram illustrating another exemplary system 50 configured to perform various aspects of the techniques described in this disclosure. The system 50 is similar to Figure 1A The system 10 shown in FIG. Figure 1A The audio renderer 32 shown in FIG. 4 is replaced with a binaural renderer 42 capable of performing binaural rendering using one or more head related transfer functions HRTFs or other functions capable of rendering to left and right speaker feeds 43 .

[0099] The audio playback system 16B may output the left and right speaker feeds 43 to headphones 1104, which may represent another example of a wearable device and which may be coupled to additional wearable devices to facilitate reproduction of the sound field, such as a watch, the VR headset mentioned above, smart glasses, smart clothing, smart rings, smart bracelets, or any other type of smart jewelry (including smart necklaces), etc. The headphones 1104 may be coupled to the additional wearable device wirelessly or via a wired connection.

[0100] In addition, the headset 1104 can be coupled to the audio playback system 16B via a wired connection (such as a standard 3.5 mm audio jack, a universal system bus (USB) connection, an optical audio jack, or other forms of wired connection) or wirelessly (such as through a Bluetooth™ connection, a wireless network connection, etc.). The headset 1104 can reconstruct the sound field represented by the audio data 19' based on the left and right speaker feeds 43. The headset 1104 can include a left headset speaker and a right headset speaker, which are powered (or in other words, driven) by the corresponding left and right speaker feeds 43.

[0101] FIG. 3A to FIG. 3D is shown in more detail Figure 1A An illustration of an exemplary operation of a stream selection unit as shown in the example of FIG. Figure 3AAs shown in the example of , the stream selection unit 44 can determine that the VLI 45B indicates that the content consumer device 14 (shown as the VR device 1100) is at the virtual location 300A. The stream selection unit 44 can then determine the audio elements 302A to 302J (which can represent not only microphones, such as Figure 1A 18 , and may also represent other types of capture devices, including a CLI 45A of one or more of a microphone array, other XR devices, a synthesized audio source, a mobile phone—including so-called smart phones—and the like).

[0102] As described above, the stream selection unit 44 may obtain the audio stream 27. The stream selection unit 44 may interface with the audio elements 302A to 302J to obtain the audio stream 27. In some examples, the stream selection unit 44 may select the audio stream 27 based on the fifth generation (5G) cellular standard, a personal area network (PAN) such as Bluetooth, or a wireless network. TM ) or some other open source, proprietary or standardized communication protocol to interact with an interface (such as a modem, receiver, transmitter and / or transceiver) to obtain the audio stream 27. Wireless communication of the audio stream is FIG. 3A to FIG. 3D In the example shown as lightning, the selected audio stream 19' is shown as being communicated from the selected one or more of the audio elements 302 to the VR device 1100.

[0103] In any case, the stream selection unit 44 may then obtain an energy map in the manner described above, analyze the energy map to determine the audio source location 304, which may represent Figure 1A An example of ASL 49 is shown in the example of FIG. The energy map can denote the audio source location 304 because the energy at the audio source location 304 can be higher than the surrounding area. Given that each of the energy maps can denote the higher energy, the stream selection unit 44 can triangulate the audio source location 304 based on the higher energy in the energy map.

[0104] Next, the stream selection unit 44 may determine the audio source distance 306A as the distance between the audio source position 304 and the virtual position 300A of the VR device 1100. The stream selection unit 44 may compare the audio source distance 306A with an audio source distance threshold. In some examples, the stream selection unit 44 may derive the audio source distance threshold based on the energy of the audio source 308. That is, when the sound source 308 has higher energy (or in other words, when the sound source 308 is noisier), the stream selection unit 44 may increase the sound source distance threshold. When the sound source 308 has low high energy (or in other words, when the sound source 308 is quieter), the stream selection unit 44 may reduce the sound source distance threshold. In other examples, the stream selection unit 44 may obtain a statically defined audio source distance threshold, which may be statically defined or specified by the user 1102.

[0105] In any case, when the audio source distance 306A is greater than the audio source distance threshold (the audio source distance threshold is assumed for illustration purposes in this example), the stream selection unit 44 may select a single audio stream (i.e., the audio source distance 306A) of the audio streams 27 captured by the audio elements 302A to 302J (“audio elements 302”). Figure 3A The stream selection unit 44 may output the corresponding audio stream in the audio stream 27, which the audio decoding device 34 may decode and output as the audio stream 19'.

[0106] Assuming the user moves from virtual position 300A to virtual position 300B, stream selection unit 44 may determine audio source distance 306B as the distance between audio source position 304 and virtual position 300B. In some examples, stream selection unit 44 may update only after some configurable release time, which may refer to the time until the receiver area increases after the listener stops moving.

[0107] In any case, the stream selection unit 44 may again compare the audio source distance 306B to the audio source distance threshold. When the audio source distance 306 is less than or equal to the audio source distance threshold (the audio source distance threshold is assumed for illustration purposes in this example), the stream selection unit 44 may select a plurality of audio streams (i.e., in the audio elements 302A to 302J) of the audio stream 27 captured by the audio elements 302A to 302J (“audio elements 302”). Figure 3A The stream selection unit 44 may output the corresponding audio stream in the audio stream 27, which the audio decoding device 34 may decode and output as the audio stream 19'.

[0108] The stream selection unit 44 may also determine a proximity distance between the virtual position 300A and one or more (and possibly each) of the capture positions represented by the CLI 45A to obtain one or more proximity distances. The stream selection unit 44 may then compare the one or more proximity distances to a threshold proximity distance. The stream selection unit 44 may select a smaller number of audio streams 27 to obtain the audio stream 19' when the one or more proximity distances are greater than the threshold proximity distance than when the proximity distance is less than or equal to the threshold proximity distance. However, the stream selection unit 44 may select a larger number of audio streams 27 to obtain the audio stream 19' when one or more of the proximity distances are less than or equal to the threshold proximity distance than when the proximity distance is less than or equal to the threshold proximity distance.

[0109] In other words, the stream selection unit 44 can attempt to select those of the audio streams 27 such that the audio stream 19' is most closely aligned with and surrounds the virtual location 300B. The proximity distance threshold can define such a threshold, which the wearable device 1100 can set or the stream selection unit 44 can dynamically determine again based on the quality of the audio elements 302F to 302J, the gain or loudness of the audio source 308, the tracking information 41 (e.g., to determine whether the user is facing the audio source 308), or any other factors.

[0110] In this regard, when the listener is at position 300B, the stream selection unit 44 can increase the audio spatialization accuracy. In addition, when at position 300A, the stream selection unit 44 can reduce the bit rate because only the audio stream captured by the audio element 302A is used instead of the multiple audio streams captured by the audio elements 302B to 302J to reproduce the sound field.

[0111] Next reference Figure 3B For example, stream selection unit 44 may determine that the audio stream captured by audio element 302A is corrupted, noisy, or unavailable. Given that audio source distance 306A is greater than an audio source distance threshold, stream selection unit 44 may remove audio streams from CM 47 and iterate through audio streams 27 according to the techniques described in more detail above to select a single audio stream in audio streams 27 (i.e., Figure 3B In the example of FIG. 1 , the audio stream captured by audio element 302B).

[0112] Next reference Figure 3C, stream selection unit 44 may obtain a new audio stream (generated by audio element 302K) and corresponding new audio metadata including CLI 45A. Stream selection unit 44 may add the new audio stream to CM 47 representing audio stream 27. Given that audio source distance 306A is greater than an audio source distance threshold, stream selection unit 44 may then iterate through audio streams 27 to select a single audio stream in audio streams 27 (i.e., at Figure 3C In the example of FIG. 1 , the audio stream captured by audio element 302B).

[0113] exist Figure 3D In the example of FIG. 3 , the audio element 302 is replaced with a specific device 320A to 320J (“device 320”), where device 320A represents a dedicated microphone of device 320A, and devices 320B, 320C, 320D, 320G, 320H, and 320J represent smartphones 320. Devices 320E, 320F, and 320I may represent VR devices 320. Each of the devices 320 may include an audio element 302 that captures an audio stream 27 selected according to various aspects of the stream selection techniques described in the present disclosure.

[0114] FIG. 4A to FIG. 4D It is shown by Figure 1A and Figure 1B 1. As described above, in some use cases, it may be desirable to be able to control which of the multiple audio streams generated by source device 12 are available for playback by content consumer device 14.

[0115] For example, audio from certain capture devices of the content capture devices 20 may contain sensitive information and / or audio from certain capture devices of the content capture devices 20 may not be meant for exclusive access (e.g., unrestricted access for all users). It may be desirable to restrict access to audio from certain capture devices of the content capture devices 20 based on the type of information captured by the content capture devices 20 and / or based on the location of the physical zone in which the content capture devices 20 are located.

[0116] like Figure 4A As shown in the example of , stream selection unit 44 may determine that VLI 45B indicates that content consumer device 14 (shown as VR device 400) is at virtual location 401. VR device 400 may be a listener of a 6DoF playback system. Stream selection unit 44 may then determine the audio elements 402A to 402J (which may represent not only microphones, such as Figure 1AThe microphone 18 shown in the figure can also represent other types of capture devices and sound generating devices, including the CLI 45A of one or more of microphone arrays, microphone groups, other XR devices, synthetic sound sources, mobile phones - including so-called smart phones - and the like.

[0117] As described above, the stream selection unit 44 may obtain the audio stream 27. The stream selection unit 44 may interface with the audio elements 402A to 402H and / or the source device 12 to obtain the audio stream 27. In some examples, the stream selection unit 44 may interact with an interface (such as a modem, a receiver, a transmitter, and / or a transceiver) according to a fifth generation (5G) cellular standard, a personal area network (PAN) such as Bluetooth™, or some other open source, proprietary, or standardized communication protocol to obtain the audio stream 27. Wireless communication of the audio stream is Figure 4A , represented as lightning in the example, where the selected audio stream 19 ′ is shown as being communicated from the selected one or more of the audio elements 402 and / or the source device 12 to the VR device 400 .

[0118] exist Figure 4A In the example of FIG. 4 , VR device 400 is at position 401 , which is near audio source 408 . Using the techniques described above and in more detail below, VR device 400 can use the energy map to determine that audio source 408 is at position 401 . Figure 4A Audio elements 402D-402H are shown at location 401. Audio elements 402A-402C are not near VR device 400.

[0119] In general, the present disclosure includes techniques for receiving one or more audio streams based on conditional privacy restrictions for zones of audio streams, groups of audio streams, and / or respective audio streams associated with passwords. The conditional privacy restrictions may include restrictions indicating situations in which a rendering side device (e.g., VR device 400) should be muted, and / or not decode (one or more) audio streams, and / or not play back (one or more) audio streams. The source device 12 may generate passwords for one or more groups of audio streams and / or audio streams associated with the groups, and authentication may be performed based on a listener or user associated with the password. Rendering side privacy access for situations in which audio streams cannot be restricted from the capture side provides privacy flexibility and an added layer of security.

[0120] In an example of the present disclosure, one or more audio streams from audio element 402 come from an audio element represented in an acoustic environment, and the acoustic environment includes one or more sub-acoustic spaces. Each of the one or more audio streams represents a corresponding sound field. As used herein, an acoustic environment is represented as an indoor environment or an outdoor environment, or both an indoor environment and an outdoor environment. An acoustic environment may include one or more sub-acoustic spaces, which may include various acoustic elements. In this context, a group of audio elements may correspond to a sub-acoustic space. A sub-acoustic space may be, for example, a room, or an area in a room or a building, a compartment of a vehicle, a classroom, an area in a classroom, or any subset of any type of acoustic space. Examples of outdoor environments may include cars, buildings, walls, forests, etc. An acoustic space (sub-acoustic space) may be an example of an acoustic environment, and may be an indoor space, an outdoor space, and / or a virtual space. As used herein, an audio element may be sound captured by a microphone (e.g., directly from a near-field source or as a reflection from a far-field source, whether real or synthesized), sound captured by a microphone array, a previously synthesized sound field (e.g., special effects and / or immersion effects), monophonic sound synthesized from text to speech, and / or reflections of virtual sound from objects in the acoustic environment.

[0121] In some examples, the content creator may be configured to set privacy settings at each audio source or group / zone of audio sources. In other examples, the controller 31 at the source device 12 may be configured to determine whether the privacy setting is required for a collection of multiple audio streams (such as by explicit instructions received by the controller 31). Based on the privacy settings for the zone, the controller 31 may cause the password generator to generate a password for the specific privacy settings of the zone / group (e.g., sub-acoustic space). The privacy settings may include one or more of the following: masking the zone, zeroing the zone, switching the zone (toggle) and / or applying an acoustic occluder to be restricted or unrestricted. In one example, switching the zone indicates whether one or more of the multiple audio streams is restricted or unrestricted. In other examples, switching the privacy restrictions only indicates the restricted audio streams.

[0122] The controller 31 may be configured to embed the password in the bitstream and / or side channel and send the password to the VR device 400. The VR device 400 may be configured to receive the password from the controller (or from another source) and send the password back to the controller when requesting an audio stream.

[0123] Figure 4B 1 is a block diagram illustrating the operation of the controller 31 of the source device 12 in one example of the present disclosure. In one example, the controller 31 may be implemented as a processor 712. Figure 7 The processor 712 is described in more detail. Figure 1A As described, source device 12 can use content capture device 20 to capture and / or generate audio data. Content capture device 20 can capture and / or generate audio data from audio element 402. Audio element 402 may include a static source, such as a static single microphone or microphone group. Audio element 402 may be a real-time source. Alternatively or in addition, audio element 402 may include a dynamic audio source (e.g., dynamic in terms of use and / or position), such as a mobile phone. In some examples, the dynamic audio source may be a synthetic audio source. The audio stream may come from a single physically spaced audio source or from a group of audio sources in a single physical location.

[0124] According to the technology of the present disclosure, the controller 31 can be configured to control the playback of audio streams from various audio elements at the rendering side (e.g., VR device 400) without the audio elements themselves controlling privacy restrictions. In this way, audio streams from various audio elements can be aggregated and distributed in dynamic settings using privacy restrictions without requiring the respective audio elements to set and / or manage such privacy settings. The technology of the present disclosure provides privacy flexibility when serving dynamic sources of audio data, and can add additional security layers.

[0125] In some examples, it may be beneficial to group audio sources that are physically located close to each other into groups or zones (e.g., sub-acoustic spaces) because each individual audio source in a physically co-located group can sense some or all of the audio as each of the other audio sources in the same physical zone. As such, in some examples of the present disclosure, the controller 31 may be configured to mask, zero, and / or switch the audio stream from the zone of the audio source. In this context, masking a zone may refer to turning down the audio gain of the zone. Zeroing a zone may refer to muting the audio from the zone (e.g., using beamforming). Switching a zone may refer to marking an audio stream or group of audio streams as unrestricted (e.g., able to be decoded and / or played) or restricted (e.g., not able to be decoded and / or played). An on (e.g., restricted) privacy switch indicates that the VR device should mute the audio stream and / or generally not decode or play the audio stream. A closed (e.g., unrestricted or co-accessible) privacy switch indicates that any user can decode and play the audio stream. In this way, an audio engineer or content creator can grant exclusive access to certain audio sources to unrestricted users or based on tiered levels of privacy settings.

[0126] like Figure 4BAs shown in , the zone controller 405 of the controller 31 can be configured to receive and / or access multiple audio streams captured by the content capture device 20. The zone controller 405 can be configured to divide the audio streams into certain zones by the physical location of the audio source. In some examples, the zone controller 405 can mark (e.g., generate metadata) indicating which zone a particular audio source belongs to. The controller 31 can also generate boundary metadata for the zone, including the centroid position and radius.

[0127] In some examples, the content creator may be configured to set privacy settings at each audio source or group / zone of audio sources. In other examples, the controller 31 may be configured to determine whether a privacy setting is required for a collection of multiple audio streams (such as through explicit instructions received by the controller 31). Based on the privacy setting for the zone, the controller 31 may cause the password generator 411 to generate a password for the specific privacy setting for the zone. In some examples, the controller 31 may encrypt the password according to the encryption type 409 (e.g., Advanced Encryption Standard, Rivest-Shamir-Adleman (RSA) encryption, etc.).

[0128] The embedding and authentication unit 407 of the controller 31 may be configured to embed the password in the bit stream 27 and / or the side channel 33 (see Figure 1A ) and to the VR device 400 or any other content consumer device (including Figure 1A and Figure 1B The password may be sent to the content consumer device 14 of the VR device 400. In other examples, the VR device 400 may obtain the password by other means (e.g., offline through another communication, or from another source). The VR device 400 may be configured to receive the password from the controller 31 (or from another source) and send the password back to the controller 31 when requesting an audio stream.

[0129] The embedding and authentication unit 407 of the controller 31 embeds respective passwords generated for groups of audio elements and / or individual audio elements using the audio stream and metadata retrieved by the controller 31. The embedding and authentication unit 407 of the controller 31 also performs authentication based on the password provided by the VR device 400. The embedding and authentication unit 407 may send the password to the VR device 400 in a data packet (e.g., via a modem or transceiver). In some examples, the embedding and authentication unit 407 may be encoded in a binary representation and may be included in the audio packet.

[0130] In one example, the controller 31 may be configured to send only unrestricted audio streams to the VR device 400 based on an authentication password. That is, in general, the VR device 400 may be configured to receive an unrestricted audio stream from one or more audio streams based on privacy restrictions associated with a password. As described above, the one or more audio streams come from audio elements represented in an acoustic environment, which includes one or more sub-acoustic spaces, each of which represents a corresponding sound field. The VR device 400 may then be configured to generate a corresponding sound field for the unrestricted audio stream. Thus, in this example, the VR device 400 may obtain a password from the controller 31 that sends one or more audio streams, or may obtain the password by other means. The VR device 400 may request an audio stream from the controller 31, and may provide the password back to the controller 31. In this example, the controller 31 may only send an audio stream that is unrestricted based on a password.

[0131] In other examples, the controller device 31 can be configured to send one or more of the audio streams to the VR device 400 along with instructions (e.g., data or audio metadata) regarding the manner in which the audio stream should be masked, muted, zeroed, and / or switched (e.g., generally, how playback should be restricted). For example, the VR device 400 can request an audio stream and provide a password back to the controller 31. The controller 31 can send all audio streams to the VR device 400. However, the controller 31 can also include data (e.g., audio metadata) indicating whether a particular stream in the sent stream is restricted based on the password authenticated by the embedding and authentication unit 407. The VR device 400 will then restrict playback of the restricted audio stream based on the data (e.g., audio metadata).

[0132] Therefore, in this example, the VR device 400 can be configured to send a password to a host device (e.g., controller 31) and receive an unrestricted audio stream in one or more audio streams and a restricted audio stream in one or more audio streams from the host device. The VR device 400 can also receive data indicating restricted playback of the restricted audio stream based on the privacy restrictions associated with the password. The VR device 400 will then limit the playback of the corresponding sound field of the restricted audio stream based on the data indicating the restricted playback. The VR device 400 will also generate a corresponding sound field for the unrestricted audio stream.

[0133] In one example, to limit the playback of the corresponding sound field of the restricted audio stream based on the data indicating the restricted playback, the VR device 400 may mask, switch, or zero the restricted audio stream. In other examples, to limit the playback of the corresponding sound field of the restricted audio stream based on the data indicating the restricted playback, the VR device 400 may be configured to apply an acoustic occluder to the restricted audio stream.

[0134] In yet other examples, the VR device 400 does not provide the password to the controller 31. Instead, the host 31 can send all audio streams and multiple data (e.g., audio metadata) that indicates whether a particular stream is restricted based on a particular type of password. That is, the audio metadata can include privacy restriction instructions for each of the multiple passwords (e.g., masking, zeroing, switching, blocking, etc.). The VR device 400 can then associate the password stored at the VR device 400 with the audio metadata corresponding to the same type of password. The VR device 400 can then limit playback based on the stream that is restricted based on the association.

[0135] Therefore, in this example, the VR device 400 can be configured to receive an unrestricted audio stream in one or more audio streams and a restricted audio stream in one or more audio streams from a host device (e.g., the controller 31). The VR device 400 can also receive data indicating restricted playback of the restricted audio stream, and can associate a password with the data indicating restricted playback. The VR device 400 can then limit the playback of the corresponding sound field of the restricted audio stream based on the data indicating restricted playback associated with the password. The VR device 400 will also generate a corresponding sound field for the unrestricted audio stream.

[0136] Moreover, in this example, in order to limit the playback of the corresponding sound field of the restricted audio stream based on the data indicating the restricted playback, the VR device 400 can mask, switch or zero the restricted audio stream. In other examples, in order to limit the playback of the corresponding sound field of the restricted audio stream based on the data indicating the restricted playback, the VR device 400 can be configured to apply an acoustic occluder to the restricted audio stream.

[0137] As described above, the privacy settings may include one or more of: masking a zone, zeroing a zone, or switching a zone to be restricted or unrestricted. In one example, switching a zone indicates whether one or more of the multiple audio streams is restricted or unrestricted. In other examples, switching the privacy restrictions indicates only the restricted audio streams.

[0138] As generally described above, the controller 31 can generate passwords at many different hierarchical levels with reference to the audio stream. The privacy restrictions associated with any one password can be based on individual audio elements, any combination of audio elements, groups or zones of audio elements (e.g., associated with sub-acoustic spaces), groups or zones of audio elements within a specific area of ​​sub-acoustic spaces, and / or combinations of groups or zones of audio elements. In addition, the privacy restrictions associated with the password can be associated not only with physical audio sources (e.g., static and dynamic audio elements 402), but also with synthetic audio sources. Synthetic audio sources can include so-called immersive effects (e.g., reflections, echoes, reverberation, etc.), which can be used to change audio streams to create special effects and / or simulate certain acoustic environments. The privacy restrictions associated with the password can be used to turn on or off such immersive effects.

[0139] In other examples, a privacy password may be associated with a zone or group of audio elements, but only with specific audio elements within the group. For example, if password authentication fails for a particular zone, the privacy restrictions associated with the password may instruct the VR device to limit playback to only a subset of audio elements within the zone or group. In still other examples, the privacy restrictions associated with the password may be associated with a scene graph. In this context, a scene graph is a hierarchical data structure that can be used to indicate relationships between various audio streams in a plurality of audio streams. Passwords may be associated with privacy restrictions at various levels of the audio scene graph.

[0140] In one example of the present disclosure, the password is a master password associated with a restricted privacy restriction. In this example, the controller 31 can be configured to generate each of the multiple audio streams. The master password can be a password for a super user / administrator. The master device password gives unrestricted access to all audio streams as a whole.

[0141] In another example of the present disclosure, the password is a permanent password associated with conditional privacy restrictions. In this example, the controller 31 can be configured to generate one or more of a plurality of audio streams based on conditional privacy restrictions, wherein the conditional privacy restrictions indicate whether one or more of a plurality of audio streams is restricted or unrestricted. In one example, the controller 31 can be configured to generate audio metadata (such as the above-mentioned audio metadata), which also includes corresponding conditional privacy restrictions, and whether one or more of the conditional privacy restrictions indicate whether a plurality of audio streams is restricted or unrestricted based on the permanent password. As will be described below, the conditional privacy restrictions may include shielding (e.g., as indicated by gain values), zeroing and / or switching. In one example, the permanent password remains valid until reset. The controller 31 can generate a permanent password for a separate zone, a zone group, a separate audio element, an audio element group and / or a synthetic source.

[0142] In another example of the present disclosure, the password is a temporary password associated with conditional privacy restrictions. In this example, the controller 31 can be configured to generate one or more of a plurality of audio streams based on conditional privacy restrictions, wherein the conditional privacy restriction authority indicates whether one or more of a plurality of audio streams is restricted or unrestricted. In one example, the controller 31 can be configured to generate audio metadata (such as the above-mentioned audio metadata), which also includes corresponding conditional privacy restrictions, and whether one or more of the conditional privacy restrictions indicate a plurality of audio streams is restricted or unrestricted based on the temporary password. As will be described below, the conditional privacy restrictions may include masking (e.g., as indicated by a gain value), zeroing and / or switching. In one example, the permanent password remains valid until reset. The temporary password remains valid for a fixed duration and expires after the fixed duration. The controller 31 can automatically invalidate the temporary password after the duration expires.

[0143] In one example, the privacy constraint comprises a respective gain value associated with a respective audio stream in one or more of the plurality of audio streams. In another example, the privacy constraint comprises a respective zeroing indication associated with a respective audio stream in one or more of the plurality of audio streams. In another example, the privacy constraint comprises a respective switching indication associated with a respective audio stream in one or more of the plurality of audio streams.

[0144] Figure 4C is a diagram showing examples of masking and zeroing zones and / or individual audio sources. In scenario 420, a password is issued to the VR device 400, the password being associated with a privacy constraint that masks zone 2 (audio elements R7 to R9). In this example, the VR device 400 also receives a gain value for zone 2 to apply when playing back the audio stream from zone 2. In scenario 430, a password is issued to the VR device 400, the password being associated with a privacy constraint that zeros audio element R4. In this example, the VR device 400 can mute the audio stream from audio element R4 (e.g., by beamforming or applying zero gain).

[0145] Figure 4D is a diagram showing an example of switching zones and / or individual audio sources. In scenario 440, a password is issued to the VR device 400, the password being associated with a privacy restriction that is restricted for switching zone 2 (audio elements R7 to R9). In this example, the VR device 400 prohibits decoding and / or playback of audio streams from zone 2. In scenario 450, a password is issued to the VR device 400, the password being associated with a privacy restriction for switching audio element R4. In this example, the VR device 400 prohibits decoding and / or playback of audio streams from audio element R4.

[0146] As can be seen from the above description, the technology of the present disclosure can provide privacy access on the rendering side (e.g., VR device 400) for situations where the audio stream cannot be restricted from the capture side. Such technology provides privacy flexibility when serving dynamic sources of audio data and can add an additional security layer.

[0147] Figure 5 is a diagram showing an example of a wearable device 500 that can operate according to various aspects of the technology described in the present disclosure. In various examples, the wearable device 500 can represent a VR headset (such as the VR headset 1100 described above), an AR headset, an MR headset, or any other type of extended reality (XR) headset. Augmented reality "AR" can refer to computer-rendered images or data superimposed on the real world where the user is actually located. Mixed reality "MR" can refer to computer-rendered images or data that are locked to a specific location in the real world, or can refer to a variant of VR in which part of the computer-rendered 3D elements and part of the real elements photographed are combined into an immersive experience that simulates the user's physical presence in the environment. Extended reality "XR" can represent VR, AR, and MR collectively. More information about XR terminology can be found in the document titled "Virtual Reality, Augmented Reality, and Mixed Reality Definitions" published by Jason Peterson on July 7, 2017.

[0148] Wearable device 500 may represent other types of devices, such as watches (including so-called “smart watches”), glasses (including so-called “smart glasses”), headphones (including so-called “wireless headphones” and “smart headphones”), smart clothing, smart jewelry, etc. Regardless of whether it represents a VR device, watch, glasses and / or headphones, wearable device 500 can communicate with a computing device that supports wearable device 500 via a wired connection or a wireless connection.

[0149] In some instances, a computing device supporting the wearable device 500 can be integrated within the wearable device 500, and as such, the wearable device 500 can be considered the same device as the computing device supporting the wearable device 500. In other instances, the wearable device 500 can communicate with a separate computing device that can support the wearable device 500. In this regard, the term "support" should not be understood as requiring a separate dedicated device, but rather one or more processors configured to perform various aspects of the techniques described in this disclosure can be integrated within the wearable device 500 or integrated within a computing device separate from the wearable device 500.

[0150] For example, when the wearable device 500 represents a VR device 500, a separate dedicated computing device (such as a personal computer including one or more processors) can render audio and visual content, and the wearable device 500 can determine the translational head movement, and the dedicated computing device can render the audio content (as a speaker feed) based on the translational head movement according to various aspects of the technology described in this disclosure. As another example, when the wearable device 500 represents smart glasses, the wearable device 500 can include one or more processors that determine the translational head movement (through an interface within one or more sensors of the wearable device 500) and render the speaker feed based on the determined translational head movement.

[0151] As shown, the wearable device 500 includes one or more directional speakers, and one or more tracking and / or recording cameras. In addition, the wearable device 500 includes one or more inertial, tactile and / or health sensors, one or more eye tracking cameras, one or more high-sensitivity audio microphones, and optical / projection hardware. The optical / projection hardware of the wearable device 500 may include durable translucent display technology and hardware.

[0152] The wearable device 500 also includes connection hardware, which may represent one or more network interfaces that support multi-mode connections, such as 4G communication, 5G communication, Bluetooth, etc. The wearable device 500 also includes one or more ambient light sensors and bone conduction sensors. In some instances, the wearable device 500 may also include one or more passive and / or active cameras with fisheye lenses and / or telephoto lenses. Although not described in Figure 5 , but the wearable device 500 may also include one or more light emitting diode (LED) lights. In some examples, the (one or more) LED lights may be referred to as (one or more) "super bright" LED lights. In some embodiments, the wearable device 500 may also include one or more rear cameras. It should be understood that the wearable device 500 can exhibit a variety of different form factors.

[0153] Additionally, tracking and recording cameras and other sensors can facilitate the determination of translation distance. Figure 5 Although shown in the example of , the wearable device 500 may include other types of sensors for detecting the translation distance.

[0154] Although specific examples of wearable devices (such as those described above) Figure 2 The VR device 500 discussed in the example herein Figure 1A and Figure 1B ) are described in detail, but a person skilled in the art will understand that Figure 1A , Figure 1B and Figure 2 The related descriptions may apply to other examples of wearable devices. For example, other wearable devices (such as smart glasses) may include sensors by which translational head movement is obtained. As another example, other wearable devices (such as smart watches) may include sensors by which translational movement is obtained. As such, the techniques described in this disclosure should not be limited to a particular type of wearable device, but rather any wearable device may be configured to perform the techniques described in this disclosure.

[0155] Fig. 6A and Figure 6B is a diagram illustrating an exemplary system that can perform various aspects of the techniques described in this disclosure. Fig. 6A An example is shown in which source device 12 also includes camera 600. Camera 600 may be configured to capture video data and provide the captured raw video data to content capture device 20. Content capture device 20 may provide the video data to another component of source device 12 for further processing into divided portions of the viewport.

[0156] exist Fig. 6A In the example of , content consumer device 14 also includes wearable device 1100. It will be appreciated that in various implementations, wearable device 1100 may be included in content consumer device 14 or externally coupled to the content consumer device. Wearable device 1100 includes display hardware and speaker hardware for outputting video data (e.g., associated with various viewports) and for rendering audio data.

[0157] Figure 6B It shows Fig. 6A The audio renderer 32 shown in FIG. 1 is replaced with an example of a binaural renderer 42 capable of performing binaural rendering using one or more HRTFs or other functions that can be rendered to left and right speaker feeds 43. The audio playback system 16C may output the left and right speaker feeds 43 to headphones 1104.

[0158] The headset 1104 may be coupled to the audio playback system 16C via a wired connection (such as a standard 3.5 mm audio jack, a universal system bus (USB) connection, an optical audio jack, or other form of wired connection) or wirelessly (such as through a Bluetooth™ connection, a wireless network connection, etc.). The headset 1104 may reconstruct the sound field represented by the audio data 19′ based on the left and right speaker feeds 43. The headset 1104 may include a left headset speaker and a right headset speaker that are powered (or in other words, driven) by the corresponding left and right speaker feeds 43.

[0159] Figure 71 is a block diagram illustrating exemplary components of one or more of the source device and the content consumer device shown in the example of FIG. Figure 7 In the example of the source device 710, the device 710 includes a processor 712 (which may be referred to as "one or more processors" or "processor"), a graphics processing unit (GPU) 714, a system memory 716, a display processor 718, one or more integrated speakers 740, a display 703, a user interface 720, an antenna 721, and a transceiver module 722. In the example where the device 710 is a mobile device, the display processor 718 is a mobile display processor (MDP). In some examples, such as examples where the source device 710 is a mobile device, the processor 712, the GPU 714, and the display processor 718 may be formed as an integrated circuit (IC).

[0160] For example, an IC may be considered a processing chip within a chip package and may be a system on chip (SoC). In some examples, two of processor 712, GPU 714, and display processor 718 may be housed together in the same IC, while another may be housed in a different integrated circuit (i.e., in a different chip package), or all three may be housed in different ICs or on the same IC. However, in an example where device 710 is a mobile device, processor 712, GPU 714, and display processor 718 may all be housed in different integrated circuits.

[0161] Examples of processor 712, GPU 714, and display processor 718 include, but are not limited to, one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuit systems. Processor 712 may be a central processing unit (CPU) of source device 710. In some examples, GPU 714 may be dedicated hardware that includes integrated and / or discrete logic circuits that provide GPU 714 with large-scale parallel processing capabilities suitable for graphics processing. In some instances, GPU 714 may also include general-purpose processing capabilities and may be referred to as a general-purpose GPU (GPGPU) when performing general-purpose processing tasks (i.e., non-graphics-related tasks). Display processor 718 may also be dedicated integrated circuit hardware that is designed to retrieve image content from system memory 716, combine the image content into image frames, and output the image frames to display 703.

[0162] Processor 712 can execute various types of applications. Examples of applications include web browsers, email applications, spreadsheets, video games, other applications that generate visual objects for display, or any of the types of applications listed in more detail above. System memory 716 can store instructions for executing applications. Execution of one of the applications on processor 712 causes processor 712 to generate graphics data of image content to be displayed and audio data 19 to be played (possibly via integrated speakers 740). Processor 712 can send graphics data of image content to GPU 714 for further processing based on instructions or commands sent by processor 712 to GPU 714.

[0163] Processor 712 may communicate with GPU 714 according to a particular application processing interface (API). Examples of such APIs include of API, Khronos Group or and OpenCL TM However, aspects of the present disclosure are not limited to DirectX, OpenGL or OpenCL APIs, and can be extended to other types of APIs. In addition, the techniques described in the present disclosure do not need to work according to an API, and the processor 712 and the GPU 714 can communicate using any process.

[0164] The system memory 716 may be a memory of the device 710. The system memory 716 may include one or more computer-readable storage media. Examples of the system memory 716 include, but are not limited to, random access memory (RAM), electrically erasable programmable read-only memory (EEPROM), flash memory, or other media that can be used to carry or store desired data in the form of instructions and / or data structures and can be accessed by a computer or processor.

[0165] In some examples, system memory 716 may include instructions that cause processor 712, GPU 714, and / or display processor 718 to perform functions attributed in this disclosure to processor 712, GPU 714, and / or display processor 718. Thus, system memory 716 may be a computer-readable storage medium having stored thereon instructions that, when executed, cause one or more processors (e.g., processor 712, GPU 714, and / or display processor 718) to perform various functions.

[0166] System memory 716 may include non-transitory storage media. The term "non-transitory" indicates that the storage media is not embodied in a carrier wave or propagating signal. However, the term "non-transitory" should not be interpreted to mean that system memory 716 is non-removable or that its contents are static. As an example, system memory 716 may be removed from source device 710 and moved to another device. As another example, a memory substantially similar to system memory 716 may be inserted into device 710. In some examples, non-transitory storage media may store data that changes over time (e.g., stored in RAM).

[0167] The user interface 720 may represent one or more hardware or virtual (meaning a combination of hardware and software) user interfaces through which a user may interact with the device 710. The user interface 720 may include physical buttons, switches, toggle switches, lights, or virtual versions thereof. The user interface 720 may also include a physical or virtual keyboard, a touch interface such as a touch screen, tactile feedback, and the like.

[0168] The processor 712 may include one or more hardware units (including so-called "processing cores") configured to perform all or part of the operations discussed above with respect to any one or more of the modules, units or other functional components of the content creator device and / or the content consumer device. The antenna 721 and the transceiver module 722 may represent units configured to establish and maintain a connection between the content consumer device 14 and the content consumer device 14. The antenna 721 and the transceiver module 722 may represent a device capable of communicating with the content consumer device 14 in accordance with one or more wireless communication protocols such as the fifth generation (5G) cellular standard, a personal area network (PAN) protocol such as Bluetooth, or a wireless communication protocol such as a cellular network. TM ), or other open source, proprietary or other communication standards) for wireless communication. That is, the transceiver module 722 can represent a separate transmitter, a separate receiver, a separate transmitter and a separate receiver, or a combined transmitter and receiver. Antenna 721 and transceiver 722 can be configured to receive encoded audio data. Similarly, antenna 721 and transceiver 722 can be configured to send encoded audio data. Transceiver 722 can also be configured as a modem.

[0169] FIG. 8A to FIG. 8C It is shown Figure 1A and Figure 1B A flowchart of an exemplary operation of the stream selection unit shown in the example of FIG. 1 when performing various aspects of the stream selection technique. Fig. 8AFor example, stream selection unit 44 may obtain all audio streams 27 from all enabled receivers (which is another way to refer to microphones, such as microphone 18), where audio streams 27 may include corresponding audio metadata, such as CLI 45A (800). Stream selection unit 44 may perform energy analysis for each of audio streams 27 to calculate a corresponding energy map (802).

[0170] Stream selection unit 44 may then iterate through different combinations of receivers (defined at CM 47) based on proximity to audio source 308 (as defined by audio source distances 306A and / or 306B) and proximity to the receivers (as defined by proximity distances discussed above) (804). Fig. 8A Receivers may be ordered or otherwise associated with different access permissions as shown in . Stream selection unit 44 may loop in the manner described above based on the listener position represented by VLI 45B (which is another way to refer to "virtual position") and the receiver position represented by CLI 45A to identify whether a larger subset of audio streams 27 or a reduced subset of audio streams 27 is desired (806, 808).

[0171] When a larger subset of the audio stream 27 is needed, the stream selection unit 44 can add receivers to the audio stream 19', or in other words, add additional audio streams (such as when the user is closer to the Figure 3A When a reduced subset of the audio stream 27 is needed, the stream selection unit 44 can remove a receiver from the audio stream 19', or in other words, remove the existing audio stream (such as when the user is away from Figure 3A ) (812) when the audio source in the example is used.

[0172] In some examples, the stream selection unit 44 may determine that the current clustering of the receiver is the best set (or in other words, the existing audio streams 19' will remain the same as the selection process described herein resulting in the same audio stream 19') (804). However, when audio streams are added to the audio stream 19' or removed from the audio data, the stream selection unit 44 may update the CM 47 (814), generating a clustering history (815).

[0173] In addition, the stream selection unit 44 can determine whether the privacy setting enables or disables the addition of receivers (where the privacy setting can refer to digital access permissions that restrict access to one or more of the audio streams 27, such as by passwords, authorization levels or rankings, time, etc.) (816, 818). When the privacy setting enables the addition of receivers, the stream selection unit 44 can add the receivers to the updated CM 47 (which means adding the audio streams to the audio streams 19') (820). When the privacy setting prohibits the addition of receivers, the stream selection unit 44 can remove the receivers from the updated CM 47 (which means removing the audio streams from the audio streams 19') (822). In this way, the stream selection unit 44 can identify that the new set of receivers is enabled (824).

[0174] The stream selection unit 44 may loop through this manner and update the various inputs according to any given frequency. For example, the stream selection unit 44 may update the privacy settings at the user interface rate (meaning that the updates are driven by updates input via the user interface). The stream selection unit 44 may update the position at the sensor rate (meaning that the position changes through movement of the receiver). The stream selection unit 44 may also update the energy map at the audio frame rate (meaning that the energy map is updated once per frame).

[0175] Next reference Figure 8B For example, the stream selection unit 44 may Fig. 8A The stream selection unit 44 may operate in the same manner as described above, except that the stream selection unit 44 may not determine the CM 47 based on the energy map. In this way, the stream selection unit 44 may obtain all audio streams 27 from all enabled receivers (which is another way to refer to microphones (such as microphone 18)), where the audio streams 27 may include corresponding audio metadata, such as CLI 45A (840). The stream selection unit 44 may determine whether a privacy setting enables or disables the addition of a receiver (where a privacy setting may refer to a digital access permission that limits access to one or more of the audio streams 27, such as by a password, authorization level or ranking, time, etc.) (842, 844).

[0176] When the privacy setting enables adding a receiver, the stream selection unit 44 may add the receiver to the updated CM 47 (which means adding the audio stream to the audio stream 19') (846). When the privacy setting prohibits adding a receiver, the stream selection unit 44 may remove the receiver from the updated CM 47 (which means removing the audio stream from the audio stream 19') (848). In this way, the stream selection unit 44 may identify that a new set of receivers is enabled (850). The stream selection unit 44 may cycle through different combinations of receivers in the CM 47 to determine a cluster map history (854), which represents the audio stream 19'.

[0177] Stream selection unit 44 may cycle through this manner and update various inputs according to any given frequency. For example, stream selection unit 44 may update privacy settings at a user interface rate (meaning that the updates are driven by updates input via the user interface). As another example, stream selection unit 44 may update location at a sensor rate (meaning that the location changes through movement of the receiver).

[0178] Next reference Figure 8C For example, the stream selection unit 44 may Fig. 8A 45A (860). Stream selection unit 44 may perform energy analysis for each of audio streams 27 to calculate a corresponding energy map (862).

[0179] Stream selection unit 44 may then cycle through different combinations of receivers (defined at CM 47) based on proximity to audio source 308 (as defined by audio source distances 306A and / or 306B) and proximity to the receivers (as defined by the proximity distances discussed above) (864). Figure 8C Receivers may be ordered or otherwise associated with different access permissions as shown in . Stream selection unit 44 may loop in the manner described above based on the listener position represented by VLI 45B (which is again another way of referring to the “virtual position” discussed above) and the receiver position represented by CLI 45A to identify whether a larger subset of audio streams 27 or a reduced subset of audio streams 27 is desired (866, 868).

[0180] When a larger subset of the audio stream 27 is needed, the stream selection unit 44 can add receivers to the audio stream 19', or in other words, add additional audio streams (such as when the user is closer to the Figure 3A When a reduced subset of the audio stream 27 is needed, the stream selection unit 44 can remove a receiver from the audio stream 19', or in other words, remove the existing audio stream (such as when the user is away from Figure 3A ) (872) when the audio source in the example is used.

[0181] In some examples, the stream selection unit 44 may determine that the current clustering of the receiver is the best set (or in other words, the existing audio streams 19' will remain the same as the selection process described herein resulting in the same audio stream 19') (864). However, when audio streams are added to the audio data 19' or removed from the audio data 19', the stream selection unit 44 may update the CM 47 (874), generating a clustering history (875).

[0182] The stream selection unit 44 may loop in this manner and update the various inputs according to any given frequency. For example, the stream selection unit 44 may update the position at the sensor rate (meaning that the position changes through movement of the receiver). The stream selection unit 44 may also update the energy map at the audio frame rate (meaning that the energy map is updated once per frame).

[0183] It should be appreciated that, depending on the example, certain actions or events of any technique described herein may be performed in a different order, may be added, combined, or excluded entirely (e.g., not all described actions or events are required to practice the technique). In addition, in some examples, actions or events may be performed simultaneously, e.g., via multithreading, interrupt handling, or multiple processor execution, rather than sequentially.

[0184] In some examples, a VR device (or streaming device) can use a network interface coupled to a memory of the VR / streaming device to transmit an exchange message to an external device, wherein the exchange message is associated with multiple available representations of a sound field. In some examples, the VR device can use an antenna coupled to the network interface to receive a wireless signal, including data packets, audio packets, video protocols, or transmission protocol data associated with multiple available representations of a sound field. In some examples, one or more microphone arrays can capture the sound field.

[0185] In some examples, the multiple available representations of the sound field stored to the memory device may include multiple object-based representations of the sound field, a higher-order surround sound representation of the sound field, a mixed-order surround sound representation of the sound field, a combination of the object-based representation of the sound field and the higher-order surround sound representation of the sound field, a combination of the object-based representation of the sound field and the mixed-order surround sound representation of the sound field, or a combination of the mixed-order representation of the sound field and the higher-order surround sound representation of the sound field.

[0186] In some examples, one or more of the sound field representations in multiple available representations of the sound field may include at least one high resolution region and at least one lower resolution region, and wherein the presentation based on the steering angle selection provides greater spatial accuracy with respect to the at least one high resolution region and lower spatial accuracy with respect to the lower resolution region.

[0187] 9A to 9Dis a flow chart illustrating an exemplary technique for processing one or more audio streams based on a cipher using the techniques of this disclosure. 9A to 9D technology can be used by content consumer devices such as Figure 1A and Figure 1B The content consumer device 14) is executed by one or more processors and / or circuits. As discussed above, in some examples, Figure 1A and Figure 1B The content consumer device 14 may be a VR device 400 (see, for example, Figure 4B ) or another XR device (e.g., an XR headset).

[0188] exist Fig.9A In a general example, the content consumer device 14 can be configured to: receive an unrestricted audio stream (900) from the one or more audio streams based on a privacy restriction associated with a password, wherein the one or more audio streams are from audio elements represented in an acoustic environment, the acoustic environment including one or more sub-acoustic spaces, each of the one or more audio streams representing a corresponding sound field; and generate a corresponding sound field (902) for the unrestricted audio stream.

[0189] Fig. 9B Shown in more detail Fig.9A An example of the technique. Fig. 9B In the example of , content consumer device 14 may be configured to request an audio stream from a host device (e.g., source device 12) (910) and send a password to the host device (912). Content consumer device 14 may also be configured to receive only an unrestricted audio stream of one or more audio streams from the host device based on a privacy restriction associated with the password (914) and generate a corresponding sound field for the unrestricted audio stream (916).

[0190] Fig. 9C Shown in more detail Fig.9A Another example of the technology. Fig. 9C In an example of , content consumer device 14 may be configured to request an audio stream from a host device (e.g., source device 12) (920) and send a password to the host device (922). Content consumer device 14 may also be configured to receive an unrestricted audio stream from one or more audio streams and a restricted audio stream from one or more audio streams from the host device (924). The content consumer device may also be configured to receive data indicating restricted playback of the restricted audio stream based on a privacy restriction associated with the password (926). Content consumer device 14 may restrict playback of a corresponding sound field of the restricted audio stream based on the data indicating restricted playback (928), and may generate a corresponding sound field of the unrestricted audio stream (930).

[0191] Fig.9D Shown in more detail Fig.9A Another example of the technology. Fig.9D In an example of , content consumer device 14 may be configured to request an audio stream from a host device (e.g., source device 12) (940). Content consumer device 14 may also be configured to receive an unrestricted audio stream from one or more audio streams and a restricted audio stream from one or more audio streams from the host device (942), and receive data indicating restricted playback of the restricted audio stream (944). Content consumer device 14 may associate a password with the data indicating restricted playback (946), and restrict playback of a corresponding sound field of the restricted audio stream based on the data indicating restricted playback associated with the password (948). Content consumer device 14 may generate a corresponding sound field of the unrestricted audio stream (950).

[0192] exist Fig. 9C and Fig.9D In each of the examples, to limit playback of the corresponding sound field of the restricted audio stream based on the data indicating restricted playback, the content consumer device 14 can be configured to mask, switch, or zero the restricted audio stream. In another example, to limit playback of the corresponding sound field of the restricted audio stream based on the data indicating restricted playback, the content consumer device 14 can be configured to apply an acoustic occluder to the restricted audio stream.

[0193] In one example of the present disclosure, the password is a master device password associated with a restricted privacy restriction. In this example, in order to receive an unrestricted audio stream in the one or more audio streams based on the privacy restriction associated with the password, the content consumer device 14 is further configured to receive each of the one or more audio streams.

[0194] In another example of the present disclosure, the password is a permanent password associated with a conditional privacy restriction. In this example, in order to receive an unrestricted audio stream in one or more audio streams based on the privacy restriction associated with the password, the content consumer device 14 is further configured to receive an unrestricted audio stream in one or more audio streams based on the conditional privacy restriction associated with the permanent password, wherein the permanent password remains valid until reset.

[0195] In another example of the present disclosure, the password is a temporary password associated with a conditional privacy restriction. In this example, in order to receive an unrestricted audio stream from one or more audio streams based on the privacy restriction associated with the password, the content consumer device 14 is further configured to receive the unrestricted audio stream from one or more audio streams based on the conditional privacy restriction associated with the temporary password, wherein the temporary password remains valid for a fixed duration and expires after the fixed duration.

[0196] Fig.10 An example of a wireless communication system 100 supporting password-based privacy restrictions in accordance with various aspects of the present disclosure is shown. The wireless communication system 100 includes a base station 105, a UE 115, and a core network 130. In some examples, the wireless communication system 100 may be a long term evolution (LTE) network, an advanced LTE (LTE-A) network, an LTE-A Pro network, or a new radio (NR) network. In some cases, the wireless communication system 100 may support enhanced broadband communications, ultra-reliable (e.g., mission-critical) communications, low-latency communications, or communications with low-cost and low-complexity devices.

[0197] The base station 105 may communicate wirelessly with the UE 115 via one or more base station antennas. The base station 105 described herein may include or may be referred to by those skilled in the art as a base station transceiver, a radio base station, an access point, a radio transceiver, a NodeB, an eNodeB (eNB), a next generation NodeB or a giga-NodeB (any of which may be referred to as a gNB), a home NodeB, a home eNodeB, or some other suitable term. The wireless communication system 100 may include different types of base stations 105 (e.g., macro cell base stations or small cell base stations). The UE 115 described herein may be able to communicate with various types of base stations 105 and network devices, including macro eNBs, small cell eNBs, gNBs, and relay base stations, among others.

[0198] Each base station 105 may be associated with a particular geographic coverage area 110 in which communications with various UEs 115 are supported. Each base station 105 may provide communications coverage for a respective geographic coverage area 110 via a communications link 125, and the communications link 125 between the base station 105 and the UE 115 may utilize one or more carriers. The communications link 125 shown in the wireless communication system 100 may include uplink transmissions from the UE 115 to the base station 105, or downlink transmissions from the base station 105 to the UE 115. Downlink transmissions may also be referred to as forward link transmissions, and uplink transmissions may also be referred to as reverse link transmissions.

[0199] The geographic coverage area 110 of the base station 105 can be divided into sectors that constitute a portion of the geographic coverage area 110, and each sector can be associated with a cell. For example, each base station 105 can provide communication coverage for a macro cell, a small cell, a hotspot, or other type of cell, or various combinations thereof. In some examples, the base station 105 can be mobile and thus provide communication coverage for a mobile geographic coverage area 110. In some examples, different geographic coverage areas 110 associated with different technologies can overlap, and the same base station 105 or different base stations 105 can support overlapping geographic coverage areas 110 associated with different technologies. The wireless communication system 100 may include, for example, a heterogeneous LTE / LTE-A / LTE-A Pro or NR network, in which different types of base stations 105 provide coverage for various geographic coverage areas 110.

[0200] UE 115 can be dispersed throughout the wireless communication system 100, and each UE 115 can be fixed or mobile. UE 115 can also be referred to as a mobile device, a wireless device, a remote device, a handheld device, or a subscriber device, or some other suitable term, where "device" can also be referred to as a unit, a station, a terminal, or a client. UE 115 can also be a personal electronic device, such as a cellular phone, a personal digital assistant (PDA), a tablet computer, a laptop computer, or a personal computer. In the examples of the present disclosure, UE 115 can be any audio source described in the present disclosure, including a VR head-mounted device, an XR head-mounted device, an AR head-mounted device, a vehicle, a smart phone, a microphone, a microphone array, or any other device including a microphone or capable of sending a captured and / or synthesized audio stream. In some examples, the synthesized audio stream can be an audio stream stored in a memory or previously created or synthesized. In some examples, UE 115 may also refer to a wireless local loop (WLL) station, an Internet of Things (IoT) device, an Internet of Everything (IoE) device, or an MTC device, etc., and the UE may be implemented in various articles such as appliances, vehicles, meters, etc.

[0201] Some UEs 115, such as MTC or IoT devices, may be low-cost or low-complexity devices and may provide automated communication between machines (e.g., via machine-to-machine (M2M) communication). M2M communication or MTC may refer to data communication technology that allows devices to communicate with each other or with a base station 105 without human intervention. In some examples, M2M communication or MTC may include communications from devices that exchange and / or use cryptographic-based privacy data to switch, mask, and / or zero various audio streams and / or audio sources, as described above.

[0202] In some cases, UE 115 may also be able to communicate directly with other UE 115 (e.g., using peer-to-peer (P2P) or device-to-device (D2D) protocols). One or more of a group of UEs 115 utilizing D2D communication may be within the geographic coverage area 110 of the base station 105. Other UEs 115 in the group may be outside the geographic coverage area 110 of the base station 105 or unable to receive transmissions from the base station 105. In some cases, multiple groups of UEs 115 communicating via D2D communication may utilize a one-to-many (1:M) system, in which each UE 115 transmits to each other UE 115 in the group. In some cases, the base station 105 facilitates the scheduling of resources for D2D communication. In other cases, D2D communication is performed between UEs 115 without the involvement of the base station 105.

[0203] The base stations 105 may communicate with the core network 130 and with each other. For example, the base stations 105 may interface with the core network 130 via a backhaul link 132 (e.g., via an S1, N2, N3, or other interface). The base stations 105 may communicate with each other directly (e.g., directly between the base stations 105) or indirectly (e.g., via the core network 130) via a backhaul link 134 (e.g., via an X2, Xn, or other interface).

[0204] In some cases, the wireless communication system 100 can utilize both licensed radio spectrum bands and unlicensed radio spectrum bands. For example, the wireless communication system 100 can adopt licensed assisted access (LAA), unlicensed LTE (LTE-U) radio access technology, or NR technology in unlicensed bands such as the 5GHz ISM band. When operating in an unlicensed radio frequency spectrum band, wireless devices such as base stations 105 and UE 115 can adopt a listen-before-talk (LBT) procedure to ensure that the channel is cleared before sending data. In some cases, operations in unlicensed bands can be based on a combination of carrier aggregation configurations and component carriers operating in licensed bands (e.g., LAA). Operations in unlicensed spectrum can include downlink transmissions, uplink transmissions, peer-to-peer transmissions, or a combination of these. Duplexing in unlicensed spectrum can be based on frequency division duplexing (FDD), time division duplexing (TDD), or a combination of both.

[0205] Other illustrative examples of the present disclosure are described below.

[0206] Example 1A - A device configured to process one or more of a plurality of audio streams, the device comprising: a memory configured to store a plurality of audio streams, each of the audio streams representing a sound field; and one or more processors coupled to the memory and configured to: receive one or more of the plurality of audio streams based on privacy restrictions associated with a password; and generate a corresponding sound field based on the one or more of the plurality of audio streams.

[0207] Example 2A - A device according to Example 1A, wherein the password is a master password associated with an unlimited privacy restriction, and wherein to receive one or more of the multiple audio streams, the one or more processors are further configured to: receive each of the multiple audio streams.

[0208] Example 3A - A device according to Example 1A, wherein the password is a permanent password associated with a conditional privacy restriction, and wherein to receive one or more of the multiple audio streams, the one or more processors are further configured to: receive one or more of the multiple audio streams based on the conditional privacy restriction, wherein the conditional privacy restriction indicates whether one or more of the multiple audio streams is restricted or unrestricted.

[0209] Example 4A - A device according to Example 3A, wherein the one or more processors are further configured to: receive audio metadata that also includes corresponding conditional privacy restrictions, and the corresponding conditional privacy restrictions indicate whether one or more of the multiple audio streams are restricted or unrestricted based on the permanent password.

[0210] Example 5A - The device of Example 3A, wherein the permanent password remains in effect until reset.

[0211] Example 6A - A device according to Example 1A, wherein the password is a temporary password associated with a conditional privacy restriction, and wherein in order to receive one or more of the multiple audio streams, the one or more processors are further configured to: receive one or more of the multiple audio streams based on the conditional privacy restriction, wherein the conditional privacy restriction permission indicates whether one or more of the multiple audio streams is restricted or unrestricted.

[0212] Example 7A - A device according to Example 6A, wherein the one or more processors are further configured to: receive audio metadata that also includes corresponding conditional privacy restrictions, and the corresponding conditional privacy restrictions indicate whether one or more of the multiple audio streams are restricted or unrestricted based on the temporary password.

[0213] Example 8A - The device of Example 6A, wherein the temporary password remains valid for a fixed duration and expires after the fixed duration.

[0214] Example 9A - The device of Example 1A, wherein the privacy constraint comprises a respective gain value associated with a respective audio stream of one or more of the plurality of audio streams.

[0215] Example 10A - The device of Example 1A, wherein the privacy restriction comprises a respective zeroing indication associated with a respective audio stream of one or more of the plurality of audio streams.

[0216] Example 11A - The device of Example 1A, wherein the privacy restriction comprises a respective switching indication associated with a respective audio stream of one or more of the plurality of audio streams.

[0217] Example 12A - The device of Example 1A, wherein the one or more processors are further configured to: receive a password from a host.

[0218] Example 13A - The device of Example 1A, wherein the one or more processors are further configured to: receive a password from a source other than the host.

[0219] Example 14A - The device of Example 1A, wherein the plurality of audio streams comprises audio streams from one or more of an audio capture device or a group of audio capture devices.

[0220] Example 15A - The device of Example 14A, wherein the audio capture device may include one or more of a single microphone and a microphone array.

[0221] Example 16A - The device of Example 14A, wherein the audio capture device may include a mobile device.

[0222] Example 17A - The device of Example 1A, wherein the one or more processors are further configured to: receive multiple audio streams via a wireless link.

[0223] Example 18A - The apparatus of Example 17A, wherein the wireless link is a 5G air interface.

[0224] Example 19A - The apparatus of Example 17A, wherein the wireless link is a Bluetooth interface.

[0225] Example 20A - A device according to any combination of Examples 1A to 19A, wherein the device includes an extended reality head-mounted device.

[0226] Example 21A - A device according to any combination of Examples 1A to 20A, further comprising a head-mounted display configured to present the displayed world.

[0227] Example 22A - A device according to any combination of Examples 1A to 19A, wherein the device comprises a mobile handheld terminal.

[0228] Example 23A - A device according to Example 1A, wherein the one or more processors are configured to: receive audio metadata including capture location information representing a capture location in a displayed world, at which a corresponding audio stream in a plurality of audio streams is captured; determine position information representing the position of the device in the displayed world; select a subset of the plurality of audio streams based on the position information and the capture location information, the subset of the plurality of audio streams not including at least one of the plurality of audio streams; and generate a corresponding sound field based on the subset of the plurality of audio streams.

[0229] Example 1B - A method for processing one or more of a plurality of audio streams, the method comprising: storing a plurality of audio streams by a memory, each of the audio streams representing a sound field; receiving one or more of the plurality of audio streams by one or more processors based on privacy restrictions associated with a password; and generating a corresponding sound field based on one or more of the plurality of audio streams by the one or more processors.

[0230] Example 2B - The method of Example 1B, wherein the password is a master password associated with an unlimited privacy restriction, and wherein receiving one or more of the plurality of audio streams comprises: receiving each of the plurality of audio streams.

[0231] Example 3B - A method according to Example 1B, wherein the password is a permanent password associated with a conditional privacy restriction, and wherein receiving one or more of the multiple audio streams includes: receiving one or more of the multiple audio streams based on the conditional privacy restriction, wherein the conditional privacy restriction indicates whether one or more of the multiple audio streams is restricted or unrestricted.

[0232] Example 4B - The method of Example 3B further includes: receiving audio metadata that also includes corresponding conditional privacy restrictions, the corresponding conditional privacy restrictions indicating whether one or more of the multiple audio streams are restricted or unrestricted based on the permanent password.

[0233] Example 5B - The method of Example 3B, wherein the permanent password remains in effect until reset.

[0234] Example 6B - A method according to Example 1B, wherein the password is a temporary password associated with a conditional privacy restriction, and wherein receiving one or more of the multiple audio streams includes: receiving one or more of the multiple audio streams based on the conditional privacy restriction, wherein the conditional privacy restriction permission indicates whether one or more of the multiple audio streams is restricted or unrestricted.

[0235] Example 7B - The method according to Example 6B also includes: receiving audio metadata that also includes corresponding conditional privacy restrictions, and the corresponding conditional privacy restrictions indicate whether one or more of the multiple audio streams are restricted or unrestricted based on the temporary password.

[0236] Example 8B - The method of Example 6B, wherein the temporary password remains valid for a fixed duration and expires after the fixed duration.

[0237] Example 9B - The method of Example 1B, wherein the privacy constraints include respective gain values ​​associated with respective audio streams of one or more of the plurality of audio streams.

[0238] Example 10B - The method of Example 1B, wherein the privacy restriction comprises a respective zeroing indication associated with a respective audio stream of one or more of the plurality of audio streams.

[0239] Example 11B - The method of Example 1B, wherein the privacy restriction comprises a respective switching indication associated with a respective audio stream of one or more of the plurality of audio streams.

[0240] Example 12B - The method of Example 1B further comprising: receiving a password from a host.

[0241] Example 13B - The method of Example 1B further comprising: receiving a password from a source other than the host.

[0242] Example 14B —The method of Example 1B, wherein the plurality of audio streams includes audio streams from one or more of an audio capture device or a group of audio capture devices.

[0243] Example 15B —The method of Example 14B, wherein the audio capture device may include one or more of a single microphone and a microphone array.

[0244] Example 16B - The method of Example 14B, wherein the audio capture device may include a mobile device.

[0245] Example 17B - The method of Example 1B further comprising: receiving multiple audio streams via a wireless link.

[0246] Example 18B - The method of Example 17B, wherein the wireless link is a 5G air interface.

[0247] Example 19B - The method of Example 17B, wherein the wireless link is a Bluetooth interface.

[0248] Example 20B - A method according to any combination of Examples 1B to 19B, wherein the method is performed via an extended reality head-mounted device.

[0249] Example 21B - The method of any combination of Examples 1B to 20B, further comprising: presenting the displayed world using a head-mounted display.

[0250] Example 22B - A method according to any combination of Examples 1B to 19B, wherein the method is performed by a mobile handheld terminal.

[0251] Example 23B - The method according to Example 1B also includes: receiving audio metadata representing capture location information of a capture location in the displayed world, at which a corresponding audio stream among the multiple audio streams is captured; determining position information representing the position of the device in the displayed world; selecting a subset of the multiple audio streams based on the position information and the capture location information, the subset of the multiple audio streams does not include at least one of the multiple audio streams; and generating a corresponding sound field based on the subset of the multiple audio streams.

[0252] Example 1C - A device configured to process one or more of a plurality of audio streams, the device comprising: a device for storing the plurality of audio streams, each of the audio streams representing a sound field; a device for receiving one or more of the plurality of audio streams based on privacy restrictions associated with a password; and a device for generating a corresponding sound field based on one or more of the plurality of audio streams.

[0253] Example 2C - An apparatus according to Example 1C, wherein the password is a master password associated with an unlimited privacy restriction, and wherein the means for receiving one or more of the plurality of audio streams includes: means for receiving each of the plurality of audio streams.

[0254] Example 3C - A device according to Example 1C, wherein the password is a permanent password associated with a conditional privacy restriction, and wherein the means for receiving one or more of the multiple audio streams includes: means for receiving one or more of the multiple audio streams based on the conditional privacy restriction, wherein the conditional privacy restriction indicates whether one or more of the multiple audio streams is restricted or unrestricted.

[0255] Example 4C - The apparatus of Example 3C further includes: a device for receiving audio metadata that also includes corresponding conditional privacy restrictions, the corresponding conditional privacy restrictions indicating whether one or more of the multiple audio streams are restricted or unrestricted based on the permanent password.

[0256] Example 5C - A device according to Example 3C, wherein the permanent password remains in effect until reset.

[0257] Example 6C - A device according to Example 1C, wherein the password is a temporary password associated with a conditional privacy restriction, and wherein the device for receiving one or more of the multiple audio streams includes: a device for receiving one or more of the multiple audio streams based on the conditional privacy restriction, wherein the conditional privacy restriction permission indicates whether one or more of the multiple audio streams is restricted or unrestricted.

[0258] Example 7C - The apparatus of Example 6C further includes: a device for receiving audio metadata that also includes corresponding conditional privacy restrictions, the corresponding conditional privacy restrictions indicating whether one or more of the multiple audio streams are restricted or unrestricted based on the temporary password.

[0259] Example 8C - The device of Example 6C, wherein the temporary password remains valid for a fixed duration and expires after the fixed duration.

[0260] Example 9C - A device according to Example 1C, wherein the privacy constraint includes a respective gain value associated with a respective audio stream of one or more of the plurality of audio streams.

[0261] Example 10C - The device of Example 1C, wherein the privacy restriction comprises a respective zeroing indication associated with a respective audio stream of one or more of the plurality of audio streams.

[0262] Example 11C - A device according to Example 1C, wherein the privacy restriction includes a respective switching indication associated with a respective audio stream of one or more of the plurality of audio streams.

[0263] Example 12C - The apparatus of Example 1C further comprising: means for receiving a password from a host.

[0264] Example 13C - The apparatus of Example 1C, further comprising: means for receiving a password from a source other than the host.

[0265] Example 14C —The device of Example 1C, wherein the plurality of audio streams comprises audio streams from one or more of an audio capture device or a group of audio capture devices.

[0266] Example 15C —The device of Example 14C, wherein the audio capture device may include one or more of a single microphone and a microphone array.

[0267] Example 16C —The device of Example 14C, wherein the audio capture device may include a mobile device.

[0268] Example 17C - The apparatus of Example 1C further comprising means for receiving a plurality of audio streams via a wireless link.

[0269] Example 18C - The apparatus of Example 17C, wherein the wireless link is a 5G air interface.

[0270] Example 19C - The apparatus of Example 17C, wherein the wireless link is a Bluetooth interface.

[0271] Example 20C - A device according to any combination of Examples 1C to 19C, wherein the device is an extended reality head-mounted device.

[0272] Example 21C - The apparatus of any combination of Examples 1C to 20C, further comprising: means for presenting the displayed world using a head mounted display.

[0273] Example 22C - A device according to any combination of Examples 1C to 19C, wherein the device is a mobile handheld terminal.

[0274] Example 23C - The device of Example 1C further includes: a device for receiving audio metadata including capture location information representing a capture location in the displayed world, at which a corresponding audio stream of the multiple audio streams is captured; a device for determining position information representing the position of the device in the displayed world; a device for selecting a subset of the multiple audio streams based on the position information and the capture location information, the subset of the multiple audio streams not including at least one of the multiple audio streams; and a device for generating a corresponding sound field based on the subset of the multiple audio streams.

[0275] Example 1D - A non-temporary computer-readable storage medium having instructions stored thereon that, when executed, cause one or more processors to: store multiple audio streams, each of the audio streams representing a sound field; receive one or more of the multiple audio streams based on privacy restrictions associated with a password; and generate a corresponding sound field based on one or more of the multiple audio streams.

[0276] Example 2D - A non-transitory computer-readable storage medium according to Example 1D, wherein the password is a master password associated with an unlimited privacy restriction, and wherein in order to receive one or more of the multiple audio streams, the instructions further cause the one or more processors to: receive each of the multiple audio streams.

[0277] Example 3D - A non-transitory computer-readable storage medium according to Example 1D, wherein the password is a permanent password associated with a conditional privacy restriction, and wherein in order to receive one or more of the multiple audio streams, the instructions further cause one or more processors to: receive one or more of the multiple audio streams based on the conditional privacy restriction, wherein the conditional privacy restriction indicates whether one or more of the multiple audio streams is restricted or unrestricted.

[0278] Example 4D - A non-transitory computer-readable storage medium according to Example 3D, wherein the instructions further cause one or more processors to: receive audio metadata that also includes corresponding conditional privacy restrictions, and the corresponding conditional privacy restrictions indicate whether one or more of the multiple audio streams are restricted or unrestricted based on a permanent password.

[0279] Example 5D - The non-transitory computer-readable storage medium of Example 3D, wherein the permanent password remains in effect until reset.

[0280] Example 6D - A non-transitory computer-readable storage medium according to Example 1D, wherein the password is a temporary password associated with a conditional privacy restriction, and wherein in order to receive one or more of a plurality of audio streams, the instructions further cause one or more processors to: receive one or more of the plurality of audio streams based on the conditional privacy restriction, wherein the conditional privacy restriction permission indicates whether one or more of the plurality of audio streams is restricted or unrestricted.

[0281] Example 7D - A non-transitory computer-readable storage medium according to Example 6D, wherein the instructions further cause one or more processors to: receive audio metadata that also includes corresponding conditional privacy restrictions, and the corresponding conditional privacy restrictions indicate whether one or more of the multiple audio streams are restricted or unrestricted based on the temporary password.

[0282] Example 8D - The non-transitory computer-readable storage medium of Example 6D, wherein the temporary password remains valid for a fixed duration and expires after the fixed duration.

[0283] Example 9D - The non-transitory computer-readable storage medium of Example 1D, wherein the privacy constraints include respective gain values ​​associated with respective audio streams of one or more of the plurality of audio streams.

[0284] Example 10D - The non-transitory computer-readable storage medium of Example 1D, wherein the privacy restrictions include respective zeroing indications associated with respective audio streams of one or more of the plurality of audio streams.

[0285] Example 11D - The non-transitory computer-readable storage medium of Example 1D, wherein the privacy restriction comprises a respective switching indication associated with a respective audio stream of one or more of the plurality of audio streams.

[0286] Example 12D - The non-transitory computer-readable storage medium of Example 1D, wherein the instructions further cause the one or more processors to: receive a password from a host.

[0287] Example 13D - The non-transitory computer-readable storage medium of Example 1D, wherein the instructions further cause the one or more processors to: receive a password from a source other than the host.

[0288] Example 14D The non-transitory computer-readable storage medium of Example 1D, wherein the plurality of audio streams includes audio streams from one or more of an audio capture device or a group of audio capture devices.

[0289] Example 15D - The non-transitory computer-readable storage medium of Example 14D, wherein the audio capture device may include one or more of a single microphone and a microphone array.

[0290] Example 16D - The non-transitory computer-readable storage medium of Example 14D, wherein the audio capture device may include a mobile device.

[0291] Example 17D - The non-transitory computer-readable storage medium of Example 1D, wherein the instructions further cause the one or more processors to: receive multiple audio streams via a wireless link.

[0292] Example 18D - The non-transitory computer-readable storage medium of Example 17D, wherein the wireless link is a 5G air interface.

[0293] Example 19D - The non-transitory computer-readable storage medium of Example 17D, wherein the wireless link is a Bluetooth interface.

[0294] Example 20D - A non-transitory computer-readable storage medium according to any combination of Examples 1D to 19D, wherein the one or more processors are part of an extended reality head-mounted device.

[0295] Example 21D A non-transitory computer-readable storage medium according to any combination of Examples 1D to 20D, wherein the one or more processors are part of a head-mounted display configured to present a displayed world.

[0296] Example 22D - The non-transitory computer-readable storage medium of any combination of Examples 1D to 19D, wherein the one or more processors are part of a mobile handheld terminal.

[0297] Example 23D - A non-transitory computer-readable storage medium according to Example 1D, wherein the instructions further cause the one or more processors to: receive audio metadata including capture location information representing a capture location in the displayed world, at which a corresponding audio stream in the multiple audio streams is captured; determine position information representing the position of the device in the displayed world; select a subset of the multiple audio streams based on the position information and the capture location information, the subset of the multiple audio streams does not include at least one of the multiple audio streams; and generate a corresponding sound field based on the subset of the multiple audio streams.

[0298] Example 1E - A device configured to send multiple audio streams, the device comprising: a memory configured to store multiple audio streams, each of the audio streams representing a sound field; and one or more processors coupled to the memory and configured to: generate one or more of the multiple audio streams based on privacy restrictions associated with a password.

[0299] Example 2E - The device of Example 1E, wherein the one or more processors are further configured to: send one or more of the plurality of audio streams to a content consumer device.

[0300] Example 3E - A device according to Example 1E, wherein the password is a master password associated with an unlimited privacy restriction, and wherein one or more of the multiple audio streams are generated, and the one or more processors are further configured to: generate each of the multiple audio streams.

[0301] Example 4E - A device according to Example 1E, wherein the password is a permanent password associated with a conditional privacy restriction, and wherein to generate one or more of the multiple audio streams, the one or more processors are further configured to: generate one or more of the multiple audio streams based on the conditional privacy restriction, wherein the conditional privacy restriction indicates whether one or more of the multiple audio streams is restricted or unrestricted.

[0302] Example 5E - A device according to Example 4E, wherein the one or more processors are further configured to: generate audio metadata that also includes corresponding conditional privacy restrictions, and the corresponding conditional privacy restrictions indicate whether one or more of the multiple audio streams are restricted or unrestricted based on the permanent password.

[0303] Example 6E - A device according to Example 4E, wherein the permanent password remains in effect until reset.

[0304] Example 7E - A device according to Example 1E, wherein the password is a temporary password associated with a conditional privacy restriction, and wherein to generate one or more of the multiple audio streams, the one or more processors are further configured to: generate one or more of the multiple audio streams based on the conditional privacy restriction, wherein the conditional privacy restriction permission indicates whether one or more of the multiple audio streams is restricted or unrestricted.

[0305] Example 8E - A device according to Example 7E, wherein the one or more processors are further configured to: generate audio metadata that also includes corresponding conditional privacy restrictions, and the corresponding conditional privacy restrictions indicate whether one or more of the multiple audio streams are restricted or unrestricted based on the temporary password.

[0306] Example 9E - The device of Example 7E, wherein the temporary password remains valid for a fixed duration and expires after the fixed duration.

[0307] Example 10E - A device according to Example 1E, wherein the privacy constraints include respective gain values ​​associated with respective audio streams of one or more of the plurality of audio streams.

[0308] Example 11E - The device of Example 1E, wherein the privacy restriction comprises a respective zeroing indication associated with a respective audio stream of one or more of the plurality of audio streams.

[0309] Example 12E - A device according to Example 1E, wherein the privacy restriction includes a respective switching indication associated with a respective audio stream of one or more of the plurality of audio streams.

[0310] Example 13E - The device of Example 1E, wherein the one or more processors are further configured to: generate a password.

[0311] Example 14E - The device of Example 1E, wherein the one or more processors are further configured to: receive a password.

[0312] Example 15E - The device of Example 1E, wherein the plurality of audio streams comprises audio streams from one or more of an audio capture device or a group of audio capture devices.

[0313] Example 16E —The device of Example 15E, wherein the audio capture device may include one or more of a single microphone and a microphone array.

[0314] Example 17E —The device of Example 15E, wherein the audio capture device may include a mobile device.

[0315] Example 18E - The device of Example 1E, wherein the one or more processors are further configured to: send multiple audio streams over the wireless link.

[0316] Example 19E - The apparatus of Example 18E, wherein the wireless link is a 5G air interface.

[0317] Example 20E - The apparatus of Example 18E, wherein the wireless link is a Bluetooth interface.

[0318] Example 21E - A device according to any combination of Examples 1E to 20E, wherein the device comprises a content capture device.

[0319] Example 22E - A device according to any combination of Examples 1E to 21E, wherein the device comprises a mobile handheld terminal.

[0320] Example 1F - A method for processing one or more of a plurality of audio streams, the method comprising: storing the plurality of audio streams by a memory, each of the audio streams representing a sound field; and generating one or more of the plurality of audio streams by one or more processors based on privacy restrictions associated with a password.

[0321] Example 2F - The method of Example 1F further comprising sending one or more of the plurality of audio streams to a content consumer device.

[0322] Example 3F —The method of Example 1F, wherein the password is a master password associated with an unlimited privacy restriction, and wherein generating one or more of the plurality of audio streams comprises generating each of the plurality of audio streams.

[0323] Example 4F - A method according to Example 1F, wherein the password is a permanent password associated with a conditional privacy restriction, and wherein generating one or more of the multiple audio streams includes: generating one or more of the multiple audio streams based on the conditional privacy restriction, wherein the conditional privacy restriction indicates whether one or more of the multiple audio streams is restricted or unrestricted.

[0324] Example 5F - The method according to Example 4F also includes: generating audio metadata that also includes corresponding conditional privacy restrictions, and the corresponding conditional privacy restrictions indicate whether one or more of the multiple audio streams are restricted or unrestricted based on the permanent password.

[0325] Example 6F - The method of Example 4F, wherein the permanent password remains in effect until reset.

[0326] Example 7F - A method according to Example 1F, wherein the password is a temporary password associated with a conditional privacy restriction, and wherein generating one or more of the multiple audio streams includes: generating one or more of the multiple audio streams based on the conditional privacy restriction, wherein the conditional privacy restriction permission indicates whether one or more of the multiple audio streams is restricted or unrestricted.

[0327] Example 8F - A method according to Example 7F, further comprising: generating audio metadata that also includes corresponding conditional privacy restrictions, and the corresponding conditional privacy restrictions indicate whether one or more of the multiple audio streams are restricted or unrestricted based on the temporary password.

[0328] Example 9F - The method of Example 7F, wherein the temporary password remains valid for a fixed duration and expires after the fixed duration.

[0329] Example 10F —The method of Example 1F, wherein the privacy constraints include respective gain values ​​associated with respective audio streams of one or more of the plurality of audio streams.

[0330] Example 11F - The method of Example 1F, wherein the privacy restrictions include respective zeroing indications associated with respective audio streams of one or more of the plurality of audio streams.

[0331] Example 12F —The method of Example 1F, wherein the privacy restriction comprises a respective switching indication associated with a respective audio stream of one or more of the plurality of audio streams.

[0332] Example 13F - The method according to Example 1F also includes: generating a password.

[0333] Example 14F - The method according to Example 1F also includes: receiving a password.

[0334] Example 15F —The method of Example 1F, wherein the plurality of audio streams includes audio streams from one or more of an audio capture device or a group of audio capture devices.

[0335] Example 16F —The method of Example 15F, wherein the audio capture device may include one or more of a single microphone and a microphone array.

[0336] Example 17F —The method of Example 15F, wherein the audio capture device may include a mobile device.

[0337] Example 18F - The method according to Example 1F also includes: sending multiple audio streams via a wireless link.

[0338] Example 19F —The method of Example 18F, wherein the wireless link is a 5G air interface.

[0339] Example 20F —The method of Example 18F, wherein the wireless link is a Bluetooth interface.

[0340] Example 21F - A method according to any combination of Examples 1F to 20F, wherein the method is performed by a content capture device.

[0341] Example 22F - A method according to any combination of Examples 1F to 21F, wherein the method is performed by a mobile handheld terminal.

[0342] Example 1G - A device configured to process one or more of a plurality of audio streams, the device comprising: means for storing the plurality of audio streams, each of the audio streams representing a sound field; and means for generating one or more of the plurality of audio streams based on privacy restrictions associated with a password.

[0343] Example 2G - The apparatus of Example 1G further comprising means for sending one or more of the plurality of audio streams to a content consumer device.

[0344] Example 3G - The apparatus of Example 1G, wherein the password is a master password associated with an unrestricted privacy restriction, and wherein the means for generating one or more of the plurality of audio streams comprises: means for generating each of the plurality of audio streams.

[0345] Example 4G - A device according to Example 1G, wherein the password is a permanent password associated with a conditional privacy restriction, and wherein the device for generating one or more of the multiple audio streams includes: a device for generating one or more of the multiple audio streams based on the conditional privacy restriction, wherein the conditional privacy restriction indicates whether one or more of the multiple audio streams is restricted or unrestricted.

[0346] Example 5G - The apparatus according to Example 4G further includes: a device for generating audio metadata that also includes corresponding conditional privacy restrictions, wherein the corresponding conditional privacy restrictions indicate whether one or more of the multiple audio streams are restricted or unrestricted based on a permanent password.

[0347] Example 6G - The device of Example 4G, wherein the permanent password remains in effect until reset.

[0348] Example 7G - A device according to Example 1G, wherein the password is a temporary password associated with a conditional privacy restriction, and wherein the device for generating one or more of the multiple audio streams includes: a device for generating one or more of the multiple audio streams based on the conditional privacy restriction, wherein the conditional privacy restriction permission indicates whether one or more of the multiple audio streams is restricted or unrestricted.

[0349] Example 8G - The apparatus of Example 7G further includes: a device for generating audio metadata that also includes corresponding conditional privacy restrictions, wherein the corresponding conditional privacy restrictions indicate whether one or more of the multiple audio streams are restricted or unrestricted based on the temporary password.

[0350] Example 9G - The device of Example 7G, wherein the temporary password remains valid for a fixed duration and expires after the fixed duration.

[0351] Example 10G - The device of Example 1G, wherein the privacy constraint comprises a respective gain value associated with a respective audio stream of one or more of the plurality of audio streams.

[0352] Example 11G - The device of Example 1G, wherein the privacy restriction comprises a respective zeroing indication associated with a respective audio stream of one or more of the plurality of audio streams.

[0353] Example 12G - The apparatus of Example 1G, wherein the privacy restriction comprises a respective switching indication associated with a respective audio stream of one or more of the plurality of audio streams.

[0354] Example 13G - The apparatus of Example 1G further comprising: a device for generating a password.

[0355] Example 14G - The apparatus of Example 1G further comprising: means for receiving a password.

[0356] Example 15G —The device of Example 1G, wherein the plurality of audio streams comprises audio streams from one or more of an audio capture device or a group of audio capture devices.

[0357] Example 16G —The device of Example 15G, wherein the audio capture device may include one or more of a single microphone and a microphone array.

[0358] Example 17G —The device of Example 15G, wherein the audio capture device may include a mobile device.

[0359] Example 18G - The apparatus of Example 1G further comprising: means for transmitting multiple audio streams via a wireless link.

[0360] Example 19G - The apparatus of Example 18G, wherein the wireless link is a 5G air interface.

[0361] Example 20G - The apparatus of Example 18G, wherein the wireless link is a Bluetooth interface.

[0362] Example 21G —A device according to any combination of Examples 1G to 20G, wherein the device is a content capture device.

[0363] Example 22G - A device according to any combination of Examples 1C to 21G, wherein the device is a mobile handheld terminal.

[0364] Example 1H - A non-transitory computer-readable storage medium having instructions stored thereon that, when executed, cause one or more processors to: store multiple audio streams, each of the audio streams representing a sound field; and generate one or more of the multiple audio streams based on privacy restrictions associated with a password.

[0365] Example 2H - The non-transitory computer-readable storage medium of Example 1H, wherein the instructions further cause the one or more processors to: send one or more of the plurality of audio streams to a content consumer device.

[0366] Example 3H - A non-transitory computer-readable storage medium according to Example 1H, wherein the password is a master password associated with an unrestricted privacy restriction, and wherein in order to generate one or more of the multiple audio streams, the instructions also cause the one or more processors to: generate each of the multiple audio streams.

[0367] Example 4H - A non-transitory computer-readable storage medium according to Example 1H, wherein the password is a permanent password associated with a conditional privacy restriction, and wherein to generate one or more of the multiple audio streams, the instructions further cause one or more processors to: generate one or more of the multiple audio streams based on the conditional privacy restriction, wherein the conditional privacy restriction indicates whether one or more of the multiple audio streams is restricted or unrestricted.

[0368] Example 5H - A non-temporary computer-readable storage medium according to Example 4H, wherein the instructions further cause one or more processors to: generate audio metadata that also includes corresponding conditional privacy restrictions, and the corresponding conditional privacy restrictions indicate whether one or more of the multiple audio streams are restricted or unrestricted based on a permanent password.

[0369] Example 6H - The non-transitory computer-readable storage medium of Example 4H, wherein the permanent password remains in effect until reset.

[0370] Example 7H - A non-transitory computer-readable storage medium according to Example 1H, wherein the password is a temporary password associated with a conditional privacy restriction, and wherein in order to generate one or more of the multiple audio streams, the instructions also cause one or more processors to: generate one or more of the multiple audio streams based on the conditional privacy restriction, wherein the conditional privacy restriction permission indicates whether one or more of the multiple audio streams is restricted or unrestricted.

[0371] Example 8H - A non-temporary computer-readable storage medium according to Example 7H, wherein the instructions further cause one or more processors to: generate audio metadata that also includes corresponding conditional privacy restrictions, and the corresponding conditional privacy restrictions indicate whether one or more of the multiple audio streams are restricted or unrestricted based on the temporary password.

[0372] Example 9H - The non-transitory computer-readable storage medium of Example 7H, wherein the temporary password remains valid for a fixed duration and expires after the fixed duration.

[0373] Example 10H - The non-transitory computer-readable storage medium of Example 1H, wherein the privacy constraints include respective gain values ​​associated with respective audio streams of one or more of the plurality of audio streams.

[0374] Example 11H - The non-transitory computer-readable storage medium of Example 1H, wherein the privacy restrictions include respective zeroing indications associated with respective audio streams of one or more of the plurality of audio streams.

[0375] Example 12H - The non-transitory computer-readable storage medium of Example 1H, wherein the privacy restrictions include respective switching indications associated with respective audio streams of one or more of the plurality of audio streams.

[0376] Example 13H - The non-transitory computer-readable storage medium of Example 1H, wherein the instructions further cause one or more processors to: generate a password.

[0377] Example 14H - The non-transitory computer-readable storage medium of Example 1H, wherein the instructions further cause one or more processors to: receive a password.

[0378] Example 15H - The non-transitory computer-readable storage medium of Example 1H, wherein the plurality of audio streams comprises audio streams from one or more of an audio capture device or a group of audio capture devices.

[0379] Example 16H - The non-transitory computer-readable storage medium of Example 15H, wherein the audio capture device may include one or more of a single microphone and a microphone array.

[0380] Example 17H ​​- The non-transitory computer-readable storage medium of Example 15H, wherein the audio capture device may include a mobile device.

[0381] Example 18H - The non-transitory computer-readable storage medium of Example 1H, wherein the instructions further cause one or more processors to: send multiple audio streams over a wireless link.

[0382] Example 19H - A non-transitory computer-readable storage medium as described in Example 18H, wherein the wireless link is a 5G air interface.

[0383] Example 20H - The non-transitory computer-readable storage medium of Example 18H, wherein the wireless link is a Bluetooth interface.

[0384] Example 21H - A non-transitory computer-readable storage medium as described in any combination of Examples 1H to 20H, wherein the one or more processors are part of a content consumer device.

[0385] Example 22H - A non-transitory computer-readable storage medium as described in any combination of Examples 1H to 21H, wherein the one or more processors are part of a mobile handheld terminal.

[0386] It should be appreciated that, depending on the example, certain actions or events of any technique described herein may be performed in a different order, may be added, combined, or excluded entirely (e.g., not all described actions or events are required to practice the technique). In addition, in some examples, actions or events may be performed simultaneously, e.g., via multithreading, interrupt handling, or multiple processor execution, rather than sequentially.

[0387] In some examples, a VR device (or streaming device) can use a network interface coupled to a memory of the VR / streaming device to transmit an exchange message to an external device, wherein the exchange message is associated with multiple available representations of a sound field. In some examples, the VR device can use an antenna coupled to the network interface to receive a wireless signal, including data packets, audio packets, video protocols, or transmission protocol data associated with multiple available representations of a sound field. In some examples, one or more microphone arrays can capture the sound field.

[0388] In some examples, the multiple available representations of the sound field stored to the memory device may include multiple object-based representations of the sound field, a higher-order surround sound representation of the sound field, a mixed-order surround sound representation of the sound field, a combination of the object-based representation of the sound field and the higher-order surround sound representation of the sound field, a combination of the object-based representation of the sound field and the mixed-order surround sound representation of the sound field, or a combination of the mixed-order representation of the sound field and the higher-order surround sound representation of the sound field.

[0389] In some examples, one or more of the sound field representations in multiple available representations of the sound field may include at least one high resolution region and at least one lower resolution region, and wherein the selected presentation based on the steering angle provides greater spatial accuracy with respect to the at least one high resolution region and lower spatial accuracy with respect to the lower resolution region.

[0390] In one or more examples, the described functions can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions can be stored as one or more instructions or codes on a computer-readable medium or sent via a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media (which corresponds to tangible media such as data storage media) or communication media, which includes any media that facilitates, for example, transferring a computer program from one place to another according to a communication protocol. In this way, computer-readable media generally can correspond to (1) non-temporary tangible computer-readable storage media, or (2) communication media (such as, signals or carrier waves). Data storage media can be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, codes, and / or data structures for implementing the techniques described in the present disclosure. A computer program product may include a computer-readable medium.

[0391] By way of example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage media or other magnetic storage devices, flash memory, or any other medium that can be used to carry or store desired program codes in the form of instructions or data structures and can be accessed by a computer. Moreover, any connection is appropriately referred to as a computer-readable medium. For example, if instructions are sent from a website, server or other remote source using coaxial cable, optical cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio and microwave, coaxial cable, optical fiber cable, twisted pair, DSL, or wireless technologies such as infrared, radio and microwave are included in the definition of the medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals or other temporary media, but are directed to non-temporary, tangible storage media. Disks and optical disks as used herein include compact discs (CDs), laser optical disks, optical optical disks, digital versatile discs (DVDs), floppy disks, and blue-ray discs, wherein disks generally reproduce data magnetically, and optical disks reproduce data optically by means of lasers. The above combinations should also be included in the scope of computer-readable media.

[0392] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuit systems. Therefore, the term "processor" as used herein may refer to any of the aforementioned structures or any other structure suitable for implementing the techniques described herein. In addition, in some aspects, the functionality described herein may be provided in dedicated hardware and / or software modules configured for encoding and decoding or incorporated in a combined codec. Moreover, the technology may be fully implemented in one or more circuits or logic elements.

[0393] The techniques of the present disclosure may be implemented in a wide variety of devices or apparatuses, including wireless handsets, integrated circuits (ICs), or a set of ICs (e.g., chipsets). Various components, modules, or units are described in the present disclosure to emphasize functional aspects of devices configured to perform the disclosed techniques, but do not necessarily need to be implemented by different hardware units. Rather, as described above, the various units may be combined in a codec hardware unit, or provided by a collection of interoperable hardware units (including one or more processors as described above) in combination with appropriate software and / or firmware.

[0394] Various examples have been described. These and other examples are within the scope of the following claims.

Claims

1. A device configured to process one or more audio streams, the device comprising: modem; Memory; as well as one or more processors coupled to the memory and the modem, and wherein the one or more processors are configured to: sending a password to a host device via the modem; receiving, from the host device via the modem, an audio stream with unrestricted access and an audio stream with restricted access from the one or more audio streams, wherein the one or more audio streams are from audio elements represented in an acoustic environment, the acoustic environment comprising one or more subsets of an acoustic space, each of the one or more audio streams representing a respective sound field; receiving data indicating restricted playback of the audio stream with restricted access based on privacy restrictions associated with the password; The audio stream with unrestricted access and the audio stream with restricted access are rendered for playback to reconstruct corresponding sound fields, wherein rendering and / or playback of the corresponding sound field of the audio stream with restricted access is restricted based on data indicating the restricted playback.

2. The device of claim 1, wherein the modem is configured to: sending a data packet via a wireless link, the data packet including a representation of the password; and Different data packets are received, the different data packets comprising representations of the audio stream with unrestricted access and the audio stream with restricted access.

3. The device of claim 1 , wherein to restrict playback of the corresponding sound field of the audio stream with restricted access based on the data indicating the restricted playback, the one or more processors are further configured to: Mask, switch or zero the audio stream with restricted access.

4. The apparatus of claim 1 , wherein to restrict playback of the corresponding sound field of the audio stream with restricted access based on the data indicating the restricted playback, the one or more processors are further configured to: An acoustic occluder is applied to the audio stream with restricted access.

5. The device of claim 1 , wherein the one or more processors are further configured to: The password is associated with data indicative of the restricted playback.

6. The device of claim 5, wherein the modem is configured to: One or more data packets are received, the one or more data packets including representations of the audio stream with unrestricted access among the one or more audio streams and the audio stream with restricted access among the one or more audio streams.

7. The device of claim 5, wherein to restrict playback of the corresponding sound field of the audio stream with restricted access based on data indicating restricted playback associated with the password, the one or more processors are further configured to: Mask, switch or zero the audio stream with restricted access.

8. The device of claim 5, wherein to restrict playback of the corresponding sound field of the audio stream with restricted access based on data indicating restricted playback associated with the password, the one or more processors are further configured to: An acoustic occluder is applied to the audio stream with restricted access.

9. The device of claim 1 , wherein the password is a master password associated with an unrestricted privacy restriction, and wherein to receive the audio stream with unrestricted access and the audio stream with restricted access of the one or more audio streams, the one or more processors are further configured to: Each of the one or more audio streams is received.

10. The device of claim 1 , wherein the password is a permanent password associated with a conditional privacy restriction, and wherein to receive the audio stream with unrestricted access and the audio stream with restricted access of the one or more audio streams, the one or more processors are further configured to: The audio stream with unrestricted access and the audio stream with restricted access of the one or more audio streams are received based on a conditional privacy restriction associated with the permanent password, wherein the permanent password remains in effect until reset.

11. The apparatus of claim 10, wherein the conditional privacy restriction is associated with one or more audio elements, one or more groups of audio elements, or one or more corresponding subsets of the acoustic space in one or more subsets of the acoustic space.

12. The device of claim 1 , wherein the password is a temporary password associated with a conditional privacy restriction, and wherein to receive the audio stream with unrestricted access and the audio stream with restricted access of the one or more audio streams, the one or more processors are further configured to: The audio stream with unrestricted access and the audio stream with restricted access of the one or more audio streams are received based on a conditional privacy restriction associated with the temporary password, wherein the temporary password remains valid for a fixed duration and expires after the fixed duration.

13. The apparatus of claim 12, wherein the conditional privacy restriction is associated with one or more audio elements, one or more groups of audio elements, or one or more corresponding subsets of the acoustic space in one or more subsets of the acoustic space.

14. The device of claim 1, wherein the one or more processors are further configured to: The password is received from the host.

15. The apparatus of claim 1, further comprising: An audio output is configured to be coupled to headphones for playback of the one or more audio streams via a wired connection or a wireless connection.

16. The apparatus of claim 1, further comprising: A headset configured to playback the one or more audio streams.

17. The device of claim 1, wherein the device is one of: Virtual reality VR client devices; Augmented reality AR client devices; Mixed reality client device; or Any other device capable of tracking head movements and / or general translational movements of the individual operating the device.

18. A method for processing one or more audio streams, the method comprising: Send the password to the host device; receiving, from the host device, an audio stream with unrestricted access and an audio stream with restricted access from the one or more audio streams, wherein the one or more audio streams are from audio elements represented in an acoustic environment, the acoustic environment comprising one or more subsets of an acoustic space, each of the one or more audio streams representing a respective sound field; receiving data indicating restricted playback of the audio stream with restricted access based on privacy restrictions associated with the password; The audio stream with unrestricted access and the audio stream with restricted access are rendered for playback to reconstruct corresponding sound fields, wherein rendering and / or playback of the corresponding sound field of the audio stream with restricted access is restricted based on data indicating the restricted playback.

19. The method of claim 18, wherein limiting playback of the corresponding sound field of the audio stream with restricted access based on the data indicating the restricted playback comprises: Mask, switch or zero the audio stream with restricted access.

20. The method of claim 18, wherein limiting playback of the corresponding sound field of the audio stream with restricted access based on the data indicating the restricted playback comprises: An acoustic occluder is applied to the audio stream with restricted access.

21. The method of claim 18, further comprising: The password is associated with data indicative of the restricted playback.

22. The method of claim 21, wherein limiting playback of the corresponding sound field of the audio stream with restricted access based on data indicating restricted playback associated with the password comprises: Switching the audio stream to have restricted access.

23. The method of claim 21, wherein limiting playback of a corresponding sound field having the restricted-access audio stream based on data indicating restricted playback associated with the password comprises: An acoustic occluder is applied to the audio stream with restricted access.

24. The method of claim 18, wherein the password is a permanent password associated with a conditional privacy restriction, and wherein receiving the audio stream with unrestricted access and the audio stream with restricted access of the one or more audio streams comprises: The audio stream with unrestricted access and the audio stream with restricted access of the one or more audio streams are received based on a conditional privacy restriction associated with the permanent password, wherein the permanent password remains in effect until reset.

25. The method of claim 24, wherein the conditional privacy restriction is associated with one or more audio elements, one or more groups of audio elements, or one or more corresponding subsets of the acoustic space in one or more subsets of the acoustic space.

26. The method of claim 18, wherein the password is a temporary password associated with a conditional privacy restriction, and wherein receiving the audio stream with unrestricted access and the audio stream with restricted access of the one or more audio streams comprises: The audio stream with unrestricted access and the audio stream with restricted access of the one or more audio streams are received based on a conditional privacy restriction associated with the temporary password, wherein the temporary password remains valid for a fixed duration and expires after the fixed duration.

27. The method of claim 26, wherein the conditional privacy restriction is associated with one or more audio elements, one or more groups of audio elements, or one or more corresponding subsets of the acoustic space in one or more subsets of the acoustic space.

28. The method of claim 26, further comprising: The password is received from the host.

29. A non-transitory computer-readable storage medium storing instructions, which, when executed by a processor, cause the processor to perform the method according to any one of claims 18-28.

Images