Priority-based sound field codec for virtual reality audio

Abstract

An example device configured to obtain image data includes a memory configured to store one or more priority values, each of the one or more priority values ​​being associated with a type of image objects associated with the image data. The device includes one or more processors coupled to the memory and configured to associate image objects in the image data with one or more audio sources represented in one or more audio streams. The one or more processors are further configured to assign a corresponding priority value to each of the one or more audio sources represented in the one or more audio streams, and to encode and decode ambisonic reverberation coefficients based on the assigned priority values.

Description

[0001] This application claims priority to U.S. Application No. 16 / 951,636, filed on November 18, 2020, which claims the benefit of U.S. Provisional Application No. 62 / 939,470, filed on November 22, 2019, the entire contents of each application are incorporated herein by reference. Technical Field

[0002] The present disclosure relates to the processing of media data, such as audio data. Background Art

[0003] Computer-mediated reality systems are being developed to allow computing devices to augment or add to, remove or subtract from, or generally modify the existing reality that a user experiences. Computer-mediated reality systems (which may also be referred to as "extended reality systems" or "XR systems") may include, for example, virtual reality (VR) systems, augmented reality (AR) systems, and mixed reality (MR) systems. The perceived success of a computer-mediated reality system is often related to the ability of such computer-mediated reality system to provide a realistic immersive experience in terms of video and audio experiences that are tuned in a manner desired by the user. Although the human visual system is more sensitive than the human auditory system (e.g., in terms of perceived positioning of various objects in a scene), ensuring an adequate auditory experience is an increasingly important factor in ensuring a realistic immersive experience, particularly as improvements in the video experience allow for better positioning of video objects, thereby enabling users to better identify the source of audio content. Summary of the invention

[0004] The present disclosure generally relates to auditory aspects of user experience of computer-mediated reality systems, including virtual reality (VR), mixed reality (MR), augmented reality (AR), computer vision and graphics systems. Various aspects of the technology can provide adaptive audio capture and rendering for the acoustic space of an extended reality system.

[0005] In one example, various aspects of the technology relate to a device configured to obtain image data, including: a memory configured to store one or more priority values, each of the one or more priority values ​​being associated with a type of image objects associated with the image data; and one or more processors coupled to the memory and configured to: associate image objects in the image data with one or more audio sources represented in one or more audio streams; assign a corresponding priority value to each of the one or more audio sources represented in the one or more audio streams; and encode and decode ambisonic reverberation coefficients based on the assigned priority values.

[0006] In another example, various aspects of the technology are directed to a method of obtaining image data, comprising: storing, by a memory, one or more priority values, each of the one or more priority values ​​being associated with a type of an image object associated with the image data; associating, by one or more processors, the image objects in the image data with one or more audio sources represented in one or more audio streams; assigning, by the one or more processors, a corresponding priority value to each of the one or more audio sources represented in the one or more audio streams; and encoding, by the one or more processors, ambisonic reverberation coefficients based on the assigned priority values.

[0007] In another example, various aspects of the technology relate to a device configured to obtain image data, including: means for storing one or more priority values, each of the one or more priority values ​​being associated with a type of image objects associated with the image data; means for associating the image objects in the image data with one or more audio sources represented in one or more audio streams; means for assigning a corresponding priority value to each of the one or more audio sources represented in the one or more audio streams; and means for encoding and decoding ambisonic reverberation coefficients based on the assigned priority values.

[0008] In another example, various aspects of the technology relate to a non-transitory computer-readable storage medium having instructions stored thereon that, when executed, cause one or more processors to: store one or more priority values, each of the one or more priority values ​​being associated with a type of image objects associated with image data; associate the image objects in the image data with one or more audio sources represented in one or more audio streams; assign a respective priority value to each of the one or more audio sources represented in the one or more audio streams; and encode and decode ambisonic reverberation coefficients based on the assigned priority values.

[0009] The details of one or more examples of the disclosure are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of various aspects of the technology will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figures 1A to 1C Diagram showing a system that can perform various aspects of the techniques described in this disclosure.

[0011] Figure 2 is a diagram showing an example of a VR device worn by a user.

[0012] Figure 3 An example of a wireless communication system 100 is shown that supports devices and methods according to aspects of the present disclosure.

[0013] FIG. 4A to FIG. 4C is a block diagram of a device illustrating various aspects of the techniques described in this disclosure.

[0014] Figure 5 is a flow chart illustrating an example prioritization technique of the present disclosure.

[0015] Figure 6 is a diagram illustrating an example of a wearable device that can operate according to various aspects of the technology described in this disclosure.

[0016] Fig. 7A and Figure 7B is a diagram illustrating other example systems that can perform various aspects of the techniques described in this disclosure.

[0017] Figure 8 It is shown Figures 1A to 1C 00106] A block diagram of example components of one or more of a source device and a content consumer device as shown in an example of FIG. DETAILED DESCRIPTION

[0018] Audio decoding does not consider visual scene elements to determine the priority of decoding one or more audio streams. For example, audio streams associated with various visual elements may all have the same priority. This may lead to wasteful use of processing energy (e.g., computing), bandwidth, and battery life by prioritizing unimportant audio streams at the same level as audio streams that are of greater interest to the user or content creator of the audio decoding device.

[0019] According to the technology of the present disclosure, the image object in the image data can be associated with one or more audio sources represented in one or more audio streams, and a priority value can be assigned to one or more audio sources represented in one or more audio streams. In some examples, different numbers of ambisonic coefficients can be encoded and decoded based on the assigned priority values. For example, one or more audio streams associated with a person speaking near a user can have a higher priority value assigned to them than one or more audio streams associated with a tree. In this way, the technology of the present disclosure provides a way to save processing energy, bandwidth, and battery life by prioritizing audio streams based on associated visual objects. As used herein, encoding and decoding can mean encoding or decoding.

[0020] There are many different ways to represent a sound field. Example formats include channel-based audio formats, object-based audio formats, and scene-based audio formats. Channel-based audio formats refer to 5.1 surround sound formats, 7.1 surround sound formats, 22.2 surround sound formats, or any other channel-based format that positions audio channels to specific locations around the listener in order to recreate a sound field.

[0021] An object-based audio format may refer to a format in which audio objects, typically encoded using pulse code modulation (PCM) and referred to as PCM audio objects, are specified to represent a sound field. Such audio objects may include information (such as metadata) identifying the position of the audio object relative to a listener or other reference point in the sound field, so that the audio object can be rendered to one or more speaker channels for playback in an effort to recreate the sound field. The techniques described in the present disclosure may be applied to any of the aforementioned formats, including scene-based audio formats, channel-based audio formats, object-based audio formats, or any combination thereof.

[0022] A scene-based audio format may include a hierarchical set of objects that define a sound field in three dimensions. An example of a hierarchical set of objects is a set of spherical harmonic coefficients (SHCs). The following expression shows a description or representation of a sound field using SHCs:

[0023]

[0024] This expression shows that at any point in the sound field at time t The pressure at p i Can be provided by SHC Uniquely represents. Here, c is the speed of sound (~343 m / s), is the reference point (or observation point), j n (·) is a spherical Bessel function of order n, and are the spherical harmonic basis functions of order n and suborder m (also referred to as spherical basis functions). It can be appreciated that the term in the square brackets is the frequency domain representation of the signal (i.e., S ), which can be approximated by various time-frequency transforms, such as discrete Fourier transform (DFT), discrete cosine transform (DCT) or wavelet transform. Other examples of hierarchical sets include sets of wavelet transform coefficients and other sets of coefficients of multiresolution basis functions.

[0025] SHC can be physically acquired (e.g., recorded) by various microphone array configurations, or alternatively, they can be derived from a channel-based or object-based description of the sound field. SHC (which may also be referred to as ambisonic coefficients) represent scene-based audio, where the SHC can be input to an audio encoder to obtain encoded SHC that can facilitate more efficient transmission or storage. For example, a method involving (1+4) 2 (25, and hence fourth order) coefficients.

[0026] As mentioned above, SHC can be derived from microphone recordings using microphone arrays. Poletti, M., "Three-Dimensional Surround Sound Systems Based on Spherical Harmonics", J. Audio Eng. Soc., Vol. 53, No. 11, pp. 1004-1025, November 2005, describes various examples of how to physically obtain SHC from microphone arrays.

[0027] The following equation may show how the SHC may be derived from an object-based description. The coefficients for the sound field corresponding to each audio object It can be expressed as:

[0028]

[0029] where i is h n (2) (·) is a spherical Hankel function (of the second kind) of order n, and is the position of the object. Converting each PCM object and the corresponding position to SHC can be accomplished by knowing the source energy of the object as a function of frequency g(ω) (e.g., using time-frequency analysis techniques such as Fast Fourier Transformation of a PCM stream of pulse code modulated data). Furthermore, it is possible to show (since the above is a linear and orthogonal decomposition) the coefficients for each object are added. In this way, multiple PCM objects can be represented by coefficients (e.g., as the sum of coefficient vectors for a single object). The coefficients may contain information about the sound field (pressure as a function of 3D coordinates), and the above representation is at the observation point The transformation from a single object to a representation of the entire sound field is nearby.

[0030] Computer-mediated reality systems (which may also be referred to as "extended reality systems" or "XR systems") are being developed to take advantage of many of the potential benefits provided by ambisonic coefficients. For example, ambisonic coefficients may represent a three-dimensional sound field in a manner that potentially enables accurate three-dimensional (3D) positioning of sound sources within the sound field. Thus, an XR device may render the ambisonic coefficients to a speaker feed that accurately reproduces the sound field when played through one or more speakers.

[0031] As another example, ambisonic coefficients may be translated (e.g., rotated) to account for user motion without overly complex mathematical operations, thereby potentially accommodating the low latency requirements of XR. Additionally, ambisonic coefficients are layered, and thereby naturally accommodate scalability through order reduction (which may eliminate ambisonic coefficients associated with higher orders), and thereby potentially enable dynamic adaptation of the sound field to accommodate the latency and / or battery requirements of an XR device.

[0032] Using ambisonics coefficients for XR can enable the development of multiple use cases that rely on the more immersive sound field provided by ambisonics coefficients (especially for computer gaming applications and live video streaming applications). In these highly dynamic use cases that rely on low-latency reproduction of the sound field, XR devices can prefer ambisonics coefficients over other representations that are more difficult to manipulate or involve complex rendering. Figures 1A to 1C Provides more information about these use cases.

[0033] Although described in the present disclosure with respect to VR devices, various aspects of the technology may be performed in the context of other devices such as mobile devices. In this case, a mobile device (such as a so-called smart phone) may present a displayed world (which may be associated with an acoustic space) via a screen that may be mounted to the head of the user 102 or viewed as is done when using the mobile device normally. Thus, any information on the screen may be part of the mobile device. The mobile device may be able to provide tracking information 41 and thereby allow both a VR experience (when head-mounted) and a normal experience to view the displayed world, where the normal experience may still allow the user to view the displayed world, thereby demonstrating a VR-lite experience (e.g., lifting the device and rotating or translating the device to view different parts of the displayed world). Additionally, although a displayed world is mentioned in various examples of the present disclosure, the technology of the present disclosure may also be used with an acoustic space that does not correspond to a displayed world or in which a displayed world does not exist.

[0034] Figures 1A to 1C is a diagram illustrating a system that can perform various aspects of the techniques described in this disclosure. Figure 1AAs shown in the example of , system 10 includes a source device 12 and a content consumer device 14. Although described in the context of source device 12 and content consumer device 14, these techniques may be implemented in any context in which any representation of a sound field is encoded to form a bitstream representing audio data. Moreover, source device 12 may represent any form of computing device capable of generating a sound field representation, and is generally described herein in the context of being a VR content creator device. Similarly, content consumer device 14 may represent any form of computing device capable of implementing the rendering techniques described in this disclosure as well as audio playback, and is generally described herein in the context of being a VR client device.

[0035] Source device 12 may be operated by an entertainment company or other content creator, which may generate multi-channel audio content for consumption by an operator of a content consumer device, such as content consumer device 14. In some VR scenarios, source device 12 generates audio content in conjunction with video content. Source device 12 includes content capture device 20, content editing device 22, and sound field representation generator 24. Content capture device 20 may be configured to interface with microphone 18 or otherwise communicate.

[0036] Microphone 18 can indicate Or other types of 3D audio microphones that can capture a sound field and represent it as audio data 19, which can refer to one or more of the scene-based audio data (such as ambisonic reverberation coefficients), object-based audio data, and channel-based audio data mentioned above. Although described as a 3D audio microphone, microphone 18 can also represent other types of microphones configured to capture audio data 19 (such as omnidirectional microphones, point microphones, unidirectional microphones, etc.).

[0037] In some examples, the content capture device 20 may include an integrated microphone 18 integrated into the housing of the content capture device 20. The content capture device 20 may interface with the microphone 18 wirelessly or via a wired connection. Instead of or in conjunction with capturing audio data 19 via the microphone 18, the content capture device 20 may process the audio data 19 after the audio data 19 is input via some type of removable storage, wireless, and / or wired input process. Thus, various combinations of content capture devices 20 and microphones 18 are possible in accordance with the present disclosure.

[0038] The content capture device 20 may also be configured to interface or otherwise communicate with a content editing device 22. In some cases, the content capture device 20 may include a content editing device 22 (which, in some cases, may represent software or a combination of software and hardware, including software executed by the content capture device 20 to configure the content capture device 20 to perform a particular form of content editing). The content editing device 22 may represent a unit configured to edit or otherwise change the content 21 (including audio data 19) received from the content capture device 20. The content editing device 22 may output the edited content 23 and associated audio information 25 (e.g., metadata) to the sound field representation generator 24.

[0039] The sound field representation generator 24 may include any type of hardware device capable of interfacing with the content editing device 22 (or the content capturing device 20). Figure 1A , but the sound field representation generator 24 can generate one or more bitstreams 27 using the edited content 23 including the audio data 19 and the audio information 25 provided by the content editing device 22. Figure 1A In the example of , the sound field representation generator 24 may generate one or more representations of the same sound field represented by the audio data 19 to obtain a bitstream 27 including a representation of the edited content 23 and the audio information 25 .

[0040] For example, in order to generate different representations of the sound field using the ambisonics coefficients (which is also an example of audio data 19), the sound field representation generator 24 can use a codec scheme for the ambisonics representation of the sound field, which is called mixed-order ambisonics (MOA), as discussed in more detail in U.S. patent application serial number 15 / 672,058, filed on August 8, 2017 and published on January 3, 2019, U.S. Patent Publication No. 20190007781, and entitled “MIXED-ORDER AMBISONICS (MOA) AUDIO DATA FOCOMPUTER-MEDIATED REALITY SYSTEMS.”

[0041] To generate a particular MOA representation of a sound field, the sound field representation generator 24 may generate a partial subset of a complete set of ambisonic reverberation coefficients. For example, each MOA representation generated by the sound field representation generator 24 may provide accuracy with respect to some areas of the sound field, but provide lower accuracy in other areas. In one example, the MOA representation of a sound field may include eight (8) uncompressed ambisonic reverberation coefficients, while a third-order ambisonic reverberation representation of the same sound field may include sixteen (16) uncompressed ambisonic reverberation coefficients. Thus, each MOA representation of a sound field generated as a partial subset of ambisonic reverberation coefficients may be less storage-intensive and less bandwidth-intensive (if and when transmitted as part of the bitstream 27 over the transmission channel shown) than a corresponding third-order ambisonic reverberation representation of the same sound field generated from the ambisonic reverberation coefficients.

[0042] Although described with respect to MOA representation, the techniques of the present disclosure may also be performed with respect to first-order ambisonic (FOA) representation, where all ambisonic coefficients associated with first-order spherical basis functions and zero-order spherical basis functions are used to represent the sound field. In other words, the sound field representation generator 24 may use all ambisonic coefficients of a given Nth order to represent the sound field, rather than using a partial non-zero subset of the ambisonic coefficients to represent the sound field, resulting in a total of (N+1) 2 The ambisonic reverberation coefficient.

[0043] In this regard, the ambisonic reverberation audio data (which is another way of referring to ambisonic reverberation coefficients in an MOA representation or a full-order representation (such as the first-order representation mentioned above)) may include ambisonic reverberation coefficients associated with spherical basis functions of order 1 or less (which may be referred to as "first-order ambisonic reverberation audio data"), ambisonic reverberation coefficients associated with spherical basis functions having mixed orders and sub-orders (which may be referred to as the "MOA representation" discussed above), or ambisonic reverberation coefficients associated with spherical basis functions of order greater than 1 (which may be referred to as the "full-order representation" as above).

[0044] In some examples, the sound field representation generator 24 may represent an audio encoder configured to compress or otherwise reduce the number of bits used to represent the content 21 in the bitstream 27. Although not shown, in some examples, the sound field representation generator may include a psychoacoustic audio encoding device that complies with any of the various standards discussed herein.

[0045] In this example, the sound field representation generator 24 may apply SVD to the ambisonic reverberation coefficients to determine a decomposed version of the ambisonic reverberation coefficients. The decomposed version of the ambisonic reverberation coefficients may include one or more of the primary audio signals and one or more corresponding spatial components describing the spatial characteristics (e.g., direction, shape, and width) of the associated primary audio signals. Thus, the sound field representation generator 24 may apply decomposition to the ambisonic reverberation coefficients to decouple the energy (as represented by the primary audio signals) from the spatial characteristics (as represented by the spatial components).

[0046] The sound field representation generator 24 may analyze the decomposed version of the ambisonic reverberation coefficients to identify various parameters, which may facilitate reordering of the decomposed version of the ambisonic reverberation coefficients. The sound field representation generator 24 may reorder the decomposed version of the ambisonic reverberation coefficients based on the identified parameters, wherein assuming that a transform may reorder the ambisonic reverberation coefficients across a frame of ambisonic reverberation coefficients (where a frame typically includes M samples of the decomposed version of the ambisonic reverberation coefficients, and in some examples, M is), the reordering may improve codec efficiency.

[0047] After reordering the decomposed versions of the ambisonic reverberation coefficients, the sound field representation generator 24 may select one or more of the decomposed versions of the ambisonic reverberation coefficients as representations of foreground (or in other words, distinct, dominant, or significant) components of the sound field. The sound field representation generator 24 may specify decomposed versions of the ambisonic reverberation coefficients that represent foreground components (which may also be referred to as "primary sound signals," "primary audio signals," or "primary sound components") and associated directional information (which may also be referred to as "spatial components," or in some cases so-called "V-vectors" that identify spatial characteristics of corresponding audio objects). The spatial components may represent vectors (which may be referred to as "coefficients" in terms of vectors) having multiple distinct objects, and may thus be referred to as "multidimensional vectors."

[0048] The sound field representation generator 24 may then perform a sound field analysis on the ambisonic reverberation coefficients to at least partially identify ambisonic reverberation coefficients representing one or more background (or in other words, ambient) components of the sound field. Background components may also be referred to as "background audio signals" or "ambient audio signals". In some examples, assuming that the background audio signal may include only a subset of any given sample of the ambisonic reverberation coefficients (e.g., such as those samples corresponding to zero-order and first-order spherical basis functions, rather than those samples corresponding to second-order or higher-order spherical basis functions), the sound field representation generator 24 may perform energy compensation on the background audio signal. In other words, when performing order reduction, the sound field representation generator 24 may enhance the remaining background ambisonic reverberation coefficients of the ambisonic reverberation coefficients (e.g., add energy to / subtract energy from the remaining background ambisonic reverberation coefficients) to compensate for the change in total energy caused by performing order reduction.

[0049] The sound field representation generator 24 may then perform a form of interpolation on the foreground direction information (which is another way to indicate spatial components), and then perform reduction on the interpolated foreground direction information to generate reduced-order foreground direction information. In some examples, the sound field representation generator 24 may also perform quantization on the reduced-order foreground direction information, thereby outputting the encoded and decoded foreground direction information. In some cases, the quantization may include scalar / entropy quantization, which may be in the form of vector quantization. In some examples, the sound field representation generator 24 may then output the intermediate formatted audio data as the background audio signal, the foreground audio signal, and the quantized foreground direction information to a psychoacoustic audio encoding device.

[0050] In any case, in some examples, the background audio signals and the foreground audio signals may include a transmission channel. That is, the sound field representation generator 24 may output a transmission channel for each frame including the ambisonic reverberation coefficients of a corresponding one of the background audio signals (e.g., M samples of one of the ambisonic reverberation coefficients corresponding to a zero-order or first-order spherical basis function) and for each frame of the foreground audio signal (e.g., M samples of an audio object decomposed from the ambisonic reverberation coefficients). The sound field representation generator 24 may also output side information (which may also be referred to as "sideband information") including quantized spatial components corresponding to each of the foreground audio signals.

[0051] The transmission channel and side information can be Figure 1A In the examples of , ATF audio data may be collectively represented as ambisonics transmission format (ATF) audio data (which is another way of referring to intermediate formatted audio data). In other words, AFT audio data may include transmission channels and side information (which may also be referred to as "metadata"). As an example, ATF audio data may conform to the HOA (High Order Ambisonics) transmission format (HTF). More information about HTF may be found in the technical specification (TS) entitled "Higher Order Ambisonics (HOA) Transport Format" published by the European Telecommunications Standards Institute (ETSI) in June 2018 (2018-06) in ETSI TS 103 589 V1.1.1. Therefore, ATF audio data may be referred to as HTF audio data.

[0052] The sound field representation generator 24 may include dedicated hardware that is configured to (or dedicated software that, when executed, causes one or more processors to) perform psychoacoustic audio coding (such as the Unified Speech and Audio Codec denoted as “USAC”, which is set forth by the Moving Picture Experts Group (MPEG), the MPEG-H 3D Audio Codec standard, the MPEG-I Immersive Audio standard, or a proprietary standard, such as AptX™ (including various versions of AptX, such as Enhanced AptX-E-AptX, AptXlive, AptX Stereo, and AptX High Definition-AptX-HD), Advanced Audio Codec (AAC), Audio Codec 3 (AC-3), Apple Lossless Audio Codec (ALAC), MPEG-4 Audio Lossless Streaming (ALS), Enhanced AC-3, Free Lossless Audio Codec (FLAC), Monkey's Audio, MPEG-1 Audio Layer II (MP2), MPEG-1 Audio Layer III (MP3), Opus, and Microsoft Media Audio (WMA)).

[0053] In examples where the sound field representation generator 24 does not include a psychoacoustic audio encoding device, the sound field representation generator 24 may then send or otherwise output the ATF audio data to a psychoacoustic audio encoding device (not shown). The psychoacoustic audio encoding device may perform psychoacoustic audio encoding on the ATF audio data to generate a bitstream 27. The source device 12 may then send the bitstream 27 to the content consumer device 14 via a transmission channel.

[0054] In some examples, the psychoacoustic audio coding device can represent one or more instances of psychoacoustic audio codec, each of which is used to encode the transmission channel of ATF audio data. In some cases, this psychoacoustic audio coding device can represent one or more instances of (as mentioned above) coding unit. In some cases, the psychoacoustic audio codec unit can call the instance of the coding unit of each transmission channel for ATF audio data.

[0055] In some examples, the content capture device 20 or the content editing device 22 may be configured to wirelessly communicate with the sound field representation generator 24. In some examples, the content capture device 20 or the content editing device 22 may communicate with the sound field representation generator 24 via one or both of a wireless connection or a wired connection. Via the connection between the content capture device 20 and the sound field representation generator 24, the content capture device 20 may provide content in various content forms, which for discussion purposes are described herein as part of the audio data 19.

[0056] In some examples, the content capture device 20 may utilize aspects of the sound field representation generator 24 (in terms of hardware or software capabilities of the sound field representation generator 24). For example, the sound field representation generator 24 may include dedicated hardware configured to perform psychoacoustic audio coding (or dedicated software that, when executed, causes one or more processors to perform psychoacoustic audio coding).

[0057] In some examples, the content capture device 20 may not include psychoacoustic audio encoder dedicated hardware or specialized software, but instead may provide audio aspects of the content 21 in the form of a non-psychoacoustic audio codec. The sound field representation generator 24 may assist in the capture of the content 21 by performing psychoacoustic audio encoding at least in part with respect to the audio aspects of the content 21.

[0058] The sound field representation generator 24 (in the case where the audio data 19 includes scene-based audio data) may also assist in content capture and transmission by generating one or more bitstreams 27 based at least in part on audio content (e.g., MOA representations and / or third-order ambisonic representations) generated from the audio data 19. The bitstream 27 may represent a compressed version of the audio data 19 and any other different types of content 21 (such as compressed versions of spherical video data, image data, or text data).

[0059] As an example, the sound field representation generator 24 may generate a bitstream 27 for transmission across a transmission channel, which may be a wired or wireless channel, a data storage device, etc. The bitstream 27 may represent an encoded version of the audio data 19 and may include a main bitstream and another side bitstream, which may be referred to as side channel information or metadata. In some cases, the bitstream 27 representing a compressed version of the audio data 19 (which may also represent scene-based audio data, object-based audio data, channel-based audio data, or a combination thereof) may conform to a bitstream generated according to the MPEG-H 3D Audio Codec standard and / or the MPEG-I Immersive Audio standard.

[0060] The content consumer device 14 may be operated by an individual and may represent a VR client device. Although described with respect to a VR client device, the content consumer device 14 may represent other types of devices, such as an augmented reality (AR) client device, a mixed reality (MR) client device (or other XR client device), a standard computer, a head mounted device, a headset, a mobile device (including so-called smartphones), or any other device capable of tracking the head movement and / or general translational movement of the individual operating the content consumer device 14. Figure 1AAs shown in the example of , content consumer device 14 includes audio playback system 16A, which may refer to any form of audio playback system capable of rendering audio data for playback as multi-channel audio content.

[0061] Although in Figure 1A 2 is shown as being sent directly to content consumer device 14, but source device 12A may output bitstream 27 to an intermediate device located between source device 12A and content consumer device 14A. The intermediate device may store bitstream 27 for later delivery to content consumer device 14A requesting bitstream 27. The intermediate device may include a file server, a network server, a desktop computer, a laptop computer, a tablet computer, a mobile phone, a smart phone, or any other device capable of storing bitstream 27 for later retrieval by an audio decoder. The intermediate device may reside in a content delivery network that is capable of streaming bitstream 27 to subscribers such as content consumer device 14 requesting bitstream 27 (and possibly in conjunction with sending a corresponding video data bitstream).

[0062] Alternatively, source device 12A may store bitstream 27 to a storage medium such as a compact disk, a digital video disk, a high-definition video disk, or other storage medium, most of which are capable of being read by a computer and may therefore be referred to as a computer-readable storage medium or a non-transitory computer-readable storage medium. In this context, a transmission channel may refer to a channel through which content stored on the medium (e.g., in the form of one or more bitstreams 27) is transmitted (and may include retail stores and other storage-based delivery mechanisms). Therefore, the technology of the present disclosure should not be limited in this regard in any way. Figure 1A .

[0063] As described above, the content consumer device 14 includes an audio playback system 16A. The audio playback system 16A can represent any system capable of playing back multi-channel audio data. The audio playback system 16A can include a plurality of different renderers 32. Each audio renderer 32 can provide different forms of rendering, where the rendering of different audio forms can include one or more of various ways of performing basis vector amplitude phase shifting (VBAP), and / or one or more of various ways of performing sound field synthesis. As used herein, "A and / or B" means "A or B", or "both A and B".

[0064] The audio playback system 16A may also include an audio decoding device 34. The audio decoding device 34 may represent a device configured to decode the bitstream 27 to output audio data 19' (where the apostrophe notation may represent that the audio data 19' is different from the audio data 19 due to lossy compression of the audio data 19, such as quantization). Again, the audio data 19' may include scene-based audio data, which in some examples may form a complete first (or higher) order ambisonic representation or a subset thereof forming a MOA representation of the same sound field, a decomposition thereof (such as a primary audio signal, ambisonic coefficients, and vector-based signals described in the MPEG-H 3D Audio codec standard), or other forms of scene-based audio data.

[0065] Other forms of scene-based audio data include audio data defined according to the HOA (Higher Order Ambisonics) Transport Format (HTF). More information about HTF can be found in the European Telecommunications Standards Institute (ETSI) Technical Specification (TS) entitled “Higher Order Ambisonics (HOA) Transport Format” in ETSI TS 103589 V1.1.1 dated June 2018 (2018-06), and in U.S. Patent Publication No. 2019 / 0918028 entitled “PRIORITY INFORMATION FOR HIGHER ORDER AMBISONIC AUDIO DATA” filed on December 20, 2018. In any case, the audio data 19' may be similar to a complete set or a partial subset of the audio data 19', but may differ due to lossy operations (e.g., quantization) and / or transmission via a transmission channel.

[0066] The audio data 19' may include channel-based audio data as an alternative to or in combination with scene-based audio data. The audio data 19' may include object-based audio data as an alternative to or in combination with scene-based audio data. Thus, the audio data 19' may include any combination of scene-based audio data, object-based audio data, and channel-based audio data.

[0067] The audio renderer 32 of the audio playback system 16A may render the audio data 19' to output the speaker feed 35 after the audio decoding device 34 has decoded the bitstream 27 to obtain the audio data 19'. The speaker feed 35 may drive one or more speakers (for ease of illustration, Figure 1AVarious audio representations including scene-based audio data (and possibly channel-based audio data and / or object-based audio data) of the sound field may be normalized in a variety of ways including N3D, SN3D, FuMa, N2D, or SN2D.

[0068] To select an appropriate renderer, or in some cases, to generate an appropriate renderer, the audio playback system 16A may obtain speaker information 37 indicating a plurality of speakers (e.g., loudspeakers or headphone speakers) and / or the spatial geometry of the speakers. In some cases, the audio playback system 16A may obtain the speaker information 37 using a reference microphone and may drive the speaker (which may refer to the output of an electrical signal to cause the transducer to vibrate) in a manner that dynamically determines the speaker information 37. In other cases, or in conjunction with the dynamic determination of the speaker information 37, the audio playback system 16A may prompt a user to interface with the audio playback system 16A and input the speaker information 37.

[0069] The audio playback system 16A may select one of the audio renderers 32 based on the speaker information 37. In some cases, the audio playback system 16A may generate one of the audio renderers 32 based on the speaker information 37 when none of the audio renderers 32 is within a certain threshold similarity metric (in terms of speaker geometry) of the speaker geometry specified in the speaker information 37. In some cases, the audio playback system 16A may generate one of the audio renderers 32 based on the speaker information 37 without first attempting to select an existing one of the audio renderers 32.

[0070] When the speaker feeds 35 are output to the headphones, the audio playback system 16A may utilize one of the renderers 32 that provides binaural rendering using a head-related transfer function (HRTF) or other function capable of rendering the left and right speaker feeds 35 for playback by the headphone speakers, such as a binaural room impulse response renderer. The term "speaker" or "transducer" may generally refer to any speaker, including a loudspeaker, a headphone speaker, a bone conduction speaker, an earbud speaker, a wireless headphone speaker, etc. One or more speakers may then play back the rendered speaker feeds 35 to reproduce the sound field.

[0071] Although described as rendering the speaker feeds 35 from the audio data 19', references to rendering of the speaker feeds 35 may refer to other types of rendering, such as rendering incorporated directly into the decoding of the audio data from the bitstream 27. Examples of alternative rendering may be found in Annex G of the MPEG-H 3D Audio standard, where rendering occurs during main signal formation and background signal formation prior to sound field synthesis. References to rendering of the audio data 19' should therefore be understood to refer to both rendering of the actual audio data 19' or a decomposition or representation of the audio data 19' thereof (such as the above-mentioned main audio signal, ambisonics coefficients, and / or vector-based signals - which may also be referred to as V-vectors or as multi-dimensional ambisonics space vectors).

[0072] The audio playback system 16A can also adapt the audio renderer 32 based on the tracking information 41. That is, the audio playback system 16A can be connected to an interface of a tracking device 40, which is configured to track the head movement and possible translational movement of a user of the VR device. The tracking device 40 can represent one or more sensors (e.g., cameras - including depth cameras, gyroscopes, magnetometers, accelerometers, light-emitting diodes - LEDs, etc.), which are configured to track the head movement and possible translational movement of a user of the VR device. The audio playback system 16A can adapt the audio renderer 32 based on the tracking information 41 so that the speaker feed 35 reflects the changes and possible translational movement of the user's head to correctly reproduce the sound field in response to such movement.

[0073] Figure 1C is a block diagram illustrating another example system 60. The example system 60 is similar to Figure 1A 10, but the source device 12B of system 60 does not include a content capture device. Source device 12B includes a synthesis device 29. Content developers can use synthesis device 29 to generate synthesized audio sources. The synthesized audio sources can have position information associated with them, which can identify the position of the audio sources relative to a listener or other reference point in the sound field, so that the audio sources can be rendered to one or more speaker channels for playback in an attempt to reconstruct the sound field. In one example, synthesis device 29 can also synthesize visual or video data.

[0074] For example, a content developer can generate a synthesized audio stream for a video game. Figure 1C The example is compared with Figure 1A An example of a content consumer device 14 is shown together, but Figure 1C The source device 12B of the example may be connected to Figure 1B In some examples, Figure 1CSource device 12B may also include a content capture device, so that bitstream 27 may contain both captured audio stream(s) and synthesized audio stream(s).

[0075] As described above, content consumer device 14A or 14B (either of which may be referred to as content consumer device 14 hereinafter) may represent a VR device in which a human wearable display (which may also be referred to as a “head-mounted display”) is mounted in front of the eyes of a user operating the VR device. Figure 2 is a diagram illustrating an example of a VR device 400 worn by a user 402. The VR device 400 may be used to implement the techniques of the present disclosure. The VR device 400 is coupled to or otherwise includes headphones 404, which may reproduce the sound field represented by the audio data 19' through playback of a speaker feed 35. The speaker feed 35 may represent an analog or digital signal capable of causing a membrane within a transducer of the headphones 404 to vibrate at various frequencies, wherein such a process is generally referred to as driving the headphones 404.

[0076] Video, audio, and other sensory data may play an important role in a VR experience. To participate in a VR experience, a user 402 may wear a VR device 400 (also referred to as a VR head-mounted device 400) or other wearable electronic device. A VR client device (such as a VR head-mounted device 400) may include a tracking device (e.g., a tracking device 40) that is configured to track the head movement of the user 402 and adapt the video data displayed via the VR head-mounted device 400 to account for the head movement, thereby providing an immersive experience in which the user 402 can experience the displayed world displayed in the video data in three visible dimensions. The displayed world may refer to a virtual world (in which all of the world is simulated), an augmented world (in which a portion of the world is augmented by virtual objects), or a physical world (in which real-world images are virtually navigated).

[0077] Although VR (and other forms of AR and / or MR) can allow a user 402 to visually reside in a virtual world, a VR headset 400 may often lack the ability to aurally place the user in the displayed world. In other words, a VR system (which may include a computer responsible for rendering video data and audio data and a VR headset 400, for ease of illustration, the computer in Figure 2 ) may not support full three-dimensional auditory immersion (and in some cases, actually in a manner that reflects the displayed scene presented to the user via the VR head mounted device 400).

[0078] Although described in the present disclosure with respect to VR devices, various aspects of the technology may be performed in the context of other devices such as mobile devices. In this case, a mobile device (such as a so-called smartphone) may present a displayed world via a screen that may be mounted to the head of a user 402 or viewed as is done when using the mobile device normally. Thus, any information on the screen may be part of the mobile device. The mobile device may be able to provide tracking information 41 and thereby allow both a VR experience (when head mounted) and a normal experience to view the displayed world, where the normal experience may still allow the user to view the displayed world, thereby demonstrating a VR lite experience (e.g., lifting the device and rotating or translating the device to view different parts of the displayed world).

[0079] In any case, returning to the context of VR devices, the audio aspects of VR are divided into three separate immersion categories. The first category provides the lowest level of immersion and is called three degrees of freedom (3DOF). 3DOF refers to audio rendering that takes into account the movement of the head in three degrees of freedom (yaw, pitch, and roll), allowing the user to look around freely in any direction. However, 3DOF cannot account for translational head movements in which the head is not centered on the optical and acoustic center of the sound field.

[0080] The second category, called 3DOF plus (3DOF+), provides three degrees of freedom (yaw, pitch, and roll) in addition to limited spatial translational motion caused by the movement of the head away from the optical and acoustic centers within the sound field. 3DOF+ can provide support for perceptual effects such as motion parallax, which can enhance immersion.

[0081] The third category, called six degrees of freedom (6DOF), renders audio data in a way that takes into account three degrees of freedom in terms of head movement (yaw, pitch, and roll) and also takes into account the translation of the user in space (x, y, and z translation). Spatial translation can be caused by sensors that track the user's position in the physical world or by means of input controllers.

[0082] 3DOF rendering is the existing technology for the audio aspects of VR. As a result, the audio aspects of VR are less immersive than the video aspects, potentially reducing the overall immersion of the user experience. However, VR is rapidly transforming and could quickly evolve to support both 3DOF+ and 6DOF, which could provide opportunities for other use cases.

[0083] For example, interactive gaming applications can utilize 6DOF to facilitate fully immersive gaming, where the user moves around the VR world themselves and can interact with virtual objects by walking up to them. Additionally, interactive live streaming applications can utilize 6DOF to allow VR client devices to experience a live stream of a concert or sporting event as if they were at the concert themselves, allowing the user to move around the concert or sporting event.

[0084] There are multiple difficulties associated with these use cases. In the case of fully immersive gaming, latency may need to be kept low to enable a gaming experience that does not cause nausea or motion sickness. Additionally, from an audio perspective, audio playback latency that causes loss of sync with the video data may reduce immersion. Furthermore, for certain types of gaming applications, spatial accuracy may be important to allow accurate responses, including with respect to how the user perceives sound, as this allows the user to anticipate actions that are not currently in view.

[0085] In the context of a live streaming application, a large number of source devices 12A or 12B (either of which may be referred to as source device 12 hereinafter for purposes of brevity) may stream content 21, where source devices 12 may have widely varying capabilities. For example, one source device may be a smartphone with a digital fixed-lens camera and one or more microphones, while another source device may be a production-grade television rig capable of obtaining video at a much higher resolution and quality than a smartphone. However, in the context of a live streaming application, all source devices may provide streams of varying quality, and the VR device may attempt to select an appropriate stream from among the streams to provide the intended experience.

[0086] Figure 3 An example of a wireless communication system 100 supporting devices and methods according to various aspects of the present disclosure is shown. The wireless communication system 100 includes a base station 105, a UE 115, and a core network 130. In some examples, the wireless communication system 100 can be a long term evolution (LTE) network, an advanced LTE (LTE-A) network, an LTE-A Pro network, a fifth generation (5G) cellular network, or a new radio (NR) network. In some cases, the wireless communication system 100 can support enhanced broadband communications, ultra-reliable (e.g., mission-critical) communications, low-latency communications, or communications with low-cost and low-complexity devices.

[0087] The base station 105 may communicate wirelessly with the UE 115 via one or more base station antennas. The base station 105 described herein may include or may be referred to by those skilled in the art as a base transceiver station, a radio base station, an access point, a radio transceiver, a Node B, an eNodeB (eNB), a next generation Node B or a Gigabit Node B (any of which may be referred to as a gNB), a Home Node B, a Home eNodeB, or some other appropriate terminology. The wireless communication system 100 may include different types of base stations 105 (e.g., macro cell base stations or small cell base stations). The UE 115 described herein may be able to communicate with various types of base stations 105 and network equipment (including macro eNBs, small cell eNBs, gNBs, relay base stations, etc.).

[0088] Each base station 105 may be associated with a particular geographic coverage area 110 in which it supports communications with various UEs 115. Each base station 105 may provide communication coverage for a respective geographic coverage area 110 via a communication link 125, and the communication link 125 between the base station 105 and the UE 115 may utilize one or more carriers. The communication link 125 shown in the wireless communication system 100 may include an uplink transmission from the UE 115 to the base station 105, or a downlink transmission from the base station 105 to the UE 115. Downlink transmissions may also be referred to as forward link transmissions, and uplink transmissions may also be referred to as reverse link transmissions.

[0089] The geographic coverage area 110 of the base station 105 can be divided into sectors that constitute a portion of the geographic coverage area 110, and each sector can be associated with a cell. For example, each base station 105 can provide communication coverage for a macro cell, a small cell, a hotspot, or other types of cells, or various combinations thereof. In some examples, the base station 105 can be mobile and thus provide communication coverage for a mobile geographic coverage area 110. In some examples, different geographic coverage areas 110 associated with different technologies can overlap, and overlapping geographic coverage areas 110 associated with different technologies can be supported by the same base station 105 or different base stations 105. The wireless communication system 100 may include, for example, heterogeneous LTE / LTE-A / LTE-A Pro, fifth generation cellular or NR networks, in which different types of base stations 105 provide coverage for various geographic coverage areas 110.

[0090] UE 115 can be dispersed throughout the wireless communication system 100, and each UE 115 can be fixed or mobile.UE 115 can also be referred to as a mobile device, a wireless device, a remote device, a handheld device, or a subscriber device, or some other suitable term, where "device" can also be referred to as a unit, a station, a terminal, or a client.UE 115 can also be a personal electronic device, such as a cellular phone, a personal digital assistant (PDA), a tablet computer, a laptop computer, or a personal computer. In an example of the present disclosure, UE 115 can be any one of the audio sources described in the present disclosure, including a VR head-mounted device, an XR head-mounted device, an AR head-mounted device, a vehicle, a smart phone, a microphone, a microphone array, or any other device including a microphone or capable of sending a captured and / or synthesized audio stream. The captured audio stream and the synthesized audio stream can include one or more audio sources. In some examples, the synthesized audio stream can be an audio stream stored in a memory or previously created or synthesized. In some examples, UE 115 may also refer to a wireless local loop (WLL) station, an Internet of Things (IoT) device, an Internet of Everything (IoE) device, or an MTC device, etc., which may be implemented in various items such as appliances, vehicles, meters, etc.

[0091] Some UEs 115, such as MTC or IoT devices, may be low-cost or low-complexity devices and may provide automated communication between machines (e.g., via machine-to-machine (M2M) communication). M2M communication or MTC may refer to data communication technology that allows devices to communicate with each other or with a base station 105 without human intervention. In some examples, M2M communication or MTC may include communications from devices that exchange and / or use information indicating privacy restrictions and / or password-based privacy data (such as audio metadata) to switch, block, and / or disable various audio streams and / or audio sources, as will be described in more detail below.

[0092] In some cases, UE 115 may also be able to communicate directly with other UE 115 (e.g., using a peer-to-peer (P2P) or device-to-device (D2D) protocol). One or more UE 115 in a group of UEs 115 utilizing D2D communication may be within a geographic coverage area 110 of a base station 105. Other UEs 115 in the group may be outside the geographic coverage area 110 of the base station 105, or may otherwise be unable to receive transmissions from the base station 105. In some cases, a group of UEs 115 communicating via D2D communication may utilize a one-to-many (1:M) system in which each UE 115 transmits to each other UE 115 in the group. In some cases, the base station 105 facilitates the scheduling of resources for D2D communication. In other cases, D2D communication is performed between UEs 115 without involving the base station 105.

[0093] The base stations 105 may communicate with the core network 130 and with each other. For example, the base stations 105 may interface with the core network 130 via a backhaul link 132 (e.g., via S1, N2, N3, or other interface). The base stations 105 may communicate with each other directly (e.g., directly between the base stations 105) or indirectly (e.g., via the core network 130) via a backhaul link 134 (e.g., via X2, Xn, or other interface).

[0094] In some cases, the wireless communication system 100 can utilize both licensed and unlicensed radio spectrum bands. For example, the wireless communication system 100 can adopt license assisted access (LAA), LTE unlicensed (LTE-U) radio access technology, or NR technology in unlicensed bands such as the 5GHz ISM band. When operating in an unlicensed radio spectrum band, wireless devices such as base stations 105 and UE 115 can use a listen-before-talk (LBT) process to ensure that the frequency channel is idle before sending data. In some cases, operations in unlicensed bands can be based on carrier aggregation configurations that combine component carriers (e.g., LAA) operating in licensed bands. Operations in unlicensed spectrum can include downlink transmissions, uplink transmissions, peer-to-peer transmissions, or a combination of these. Duplexing in unlicensed spectrum can be based on frequency division duplexing (FDD), time division duplexing (TDD), or a combination of both.

[0095] Head mounted devices (such as Figure 2A content creator or user 402 of a VR headset 400 may not be equally concerned with all audio objects. For example, some audio sources may be inactive, distant, or just background noise, and the content creator or user 402 may not consider those objects to be important to the scene. The content consumer device 14 may consume computing resources (e.g., processing cycles, memory bandwidth, memory and / or storage space, etc.) decoding less important audio sources.

[0096] According to various aspects of the technology described in the present disclosure, the source device 12 or the content consumer device 14 can prioritize certain audio sources or objects. The content consumer device 14 can then use the prioritization to allocate more resources to decode more important objects and allocate fewer resources to decode less important objects. For example, the content consumer device 14 can encode and decode important audio objects that have more ambisonic reverberation coefficients than less important audio objects. In this way, the source device 12 or the content consumer device 14 can save processing energy, bandwidth, and battery life by prioritizing audio streams based on associated visual objects.

[0097] In operation, source device 12 and content consumer device 14 may analyze image data in a scene and audio sources in an audio stream, and associate image objects in the image data with audio sources in the audio stream. Source device 12 and content consumer device 14 may assign corresponding priority values ​​to each of the image objects and their corresponding audio sources, and encode and decode ambisonic reverberation coefficients based on the assigned priority values. For example, source device 12 or content consumer device 14 may analyze a scene and assign higher priority values ​​to active image objects and lower priority values ​​to inactive image objects. Source device 12 and content consumer device 14 may also associate audio objects with higher priority image objects and with lower priority image objects. Content consumer device 14 may decode higher priority audio objects using more bits or information or higher order ambisonic reverberation than content consumer device 14 uses to decode lower priority audio objects. The following is about Figure 4A and Figure 4B Discuss more about various aspects of prioritization techniques.

[0098] Figure 4A 2 is a block diagram illustrating various aspects of the techniques described in the present disclosure. Image data 250 may be captured by source device 12. For example, source device 12 may capture an image through a camera (such as a Figure 6 A or Figure 6 The camera 600 shown in B captures the image data 250. In some examples, the image data may be computer generated. Figure 4AIn the example of , the image contains a plurality of image objects: an image object 251 of a person standing in the foreground, an image object 252 of a person further away from the camera 600, an image object 253 of two people having a conversation, and an image object 254 of an unoccupied area.

[0099] The scene analyzer and priority classifier 255 receives the image data 250 and analyzes it to identify the image objects contained therein. The scene analyzer and priority classifier 255 can represent hardware or a combination of hardware and software that performs visual scene analysis of video data such as the image data 250. Visual scene analysis can involve various aspects of computer vision, which refers to the process by which a computer or other device can process and analyze an image to detect and identify various objects and / or aspects of the image. In some cases, computer vision can be referred to as machine vision because computer vision and machine vision have many overlapping or related concepts. Typically, machine vision adopts various aspects or concepts of computer vision, but in different contexts. Although the present disclosure involves computer vision when describing the technology, machine vision can also be used in combination with computer vision or as an alternative to computer vision to perform the technology. For this purpose, the terms "machine vision" and "computer vision" can be used interchangeably.

[0100] Despite Figure 4A Not shown in the example of , but in some cases, the scene analyzer and priority classifier 255 can communicate with an image server or other database external to the source device 12 when performing visual scene analysis. The scene analyzer and priority classifier 255 can communicate with the image server to offload various aspects of the visual scene analysis process that are often resource intensive (meaning, processing resources and / or memory resources). For example, the scene analyzer and priority classifier 255 can perform some initial analysis to detect objects, passing these objects to the image server for identification. The image server can then classify or otherwise identify the objects, passing the classification of the classified objects back to the scene analyzer and priority classifier 255. The scene analyzer and priority classifier 255 can communicate with the image server via a wireless session. Thus, the source device 12 may include one or more interfaces (although in Figure 4A ), source device 12 may communicate with peripheral devices, servers, and any other type of device or accessory wirelessly or via a wired connection through this interface. Scene analyzer and priority classifier 255 may output image objects as a result of performing visual scene analysis.

[0101] In some examples, scene analyzer and priority classifier 255 can utilize machine learning to learn to classify or otherwise identify objects in a scene. For example, scene analyzer and priority classifier 255 can utilize a machine learning algorithm to learn to classify objects that scene analyzer and priority classifier 255 passes to an image server.

[0102] According to various aspects of the present disclosure, the scene analyzer and priority classifier 255 can assign corresponding priority values ​​to the identified objects. In some examples, a priority model 256 can be used. For example, the priority model 256 can be stored in a memory as a table. The priority model 256 can associate the type of image object with the priority. For example, the priority model can place a higher priority on certain image objects than other objects. The source device 12 can apply the priority value of the identified object to object information (such as metadata) and / or spatial components (such as V vectors). Although the examples discussed herein refer to V vectors, V vectors are only an example of spatial components, and other examples can be used according to the technology of the present disclosure. The content consumer device 14 can use the prioritized object information and / or prioritized V vectors to render audio objects that may be more important to the content creator with higher quality and render less important audio objects with lower quality. This saves processing power and storage devices than rendering all audio objects with higher quality. The priority model 256 can be predetermined, can be configurable by the content creator, or both.

[0103] In some examples, the content creator can configure the priority model 256 to have 5 levels of priority values ​​1-5, with 5 being the highest priority. For example, the content creator can assign a priority value of 5 to a person, a priority value of 1 to a car, a priority value of 3 to a dog, and a priority value of 1 to a tree.

[0104] In other examples, because image object 253 contains two people talking, you may want to place a higher priority value on the object. Image object 251 contains people who are nearby, but are not currently talking. Because they are nearby, you may want to place a medium priority value on image object 251. Regarding image object 252 (people in the distance) and image object 254 (empty space), you may want to place low priority values ​​on people in the distance and empty space because they may not be important to the content creator or user 402 of the VR head-mounted device 400. Once the scene analyzer and priority classifier 255 analyzes the scene, it can assign a corresponding priority value to each analyzed audio object. For example, the scene analyzer and priority classifier 255 can apply a priority model 256 to prioritize image objects. Although in this example of the priority model 256, as described, there are three priority values ​​discussed (low, medium, and high), any number of priority values ​​can be used in the priority model 256. Additionally, the priority model 256 can focus on things other than people talking. For example, it may focus on animals, motor vehicles, bodies of water, and any other image objects that may appear in the image data 250 .

[0105] The source device 12 may also be configured to Figure 1A , Figure 1B , Figure 6 A and Figure 6 One or more microphones (not shown) of microphone 18 of B capture audio data. Audio scene analyzer 258 can receive audio data 267 and can analyze audio data 267. In some examples, audio data 267 can be synthesized. Audio data 267 may include one or more audio sources represented in one or more audio streams. For example, audio scene analyzer 258 may analyze the audio scene using known sound source separation methods such as blind source separation, independent component analysis (ICA), principal component analysis (PCA), singular value decomposition (SDV), etc. to identify audio objects in audio data 267. Audio objects may refer to different or identifiable sounds that may be classified or otherwise associated with a given object. For example, a motor vehicle engine may emit easily identifiable sounds. Audio scene analyzer 258 may attempt to detect and identify or classify these sounds in audio data 267.

[0106] Similar to the scene analyzer and priority classifier 255, in some cases, the audio scene analyzer 258 can communicate with an audio network server or other database that is external to the source device 12 and may be remote from the source device 12 when performing the auditory scene analysis. The audio scene analyzer 258 can communicate with the audio server to offload various aspects of the auditory scene analysis process that is often resource-intensive (meaning, processing resources and / or memory resources). For example, the audio scene analyzer 258 can perform some initial analysis to detect objects, pass these objects to the audio server for identification. The audio server can then classify or otherwise identify the objects, passing the classified objects back to the audio scene analyzer 258. The audio scene analyzer 258 can communicate with the audio server using the interface mentioned above when describing the scene analyzer and priority classifier 255.

[0107] In one example, the prioritized image objects 251' to 254' and the audio objects within the audio data 267 can each be defined according to a compatible or common format, meaning that the prioritized image objects 251' to 254' and the audio objects within the audio data 267 are defined in a manner that facilitates association therebetween. Each of the prioritized objects 251' to 254' and the objects within the audio data 267 can include audio information such as metadata that, to provide a few examples, defines one or more of the following: a predicted position (e.g., x, y, z coordinates) of the corresponding object, a size (or predicted size) of the corresponding object, a shape (or predicted shape) of the corresponding object, a speed (or predicted speed) of the corresponding object, a position confidence level, and whether the object is in focus or whether the object belongs to the near foreground, far foreground, or near background or far background.

[0108] The audio scene analyzer 258 may determine θ, and / or ψ angles, and can associate audio objects with the θ, The θ, θ and / or ψ angles are provided to the audio and video space mapper 260. In some examples, the audio scene analyzer may provide the audio objects and / or associated θ, θ and ψ angles to the audio and video space mapper 260. and / or ψ angle are provided to the priority applicator 262.

[0109] The audio and video space mapper 260 may receive the prioritized image objects 251' to 254' and the audio objects and map them into space. For example, the audio and video space mapper 260 may utilize the position information associated with the image objects and the θ, The audio and video space mapper 260 may associate one or more of the prioritized image objects 251' to 254' with one or more audio objects in the audio data 267 based on the mapping and / or object type. For example, if the image object is identified as a dog and the audio object is identified as a dog barking and they are mapped to approximately the same area in space, the audio and video space mapper 260 may associate the image object with the audio object.

[0110] More information regarding scene analysis, audio scene analysis, and associating image objects with audio objects can be found in U.S. Patent No. 9,338,420, entitled “Video Analysis Assisted Generation of Multi-channel Audio Data.”

[0111] The audio and video space mapper 260 can provide the priority value of the audio object to the priority applicator 262. In some examples, the priority applicator 262 can receive the priority value of the audio object directly from the scene analyzer and priority classifier 255.

[0112] The priority applicator 262 may apply the assigned priority values ​​to the audio object information and / or V vector to be generated during encoding. For example, the priority applicator may assign a high priority value to an audio object associated with the prioritized image object 253' in the audio data 267, may assign a medium priority value to an audio object associated with the prioritized image object 251', and may assign a priority (e.g., low) to audio objects associated with the prioritized image objects 252' and 254'.

[0113] The video encoder 270 may receive video data from the audio and video space mapper 260 and encode the video data. The audio encoder 265 may receive audio data from the audio and video space mapper 260 and receive a priority value from the priority applicator 262. In some examples, the audio and video space mapper may provide the priority value to the audio encoder 265 instead of the priority applicator 262. In some examples, the audio encoder 265 may encode audio objects differently based on their priority values. In some examples, the audio encoder 265 may encode audio information (such as metadata) indicating the priority value of the audio object. Encoding audio information (such as metadata) indicating the priority value of the audio object may be beneficial to content consumer devices that do not have the ability to perform scene analysis. Encoding audio information (such as metadata) indicating the priority value of the audio object may also be beneficial to content consumer devices that do have the ability to perform scene analysis but have low battery power. In these examples, the content consumer device may use the information indicating the priority value to determine how to decode each object, such as decoding a lower priority audio object using a lower order ambisonic reverb compared to a higher priority audio object without performing scene analysis.

[0114] The audio encoder 265 may also or alternatively encode the audio data itself differently. For example, the audio encoder 265 may encode a higher priority audio object with a higher order ambisonic reverberation compared to a lower priority audio object. The audio encoder 265 may modify the V vector based on the priority values ​​of the audio objects (received from the priority applicator 262) included in the scene so that when the audio encoded signal is decoded, the resulting decoded signal has a variable signal-to-noise ratio in the foreground. For example, the signal-to-noise ratio of an audio object in the foreground has a different signal-to-noise ratio than another audio object in the foreground. The video encoder 270 and the audio encoder 265 may output the encoded video data and the encoded audio data to the bitstream 27.

[0115] In some examples, source device 12 may store bitstream 27 directly into a memory (not shown) without sending bitstream 27. In other examples, source device 12 may not compress the audio data and video data from audio and video spatial mapper 260. For example, the audio and video data from audio and video spatial mapper 260 may be written to one or more buffers (not shown) for playback, such as when source device 12 may be in camcorder mode.

[0116] Figure 4B 2 is a block diagram of a content consumer device illustrating various aspects of the techniques described in this disclosure. A content consumer device 14 may receive a bitstream 27. For example, a bitstream may be received from a Figure 1A , Figure 1B , Figure 1C , Figure 4A , Figure 6 A or Figure 6 Source device 12 shown in B receives bitstream 27. Video decoder 257 may receive encoded video data from bitstream 27 and decode the encoded video data to reproduce, for example, Figure 4A The image data 250 is shown in FIG.

[0117] The scene analyzer and priority classifier 285 receives the image data 250 from the video decoder 257. The scene analyzer and priority classifier 285 can analyze the image data 250 to identify the image objects contained therein. The scene analyzer and priority classifier 285 can also determine the direction information associated with each image object contained therein. The scene analyzer and priority classifier 285 can represent hardware or a combination of hardware and software that performs visual scene analysis of video data such as the image data 250. Visual scene analysis can involve various aspects of computer vision, which refers to a process by which a computer or other device can process and analyze an image to detect and identify various objects and / or aspects of the image. In some cases, computer vision can be referred to as machine vision because computer vision and machine vision have many overlapping or related concepts. For this purpose, the terms "machine vision" and "computer vision" can be used interchangeably.

[0118] although Figure 4B , but in some cases, the scene analyzer and priority classifier 285 may communicate with an image server or other database external to the content consumer device 14 when performing visual scene analysis. The scene analyzer and priority classifier 285 may communicate with the image server to offload various aspects of the visual scene analysis process that are often resource intensive (meaning, processing resources and / or memory resources). For example, the scene analyzer and priority classifier 285 may perform some initial analysis to detect objects, passing those objects to the image server for identification. The image server may then classify or otherwise identify the objects, passing the classified objects back to the scene analyzer and priority classifier 285. The scene analyzer and priority classifier 285 may communicate with the image server via a wireless session. Thus, the content consumer device 14 may include one or more interfaces (although Figure 4B ), content consumer device 14 may communicate with peripherals, servers, and any other type of device or accessory wirelessly or via a wired connection through this interface. Scene analyzer and priority classifier 285 may output image objects as a result of performing visual scene analysis.

[0119] In some examples, scene analyzer and priority classifier 285 can utilize machine learning to learn to classify or otherwise identify objects in a scene. For example, scene analyzer and priority classifier 285 can utilize a machine learning algorithm to learn to classify objects that it passes to an image server.

[0120] According to aspects of the present disclosure, the scene analyzer and priority classifier 285 can assign a corresponding priority value to each image object identified by the scene analyzer and priority classifier 285. In some examples, a priority model 286 is used. For example, the priority model 286 can be stored in a memory as a table. The priority model 286 can associate the type of image object with a priority value. For example, the priority model can place a higher priority value on certain image objects than on other objects. The priority applicator 282 and the sound field reconstructor 275 can use the assigned priority values ​​to render audio objects that may be more important to the user 402 or the content creator with higher quality, and render less important audio objects with lower quality. This saves processing power and storage devices than rendering all audio objects with higher quality. The priority model 286 can be predetermined, can be configurable by a user of the content consumer device 14 (e.g., user 402), or both.

[0121] In some examples, user 402 can configure priority model 286 to have 5 levels of priority values ​​1-5, where 5 is the highest priority value. User 402 can assign a priority value of 5 to a person, a priority value of 1 to a car, a priority value of 3 to a dog, and a priority value of 1 to a tree.

[0122] In other examples, such as Figure 4A As shown, because image object 253 contains two people talking, you may want to place a higher priority value on this object. Image object 251 contains people nearby, but they are not currently talking. Because they are nearby, you may want to place a medium priority value on this image object. Regarding image object 252 (people in the distance) and image object 254 (empty space), you may want to place low priority values ​​on people in the distance and empty space because they may not be very important to the user 402 of the VR head-mounted device 400. Once the scene analyzer and priority classifier 285 analyze the scene, it can apply the priority model 286 to prioritize the image objects. Although in this example of the priority model 286, there are three priority values ​​(low, medium and high) discussed, any number of priority values ​​can be used in the priority model 286. Additionally, the priority model 286 can focus on things other than people talking. For example, it can focus on animals, motor vehicles, bodies of water, and anything else that may appear in the image data 250.

[0123] In some examples, HTF decoder 272 may receive encoded audio data from bitstream 27. Encoded audio data may represent one or more audio sources in one or more audio streams. HTF decoder 272 may decode encoded audio data and provide decoded audio data (e.g., V vector) to direction mapper 274. Direction mapper 274 may also receive prioritized image object information including direction information and priority values ​​associated with prioritized image objects 251' to 254'. Direction mapper 274 may search for which foreground sounds (sounds that appear to come from the front of user 402 rather than from behind) correspond to prioritized image objects 251' to 254'. Direction mapper 274 may attempt to associate audio objects with prioritized image objects, and similarly prioritize audio objects. In some examples, audio objects in the background (sounds that appear to come from behind user 402) may be assigned a predetermined priority. In some examples, the predetermined priority may be the lowest priority. In some examples, audio objects in the background may not be decoded. As discussed above, an audio object can refer to a distinct or recognizable sound that can be categorized or otherwise associated with a given object. For example, a motor vehicle engine can emit an easily recognizable sound. Direction mapper 274 can attempt to detect and identify or categorize these sounds in audio data 267.

[0124] Similar to the scene analyzer and priority classifier 285, in some cases, when performing auditory scene analysis, the direction mapper 274 can communicate with an audio network server or other database that is external to the content consumer device 14 and may be far away from the content consumer device 14. The direction mapper 274 can communicate with the audio server to offload various aspects of the auditory scene analysis process that is often resource-intensive (meaning, processing resources and / or memory resources). For example, the direction mapper 274 can perform some initial analysis to detect objects, pass these objects to the audio server for identification. The audio server can then classify or otherwise identify the objects, passing the classified objects back to the audio mapper. The direction mapper 274 can communicate with the audio server using the interface mentioned above when describing the scene analyzer and priority classifier 285.

[0125] In one example, the prioritized image objects 251' to 254' and the audio objects within the audio data 267 can each be defined according to a compatible or common format, meaning that the prioritized image objects 251' to 254' and the audio objects within the audio data 267 are defined in a manner that facilitates association therebetween. Each of the prioritized objects 251' to 254' and the objects within the audio data 267 can include information such as metadata that, to provide a few examples, defines one or more of the following: a predicted position (e.g., x, y, z coordinates) of the corresponding object, a size (or predicted size) of the corresponding object, a shape (or predicted shape) of the corresponding object, a speed (or predicted speed) of the corresponding object, a position confidence level, and whether the object is in focus or whether the object belongs to the near foreground, the far foreground, or the near background or the far background. The direction mapper 274 can associate one or more prioritized image objects 251' to 254' with one or more audio objects in the audio data 267 based on information such as metadata.

[0126] For example, the directional mapper 274 may examine the V vector associated with the foreground and attempt to match the audio objects with the information in the V vector and assign similar priorities to them. The directional mapper 274 may provide the mapped image objects, V vectors, and priority values ​​to the priority applicator 282. The directional mapper represents hardware or a combination of hardware and software that may attempt to associate the prioritized objects 251' to 254' with the audio objects within the audio data 267. The priority applicator 282 may weight the V vectors of the audio objects based on their priority values ​​or the priority values ​​of their associated image objects.

[0127] The priority applicator 282 can provide the priority values ​​of the audio objects associated with the prioritized image objects (e.g., 251' to 254') to the psychoacoustic decoder 276. The psychoacoustic decoder 276 can receive the encoded audio signal and can use the priority value to determine which quality to decode each representation of the audio source (such as the U signal). Although the U signal is discussed in the examples of this article, other representations of the audio source can be used according to the technology of the present disclosure. As used herein, the representation of an audio source or the representation of multiple audio sources may also be referred to as an audio source or multiple audio sources, respectively. For example, an audio object with a higher priority value can be decoded by the psychoacoustic decoder 276 with higher quality than an audio object with a lower priority value.

[0128] In some examples, such as the examples discussed above, where user 402 configures priority model 286 to have five priority levels, psychoacoustic decoders 276 may base their decoding on both the priority level of a given object and resource availability, such as battery energy or network link strength (such as personal area network link strength). For example: if the battery energy level is less than 50% but greater than 20%, psychoacoustic decoder 276 may only decode objects with a priority of 3 or higher; if the battery energy level is 100%, psychoacoustic decoder 276 may decode all objects; and if the battery energy level is less than or equal to 20%, psychoacoustic decoder 276 may only decode objects with a priority of 5.

[0129] The priority applicator 282 may also modify the V vector based on the priorities of the audio objects included in the scene so that the resulting decoded signal has a variable signal-to-noise ratio in the foreground (e.g., different audio objects have different signal-to-noise ratios), and send the modified V vector to the sound field reconstructor 275.

[0130] The psychoacoustic decoder 276 may provide the M decoded U signals to the sound field reconstructor 275. The sound field reconstructor 275 may receive the weighted V vector from the priority applicator 262. The sound field reconstructor 275 may then reconstruct the sound field based on the U signals and the V vector, outputting the sound field to the renderer 280 for rendering for the user 402.

[0131] In some examples, content consumer device 14 may store bitstream 27 directly into memory (not shown). In other examples, content consumer device 14 may receive an uncompressed bitstream and not decode the video and audio data.

[0132] Figure 4C is a conceptual diagram showing an example concert with three or more audio receivers. Figure 4CIn the example of , multiple musicians are depicted on stage 323. Singer 312 is located behind microphone 310A. String section 314 is depicted behind microphone 310B. Drummer 316 is depicted behind microphone 310C. Other musicians 318 are depicted behind microphone 310D. Microphones 310A to 310D can capture audio streams corresponding to the sounds received by the microphones. In some examples, microphones 310A to 310D can represent synthesized audio streams. For example, microphone 310A can capture (multiple) audio streams associated primarily with singer 312, but the (multiple) audio streams can also include sounds produced by other band members (such as string section 314, drummer 316 or other musicians 318), while microphone 310B can capture (multiple) audio streams associated primarily with string section 314, but the (multiple) audio streams include sounds produced by other band members. In this way, each of microphones 310A to 310D can capture (multiple) different audio streams.

[0133] Multiple devices are also depicted. These devices represent user devices located at multiple different listening positions. Headphones 320 are located near microphone 310A, but between microphones 310A and 310B. Thus, content consumer device 14 can select at least one of the audio streams to produce a sound stream similar to that produced by the user at the user's location. Figure 4C Similarly, VR goggles 322 are shown as being located behind microphone 310C and between drummer 316 and other musicians 318. Content consumer device 14 may select at least one audio stream to produce an audio experience for a user of VR goggles 322 similar to that of a user being located behind microphone 310C. Figure 4C The audio experience of the location where the VR goggles 322 are located.

[0134] Smart glasses 324 are shown as being located fairly centrally between microphones 310A, 310C, and 310D. Content consumer device 14 may select at least one audio stream to produce an audio stream similar to that produced by the user of smart glasses 324. Figure 4C In addition, device 326 (which may represent any device capable of implementing the techniques of this disclosure, such as a mobile handset, a speaker array, headphones, VR goggles, smart glasses, etc.) is shown as being located in front of microphone 310B. Content consumer device 14 selects at least one audio stream to produce an audio experience similar to that of the user at the location of smart glasses 324 in the device 326. Figure 4C Although specific devices are discussed with respect to specific locations, use of any of the devices depicted may provide audio experiences consistent with the location of the device 325. Figure 4C An indication of the different desired listening positions depicted in . Figure 4CAny of the devices may be used to implement the techniques of this disclosure.

[0135] Figure 5 is a flow chart illustrating an example prioritization technique of the present disclosure. Source device 12 or content consumer device 14 may store one or more priority values ​​in memory (350). Each of the one or more priority values ​​is associated with a type of image object associated with the image data. For example, one priority value may be associated with two people talking (e.g., image object 253), and another priority value may be associated with a person far away (e.g., image object 252).

[0136] One or more processors of source device 12 or content consumer device 14 associate 352 the image objects in the image data with one or more audio sources represented in the one or more audio streams. For example, source device 12 or content consumer device 14 may associate an audio stream of engine sounds with a car or an audio stream of a dog barking. In some examples, source device 12 or content consumer device 14 may also store position information for each audio source, such as angles θ and horn.

[0137] In some examples, when associating an image object in the image data with one or more audio sources represented in one or more audio streams, source device 12 or content consumer device 14 may spatially map the one or more audio sources represented in the one or more audio streams to the image object. In some examples, source device 12 or content consumer device 14 may match the spatially mapped audio sources to the V vectors. In some examples, source device 12 or content consumer device 14 may only match the spatially mapped audio sources to the V vectors that are in the foreground (e.g., not behind the user). In some examples, source device 12 or content consumer device 14 may modify the V vectors to differentiate the signal-to-noise ratios of the at least two of the one or more audio sources when encoding and decoding the at least two of the one or more audio streams.

[0138] One or more processors of source device 12 or content consumer device 14 assign a corresponding priority value to each of one or more audio sources represented in one or more audio streams (354). For example, source device 12 or content consumer device 14 may determine a priority associated with an image object and assign the same priority to an audio source associated with the image object. For example, if an image object of a dog has a high priority, an audio source of a dog barking may also be assigned a high priority. Source device 12 or content consumer device 14 may encode or decode (e.g., respectively encode or decode) ambisonic reverberation coefficients based on the assigned priority values ​​(356). For example, for an audio source having a higher assigned priority, source device 12 or content consumer device 14 may use more ambisonic reverberation coefficients to encode or decode (e.g., respectively encode or decode) the audio source compared to an audio source having a lower assigned priority. In some examples, source device 12 or content consumer device 14 may encode or decode more bits for a higher priority value audio source than for a lower priority value audio source. In some examples, source device 12 or content consumer device 14 may modify the V vector to differentiate the signal-to-noise ratio of the at least two of the one or more audio sources when encoding and decoding at least two of the one or more audio streams. In some examples, the encoding and decoding is based on the assigned priority value and the available resources. For example, source device 12 or content consumer device 14 may determine that the battery energy level is low, and source device 12 or content consumer device 14 may encode and decode the ambisonic reverberation coefficients differently than the codec that source device 12 or content consumer device 14 has when the battery energy level is higher. In some examples, content consumer device 14 may combine at least two of the one or more audio sources by at least one of mixing or interpolation.

[0139] Figure 6is a diagram showing an example of a wearable device 500 that can operate according to various aspects of the technology described in the present disclosure. In various examples, the wearable device 500 can represent a VR head-mounted device (such as the VR head-mounted device 400 described above), an AR head-mounted device, an MR head-mounted device, or any other type of extended reality (XR) head-mounted device. Augmented reality "AR" can refer to computer-rendered images or data overlaid on the real world where the user is actually located. Mixed reality "MR" can refer to computer-rendered images or data that are world-locked to a specific location in the real world, or can refer to a variation on VR in which part of the computer-rendered 3D objects and part of the photographed real objects are combined into an immersive experience that simulates the user's physical presence in the environment. Extended reality "XR" can represent VR, AR, MR in general. More information about terminology for XR can be found in Jason Peterson's document entitled "Virtual Reality, Augmented Reality, and Mixed Reality Definitions" and dated July 7, 2017.

[0140] Wearable device 500 may represent other types of devices, such as watches (including so-called “smart watches”), glasses (including so-called “smart glasses”), headphones (including so-called “wireless headphones” and “smart headphones”), smart clothing, smart jewelry, etc. Whether representing a VR device, watch, glasses and / or headphones, wearable device 500 may communicate with a computing device supporting wearable device 500 via a wired connection or a wireless connection.

[0141] In some cases, the computing device supporting the wearable device 500 may be integrated within the wearable device 500, and thus, the wearable device 500 may be considered to be the same device as the computing device supporting the wearable device 500. In other cases, the wearable device 500 may communicate with a separate computing device that may support the wearable device 500. In this regard, the term "support" should not be understood as requiring a separate dedicated device, but rather one or more processors configured to perform various aspects of the techniques described in this disclosure may be integrated within the wearable device 500 or within a computing device separate from the wearable device 500.

[0142] For example, when the wearable device 500 represents the VR device 1100, a separate dedicated computing device (such as a personal computer including one or more processors) can render audio and visual content, and the wearable device 500 can determine the translational head movement, on which the dedicated computing device can render the audio content (as a speaker feed) based on the translational head movement according to various aspects of the technology described in this disclosure. As another example, when the wearable device 500 represents smart glasses, the wearable device 500 can include one or more processors that (via one or more sensor interfaces of the wearable device 500) determine the translational head movement and render the speaker feed based on the determined translational head movement.

[0143] As shown, the wearable device 500 includes one or more directional speakers and one or more tracking and / or recording cameras. In addition, the wearable device 500 includes one or more inertial, tactile and / or health sensors, one or more eye tracking cameras, one or more high-sensitivity audio microphones, and optical devices / projection hardware. The optical devices / projection hardware of the wearable device 500 can include durable translucent display technology and hardware.

[0144] The wearable device 500 also includes connectivity hardware, which may represent one or more network interfaces that support multi-mode connectivity, such as Long Term Evolution (LTE), Advanced LTE (LTE-A), LTE-A Pro, 5G cellular, New Radio (NR), Bluetooth, etc. The wearable device 500 also includes one or more ambient light sensors and bone conduction transducers. In some cases, the wearable device 500 may also include one or more passive and / or active cameras with fisheye lenses and / or telephoto lenses. Although not described in Figure 6 , but the wearable device 500 may also include one or more light emitting diode (LED) lights. In some examples, the (multiple) LED lights may be referred to as "super bright" LED lights. In some implementations, the wearable device 500 may also include one or more rear cameras. It should be understood that the wearable device 500 may exhibit a variety of different form factors.

[0145] Additionally, tracking and recording cameras and other sensors can facilitate the determination of translation distance. Figure 6 Not shown in the example of , the wearable device 500 may include other types of sensors for detecting translation distance.

[0146] Although described with respect to specific examples of wearable devices, such as those described above with respect to Figure 6 The VR device 1100 and Figures 1A to 1C Other devices described in the examples, but those skilled in the art will understand that Figures 1A to 1Cand Figure 2 The related description can be applied to other examples of wearable devices. For example, other wearable devices such as smart glasses can include sensors by which translational head motion is obtained. As another example, other wearable devices such as smart watches can include sensors by which translational motion is obtained. Therefore, the technology described in the present disclosure should not be limited to a specific type of wearable device, but any wearable device can be configured to perform the technology described in the present disclosure.

[0147] Fig. 7A and Figure 7B is a diagram illustrating an example system that can perform various aspects of the techniques described in this disclosure. Fig. 7A Source device 12 is shown to also include an example of a camera 600. Camera 600 may be configured to capture video data and provide the captured raw video data to content capture device 20. Content capture device 20 may provide the video data to another component of source device 12 for further processing into viewport partitioned portions.

[0148] exist Fig. 7A In the example of , content consumer device 14 also includes wearable device 300. It should be understood that in various implementations, wearable device 300 may be included in content consumer device 14 or externally coupled to content consumer device 14. Wearable device 300 includes display hardware and speaker hardware for outputting video data (e.g., as associated with various viewports) and for rendering audio data.

[0149] Figure 7B The binaural renderer 42 is shown in FIG. Fig. 7A , a binaural renderer 42 can perform binaural rendering using one or more HRTFs or other functions capable of rendering left and right speaker feeds 43. The audio playback system 16C can output the left and right speaker feeds 43 to headphones 44.

[0150] The headset 44 may be connected via a wired connection (such as a standard 3.5 mm audio jack, a universal system bus (USB) connection, an optical audio jack, or other form of wired connection) or wirelessly (such as via Bluetooth TM The headset 44 may be coupled to the audio playback system 16C by a wireless connection, a wireless network connection, etc. The headset 44 may recreate the sound field represented by the audio data 19' based on the left and right speaker feeds 43. The headset 44 may include a left and right earphone speaker that is powered (or in other words, driven) by the corresponding left and right speaker feeds 43.

[0151] Figure 8 It is shown Figure 1A , Figure 1Bor Figure 1C A block diagram of example components of one or more of a source device and a content consumer device as shown in the example of FIG. Figure 8 In the example of , device 710 includes a processor 712 (which may be referred to as "one or more processors" or "processor(s)"), a graphics processing unit (GPU) 714, a system memory 716, a display processor 718, one or more integrated speakers 740, a display 703, a user interface 720, an antenna 721, and a transceiver module 722. In examples where device 710 is a mobile device, display processor 718 is a mobile display processor (MDP). In some examples, such as examples where device 710 is a mobile device, processor 712, GPU 714, and display processor 718 may be formed as an integrated circuit (IC).

[0152] For example, an IC can be considered a processing chip within a chip package, and can be a system on a chip (SoC). In some examples, two of processor 712, GPU 714, and display processor 718 can be housed together in the same IC, and the other in a different integrated circuit (i.e., a different chip package), or all three can be housed in different ICs or on the same IC. However, in examples where device 710 is a mobile device, it is possible that processor 712, GPU 714, and display processor 718 are all housed in different integrated circuits.

[0153] Examples of processor 712, GPU 714, and display processor 718 include, but are not limited to, one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Processor 712 may be a central processing unit (CPU) of device 710. In some examples, GPU 714 may be dedicated hardware that includes integrated and / or discrete logic circuits that provide GPU 714 with large-scale parallel processing capabilities suitable for graphics processing. In some cases, GPU 714 may also include general-purpose processing capabilities and may be referred to as a general-purpose GPU (GPGPU) when implementing general-purpose processing tasks (i.e., non-graphics-related tasks). Display processor 718 may also be dedicated integrated circuit hardware that is designed to retrieve image content from system memory 716, combine the image content into image frames, and output the image frames to display 703.

[0154] The processor 712 can execute various types of applications. Examples of applications include web browsers, email applications, spreadsheets, video games, other applications that generate visual objects for display, or any of the types of applications listed in more detail above. The system memory 716 can store instructions for executing applications. Executing one of the applications on the processor 712 causes the processor 712 to generate graphics data for image content to be displayed and audio data 19 to be played (possibly via the integrated speaker 740). The processor 712 can send the graphics data of the image content to the GPU 714 for further processing based on the instructions or commands sent by the processor 712 to the GPU 714.

[0155] Processor 712 may communicate with GPU 714 according to a specific application processing interface (API). Examples of these APIs include Company API, Chronos Group or OpenGL and OpenCL TM However, aspects of the present disclosure are not limited to DirectX, OpenGL or OpenCL APIs, and can be extended to other types of APIs. In addition, the techniques described in the present disclosure do not need to work according to an API, and the processor 712 and the GPU 714 can communicate using any process.

[0156] The system memory 716 may be a memory of the device 710. The system memory 716 may include one or more computer-readable storage media. Examples of the system memory 716 include, but are not limited to, random access memory (RAM), electrically erasable programmable read-only memory (EEPROM), flash memory, or other media that can be used to carry or store desired program code in the form of instructions and / or data structures and can be accessed by a computer or processor.

[0157] In some examples, system memory 716 may include instructions that cause processor 712, GPU 714, and / or display processor 718 to perform functions attributed in this disclosure to processor 712, GPU 714, and / or display processor 718. Thus, system memory 716 may be a computer-readable storage medium having stored thereon instructions that, when executed, cause one or more processors (e.g., processor 712, GPU 714, and / or display processor 718) to perform various functions.

[0158] System memory 716 may include non-transitory storage media. The term "non-transitory" indicates that the storage media is not embodied in a carrier wave or propagating signal. However, the term "non-transitory" should not be interpreted to mean that system memory 716 is non-removable or that its contents are static. As an example, system memory 716 may be removed from device 710 and moved to another device. As another example, a memory substantially similar to system memory 716 may be inserted into device 710. In some examples, non-transitory storage media may store data that may change over time (e.g., in RAM).

[0159] The user interface 720 may represent one or more hardware or virtual (meaning a combination of hardware and software) user interfaces through which a user may interface with the device 710. The user interface 720 may include physical buttons, switches, toggle switches, lights, or virtual versions thereof. The user interface 720 may also include a physical or virtual keyboard, a touch interface (such as a touch screen), tactile feedback, and the like.

[0160] The processor 712 may include one or more hardware units (including so-called "processing cores") configured to perform all or some portion of the operations discussed above with respect to one or more of the modules, units or other functional components of the content creator device and / or the content consumer device. The antenna 721 and the transceiver module 722 may represent units configured to establish and maintain a connection between the content consumer device 14 and the content consumer device 14. The antenna 721 and the transceiver module 722 may represent a processor capable of communicating with a content creator device 14 in accordance with a fifth generation (5G) cellular standard, a personal area network (PAN) protocol (e.g., Bluetooth TM ) or one or more wireless communication protocols of other open source, proprietary or other communication standards for wireless communication. For example, the transceiver module 722 can receive and / or send wireless signals. The transceiver module 722 can represent a separate transmitter, a separate receiver, a separate transmitter and a separate receiver, or a combined transmitter and receiver. The antenna 721 and the transceiver module 722 can be configured to receive encoded audio data. Similarly, the antenna 721 and the transceiver module 722 can be configured to send encoded audio data.

[0161] It should be appreciated that, depending on the example, certain actions or events of any of the techniques described herein may be performed in a different sequence, may be added, combined, or omitted entirely (e.g., not all described actions or events are necessary to practice the techniques). Furthermore, in some examples, actions or events may be performed simultaneously rather than sequentially, such as through multithreading, interrupt handling, or multiple processors.

[0162] In some examples, the VR device (or streaming device) can use a network interface coupled to the memory of the VR / streaming device to transmit exchange messages to an external device, where the exchange messages are associated with multiple available representations of the sound field. In some examples, the VR device can use an antenna coupled to the network interface to receive a wireless signal including data packets, audio packets, video packets, or transport protocol data associated with multiple available representations of the sound field. In some examples, one or more microphone arrays can capture the sound field.

[0163] In some examples, the multiple available representations of the sound field stored to the memory device may include multiple object-based representations of the sound field, a high-order ambisonic representation of the sound field, a mixed-order ambisonic representation of the sound field, a combination of the object-based representation of the sound field and a high-order ambisonic representation of the sound field, a combination of the object-based representation of the sound field and a mixed-order ambisonic representation of the sound field, or a combination of the mixed-order representation of the sound field and a high-order ambisonic representation of the sound field.

[0164] In some examples, one or more of the multiple available representations of the sound field can include at least one high resolution region and at least one lower resolution region, and wherein the selected presentation based on the steering angle provides greater spatial accuracy relative to the at least one high resolution region and less spatial accuracy relative to the lower resolution region.

[0165] This disclosure includes the following examples.

[0166] Item 1. A device configured to obtain image data, the device comprising: a memory configured to store one or more priority values, each of the one or more priority values ​​being associated with a type of image object associated with the image data; and one or more processors coupled to the memory and configured to: associate image objects in the image data with one or more audio sources in one or more audio streams; assign a corresponding priority value to each of the one or more audio sources in the one or more audio streams; and encode and decode a stereo reverberation coefficient based on the assigned priority value.

[0167] Clause 2. The device of clause 1, wherein the one or more processors are further configured to spatially map one or more audio sources in the one or more audio streams to associated image objects.

[0168] Clause 3. The apparatus of clause 2, wherein the position information comprises the angle θ and horn.

[0169] Clause 4. The device of any combination of clauses 2 to 3, wherein the positional information further comprises a V vector for each audio source, and the one or more processors are further configured to match the spatially mapped audio sources to the V vectors.

[0170] Clause 5. The device of clause 4, wherein the one or more processors are further configured to match the spatially mapped audio sources to V vectors that are only in the foreground.

[0171] Clause 6. A device as described in any combination of clauses 4 to 5, wherein the one or more processors are further configured to modify the V vector to differentiate the signal-to-noise ratio of at least two of the one or more audio sources when encoding and decoding at least two of the one or more audio sources.

[0172] Clause 7. The device of any combination of clauses 1 to 6, wherein the one or more processors are further configured to encode and decode more bits for higher priority value audio sources than for lower priority value audio sources.

[0173] Clause 7.1 The apparatus of any combination of Clauses 1 to 7, wherein the one or more processors are further configured to encode and decode the plurality of ambisonic coefficients based on the assigned priority value and available resources.

[0174] Clause 7.2 A device as described in Clause 7.1, wherein the available resource is a battery energy level.

[0175] Clause 8. The device of any combination of clauses 1 to 7.2, wherein the one or more processors are further configured to output the one or more audio sources to one or more speakers.

[0176] Clause 9. The device of any combination of clauses 1 to 8, wherein the one or more processors are further configured to combine at least two of the one or more audio sources.

[0177] Clause 10. The device of clause 9, wherein the one or more processors combine at least two of the one or more audio sources by at least one of mixing or interpolation.

[0178] Clause 11. The device of any combination of clauses 1 to 10, further comprising a display device.

[0179] Clause 12. The device of any combination of clauses 1 to 11, further comprising a microphone, wherein the one or more processors are further configured to receive voice commands embedded in an audio stream from the microphone and control the display device based on the voice commands.

[0180] Clause 13. The device of any combination of clauses 1 to 12, further comprising one or more speakers.

[0181] Clause 14. The device of any combination of clauses 1 to 12, wherein the device comprises a mobile handset.

[0182] Clause 15. A device as described in any combination of clauses 1 to 13, wherein the device comprises an extended reality head-mounted device, and wherein the acoustic space comprises a scene represented by video data captured by a camera.

[0183] Clause 16. The device of any combination of clauses 1 to 13, wherein the device comprises an extended reality head mounted device, and wherein the acoustic space comprises a virtual world.

[0184] Clause 17. The device of any combination of clauses 1 to 16, further comprising a head mounted device configured to present an acoustic space.

[0185] Clause 18. The device of any combination of clauses 1 to 17, further comprising a wireless transceiver coupled to the one or more processors and configured to receive wireless signals.

[0186] Clause 19. The device of clause 18, wherein the wireless signal complies with a personal area network standard.

[0187] Clause 19.5 The device of Clause 19, wherein the personal area network standard comprises the AptX standard.

[0188] Clause 20. The device of clause 18, wherein the wireless signal complies with a fifth generation (5G) cellular protocol.

[0189] Clause 20.1 An apparatus as described in any of Clauses 1 to 20, wherein the one or more processors are configured to analyze the image data through computer vision.

[0190] Clause 20.2 An apparatus as described in any of Clauses 1 to 20.1, wherein the one or more processors are configured to analyze the image data by machine learning.

[0191] Clause 20.3 Apparatus as described in any of Clauses 1 to 20.2, wherein the apparatus comprises a decoder.

[0192] Clause 20.4 Apparatus as described in any of Clauses 1 to 20.3, wherein the apparatus comprises an encoder.

[0193] Clause 21. A method for obtaining image data, comprising: storing one or more priority values ​​in a memory, each of the one or more priority values ​​being associated with a type of an image object associated with the image data; associating the image objects in the image data with one or more audio sources in one or more audio streams by one or more processors; assigning a corresponding priority value to each of the one or more audio sources in the one or more audio streams by one or more processors; and encoding and decoding a stereo reverberation coefficient based on the assigned priority value by the one or more processors.

[0194] Clause 22. The method of Clause 21, further comprising spatially mapping one or more audio sources in one or more audio streams to associated image objects.

[0195] Clause 23. The method of clause 22, wherein the position information includes the angle θ and horn.

[0196] Clause 24. The method of any combination of clauses 22 to 23, wherein the positional information further comprises a V vector for each audio source, and the method further comprises matching the spatially mapped audio sources to the V vectors.

[0197] Clause 25. The method of Clause 24, further comprising matching the spatially mapped audio sources to the V vectors that are only in the foreground.

[0198] Clause 26. The method of any combination of clauses 24 to 25, further comprising modifying the V vector to differentiate signal-to-noise ratios of at least two of the one or more audio sources when encoding and decoding the at least two of the one or more audio sources.

[0199] Clause 27. The method of any combination of clauses 21 to 26, wherein the codec encodes more bits for higher priority value audio sources than for lower priority value audio sources.

[0200] Clause 27.1 A method as described in any combination of Clauses 21 to 27, wherein the encoding and decoding is based on an assigned priority value and available resources.

[0201] Clause 27.2 The method of Clause 27.1, wherein the available resource is a battery energy level.

[0202] Clause 28. The method of any combination of clauses 21 to 27.2, further comprising outputting, by the one or more processors, the one or more audio sources to one or more speakers.

[0203] Clause 29. The method of any combination of clauses 21 to 28, further comprising combining, by one or more processors, at least two of the one or more audio sources.

[0204] Clause 30. The method of Clause 29, wherein at least two of the one or more audio sources are combined by at least one of mixing or interpolation.

[0205] Clause 31. The method of any combination of clauses 21 to 30, further comprising receiving voice commands embedded in an audio stream from a microphone and controlling the display device based on the voice commands by one or more processors.

[0206] Clause 32. The method of any combination of clauses 21 to 31, wherein the acoustic space comprises a scene represented by video data captured by a camera.

[0207] Clause 33. The method of any combination of clauses 21 to 32, wherein the acoustic space comprises a virtual world.

[0208] Clause 34. The method as described in any combination of clauses 21 to 33, further comprising presenting the acoustic space on a head mounted device.

[0209] Clause 35. The method of any combination of clauses 21 to 34, wherein the method is performed on a mobile handset.

[0210] Clause 36. The method of any combination of clauses 21 to 35, further comprising receiving a wireless signal.

[0211] Clause 37. The method of clause 36, wherein the wireless signal complies with a personal area network standard.

[0212] Clause 37.5 A method as described in Clause 37, wherein the personal area network standard includes the AptX standard.

[0213] Clause 38. A method as described in Clause 36, wherein the wireless signal complies with a fifth generation (5G) cellular protocol.

[0214] Clause 38.1 The method of any combination of Clauses 21 to 38, wherein analyzing the image data comprises utilizing computer vision.

[0215] Clause 38.2 A method as described in any combination of Clauses 21 to 38.1, wherein analyzing the image data includes utilizing machine learning.

[0216] Clause 38.3 A method as described in any combination of Clauses 21 to 38.2, wherein encoding and decoding is decoding.

[0217] Clause 38.4 A method as described in any combination of Clauses 21 to 38.2, wherein the encoding and decoding is encoding.

[0218] Item 39. A device configured to obtain image data, comprising: a component for storing one or more priority values, each of the one or more priority values ​​being associated with a type of an image object associated with the image data; a component for associating the image objects in the image data with one or more audio sources in one or more audio streams; a component for assigning a corresponding priority value to each of the one or more audio sources in the one or more audio streams; and a component for encoding and decoding a stereo reverberation coefficient based on the assigned priority value.

[0219] Clause 40. The apparatus of Clause 39, further comprising means for spatially mapping one or more audio sources in one or more audio streams to an image object.

[0220] Clause 41. The apparatus of clause 40, wherein the position information comprises the angle θ and horn.

[0221] Clause 42. The apparatus of any combination of clauses 40 to 41, wherein the positional information further comprises a V vector for each audio source, and the apparatus further comprises means for matching the spatially mapped audio sources to the V vectors.

[0222] Clause 43. The apparatus of Clause 42, wherein the means for matching the spatially mapped audio sources to the V vectors matches only in the foreground.

[0223] Clause 44. The apparatus of any combination of clauses 42 to 42, further comprising means for modifying the V vector to differentiate signal-to-noise ratios of at least two of the one or more audio sources when encoding and decoding the at least two of the one or more audio sources.

[0224] Clause 45. The apparatus of any combination of clauses 39 to 44, wherein the means for encoding and decoding ambisonic reverberation coefficients encodes and decodes more bits for higher priority value audio sources than for lower priority value audio sources.

[0225] Clause 45.1 The apparatus of any combination of Clauses 39 to 45, wherein the means for encoding and decoding the ambisonic reverberation coefficients encodes and decodes the ambisonic reverberation coefficients based on the assigned priority value and available resources.

[0226] Clause 45.2 A device as described in any combination of Clauses 39 to 45.1, wherein the available resource is a battery energy level.

[0227] Clause 46. The device of any combination of clauses 39 to 45, further comprising means for outputting one or more audio sources to one or more speakers.

[0228] Clause 47. The apparatus of any combination of clauses 39 to 46, further comprising means for combining at least two of the one or more audio sources.

[0229] Clause 48. The apparatus of Clause 47, wherein at least two of the one or more audio sources are combined by at least one of mixing or interpolation.

[0230] Clause 49. The device of any combination of clauses 39 to 48, further comprising means for displaying the image data.

[0231] Clause 50. The device of Clause 49, further comprising means for receiving voice commands embedded in an audio stream from a microphone, and means for controlling the display device based on the voice commands.

[0232] Clause 51. The device of any combination of clauses 39 to 50, further comprising means for playing a plurality of audio streams.

[0233] Clause 52. The device of any combination of clauses 39 to 51, wherein the device comprises a mobile handset.

[0234] Clause 53. A device as described in any combination of clauses 39 to 49, wherein the device comprises an extended reality head-mounted device, and wherein the acoustic space comprises a scene represented by video data captured by a camera.

[0235] Clause 54. The device of any combination of clauses 39 to 49, wherein the device comprises an extended reality head mounted device, and wherein the acoustic space comprises a virtual world.

[0236] Clause 55. The device of any combination of clauses 39 to 54, further comprising components for presenting an acoustic space on a head mounted device.

[0237] Clause 56. The apparatus of any combination of clauses 39 to 55, further comprising means for receiving a wireless signal.

[0238] Clause 57. The device of clause 56, wherein the wireless signal complies with a personal area network standard.

[0239] Clause 57.5 The device of Clause 57, wherein the personal area network standard comprises the AptX standard.

[0240] Clause 58. The apparatus of clause 56, wherein the wireless signal complies with a fifth generation (5G) cellular protocol.

[0241] Clause 58.1. The apparatus of any combination of clauses 39 to 58, wherein the means for analyzing the image data comprises means for computer vision.

[0242] Clause 58.2. The apparatus of any combination of clauses 39 to 58.1, wherein the component for analyzing the image data comprises a component for machine learning.

[0243] Clause 58.3. The apparatus of any combination of clauses 39 to 58.2, wherein the apparatus comprises a decoder.

[0244] Clause 58.4. The apparatus of any combination of clauses 39 to 58.3, wherein the apparatus comprises an encoder.

[0245] Item 59. A non-transitory computer-readable storage medium having instructions stored thereon that, when executed, cause one or more processors to: store one or more priority values, each of the one or more priority values ​​being associated with a type of image object associated with image data; associate the image objects in the image data with one or more audio sources in one or more audio streams; assign a corresponding priority value to each of the one or more audio sources in the one or more audio streams; and encode and decode a stereo reverberation coefficient based on the assigned priority value.

[0246] Clause 60. The non-transitory computer-readable storage medium of Clause 59, wherein the instructions further cause the one or more processors to spatially map one or more audio sources in the one or more audio streams to an image object.

[0247] Clause 61. The non-transitory computer-readable storage medium of clause 60, wherein the position information comprises an angle θ and a horn.

[0248] Clause 62. A non-transitory computer-readable storage medium as described in any combination of clauses 60 to 61, wherein the positional information also includes a V vector for each audio source, and the instructions when executed further cause the one or more processors to match the spatially mapped audio sources with the V vectors.

[0249] Clause 63. The non-transitory computer-readable storage medium of Clause 62, wherein the instructions, when executed, further cause the one or more processors to match the spatially mapped audio sources to the V vectors that are only in the foreground.

[0250] Clause 64. A non-transitory computer-readable storage medium as described in any combination of clauses 62 to 63, wherein the instruction, when executed, further causes one or more processors to modify the V vector to differentiate the signal-to-noise ratio of at least two of the one or more audio sources when encoding and decoding at least two of the one or more audio sources.

[0251] Clause 65. The non-transitory computer-readable storage medium of any combination of clauses 59 to 64, wherein the ambisonics coefficients encode more bits for higher priority value audio sources than for lower priority value audio sources.

[0252] Clause 65.1 The non-transitory computer-readable storage medium of any combination of clauses 59 to 65, wherein the instructions, when executed, cause one or more processors to encode and decode a plurality of ambisonic reverberation coefficients based on assigned priority values ​​and available resources.

[0253] Clause 65.2 The non-transitory computer-readable storage medium of Clause 65.1, wherein the available resource is a battery energy level.

[0254] Clause 66. The non-transitory computer-readable storage medium of any combination of clauses 59 to 65, wherein the instructions, when executed, cause one or more processors to output one or more audio sources to one or more speakers.

[0255] Clause 67. The non-transitory computer-readable storage medium of any combination of clauses 59 to 66, wherein the instructions, when executed, cause one or more processors to combine at least two of the one or more audio sources.

[0256] Clause 68. The non-transitory computer-readable storage medium of Clause 67, wherein the instructions, when executed, cause one or more processors to combine at least two of the one or more audio sources by at least one of mixing or interpolation.

[0257] Clause 69. The non-transitory computer-readable storage medium of any combination of clauses 59 to 68, wherein the instructions, when executed, cause one or more processors to control a display device based on voice commands embedded in an audio stream.

[0258] Clause 70. The non-transitory computer-readable storage medium of any combination of clauses 59 to 69, wherein the acoustic space comprises a scene represented by video data captured by a camera.

[0259] Clause 71. The non-transitory computer-readable storage medium of any combination of clauses 59 to 69, wherein the acoustic space comprises a virtual world.

[0260] Clause 72. The non-transitory computer-readable storage medium of any combination of clauses 59 to 71, wherein the instructions, when executed, cause one or more processors to render an acoustic space on a mobile handset.

[0261] Clause 73. The non-transitory computer-readable storage medium of any combination of clauses 59 to 72, wherein the instructions, when executed, cause one or more processors to present an acoustic space on a head mounted device.

[0262] Clause 74. The non-transitory computer-readable storage medium of any combination of clauses 59 to 73, wherein the instructions, when executed, cause one or more processors to receive a wireless signal.

[0263] Clause 75. The non-transitory computer-readable storage medium of Clause 74, wherein the wireless signal complies with a personal area network standard.

[0264] Clause 75.5 The non-transitory computer-readable storage medium of Clause 75, wherein the personal area network standard comprises the AptX standard.

[0265] Clause 76. The non-transitory computer-readable storage medium of clause 74, wherein the wireless signal complies with a fifth generation (5G) cellular protocol.

[0266] Clause 77. The non-transitory computer-readable storage medium of any combination of clauses 59 to 76, wherein the instructions, when executed, cause one or more processors to analyze the image data through computer vision.

[0267] Clause 78. The non-transitory computer-readable storage medium of any combination of clauses 59 to 77, wherein the instructions, when executed, cause one or more processors to analyze the image data through machine learning.

[0268] Clause 79. The non-transitory computer-readable storage medium of any combination of clauses 59 to 78, wherein the instructions, when executed, cause one or more processors to encode and decode ambisonic reverberation coefficients by decoding the ambisonic reverberation coefficients.

[0269] Clause 80. The non-transitory computer-readable storage medium of any combination of clauses 59 to 79, wherein the instructions, when executed, cause one or more processors to encode and decode ambisonic reverberation coefficients by encoding the ambisonic reverberation coefficients.

[0270] In one or more examples, the described functions may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or sent through a computer-readable medium as one or more instructions or codes and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media, which corresponds to tangible media such as data storage media, or communication media, including, for example, any media that facilitates the transfer of a computer program from one place to another according to a communication protocol. In this manner, computer-readable media may generally correspond to (1) non-transitory tangible computer-readable storage media, or (2) communication media such as signals or carrier waves. Data storage media may be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, codes, and / or data structures to implement the techniques described in the present disclosure. A computer program product may include a computer-readable medium.

[0271] As an example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage devices, flash memory, or any other medium that can be used to store the required program code in the form of instructions or data structures and can be accessed by a computer. In addition, any connection is appropriately referred to as a computer-readable medium. For example, if a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wireless technologies such as infrared, radio and microwaves are used to send instructions from a website, server or other remote source, then coaxial cable, fiber optic cable, twisted pair, DSL or wireless technologies such as infrared, radio and microwaves are included in the definition of the medium. However, it should be understood that computer-readable storage media and data storage media exclude connections, carriers, signals or other temporary media, but are directed to non-temporary tangible storage media. Disks and optical disks as used herein include compact disks (CDs), laser optical disks, optical optical disks, digital versatile disks (DVDs), floppy disks and blue-ray disks, wherein disks usually reproduce data magnetically, while optical disks reproduce data optically using lasers. The above combination should also be included in the scope of computer-readable media.

[0272] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Therefore, the term "processor" as used herein may refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. In addition, in some aspects, the functionality described herein may be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated in a combined codec. Similarly, the technology may be fully implemented in one or more circuits or logic elements.

[0273] The techniques of the present disclosure may be implemented in a variety of devices or apparatuses including wireless handsets, integrated circuits (ICs) or IC sets (e.g., chipsets). Various components, modules, or units are described in the present disclosure to emphasize functional aspects of devices configured to perform the disclosed techniques, but do not necessarily need to be implemented by different hardware units. Instead, as described above, the various units may be combined in a codec hardware unit, or provided by a collection of interoperable hardware units in combination with appropriate software and / or firmware, the collection of interoperable hardware units including one or more processors as described above.

[0274] Various examples have been described. These and other examples are within the scope of the following claims.

Claims

1. A device for obtaining image data, comprising: a memory configured to store one or more priority values, each of the one or more priority values ​​being associated with a type of image object associated with the image data; as well as One or more processors, coupled to the memory and configured to: associating image objects in the image data with one or more audio sources represented in one or more audio streams; spatially mapping the one or more audio sources represented in the one or more audio streams to associated image objects; storing position information for each of the one or more audio sources, the position information comprising a corresponding θ angle and φ angle; assigning a respective priority value to each of the one or more audio sources represented in the one or more audio streams; and The ambisonic reverberation coefficients are encoded and decoded based on the assigned priority values.

2. The device according to claim 1, wherein: The positional information also includes a V vector for each audio source, and the one or more processors are further configured to match the spatially mapped audio sources to the V vectors.

3. The device of claim 2, wherein: The one or more processors are also configured to match the spatially mapped audio sources to the V vectors only in the foreground.

4. The device of claim 2, wherein: The one or more processors are further configured to modify the V vector to differentiate signal-to-noise ratios of at least two of the one or more audio sources when encoding and decoding at least two of the one or more audio sources represented in the one or more audio streams.

5. The device of claim 1, wherein: The one or more processors are further configured to encode and decode more bits for higher priority value audio sources than for lower priority value audio sources.

6. The device of claim 1, wherein: The one or more processors are further configured to encode and decode a plurality of ambisonic coefficients based on the assigned priority values ​​and available resources.

7. The device of claim 6, wherein: The available resource is a battery energy level.

8. The device of claim 1, wherein: The one or more processors are further configured to combine at least two of the one or more audio sources represented in the one or more audio streams by at least one of mixing or interpolation.

9. The device of claim 1, wherein: The device comprises a mobile handset.

10. The device according to claim 1, in, The device includes an extended reality head mounted device, and Among them, the acoustic space includes the scene represented by the video data captured by the camera.

11. The device according to claim 1, in, The device includes an extended reality head mounted device, and Among them, the acoustic space includes the virtual world.

12. The device of claim 1, further comprising a head mounted device configured to render the acoustic space.

13. A method for obtaining image data, comprising: storing in a memory one or more priority values, each of the one or more priority values ​​being associated with a type of image object associated with the image data; associating, by one or more processors, image objects in the image data with one or more audio sources represented in one or more audio streams; spatially mapping, by the one or more processors, the one or more audio sources represented in the one or more audio streams to associated image objects; storing in the memory position information for each of the one or more audio sources, the position information comprising a corresponding angle θ and an angle φ; assigning, by the one or more processors, a respective priority value to each of the one or more audio sources represented in the one or more audio streams; as well as The ambisonic reverberation coefficients are encoded and decoded by the one or more processors based on the assigned priority values.

14. The method of claim 13, wherein: The positional information also includes a V vector for each audio source, and the method further includes matching the spatially mapped audio sources to the V vectors.

15. The method of claim 14, further comprising matching the spatially mapped audio sources to V vectors that are only in the foreground.

16. The method of claim 14, further comprising modifying the V vector to differentiate signal-to-noise ratios of at least two of the one or more audio sources when encoding and decoding at least two of the one or more audio sources represented in the one or more audio streams.

17. The method of claim 13, wherein: The codec encodes more bits for higher priority value audio sources than for lower priority value audio sources.

18. The method of claim 13, wherein: The encoding and decoding is based on the assigned priority value and available resources.

19. The method of claim 18, wherein: The available resource is a battery energy level.

20. The method of claim 13, further comprising combining, by the one or more processors, at least two of the one or more audio sources represented in the one or more audio streams by at least one of mixing or interpolation.

21. The method of claim 13, wherein: The acoustic space includes the scene represented by the video data captured by the camera.

22. The method of claim 13, wherein: The acoustic space includes the virtual world.

23. The method of claim 13, further comprising presenting the acoustic space on a head mounted device.

24. The method of claim 13, wherein: The method is performed on a mobile handset.

25. A device configured to obtain image data, comprising: means for storing one or more priority values, each of the one or more priority values ​​being associated with a type of image object associated with the image data; means for associating image objects in the image data with one or more audio sources represented in one or more audio streams; means for spatially mapping the one or more audio sources represented in the one or more audio streams to associated image objects; means for storing position information for each of the one or more audio sources, the position information comprising respective angles θ and φ; means for assigning a respective priority value to each of the one or more audio sources represented in the one or more audio streams; as well as Means for encoding and decoding ambisonic coefficients based on the assigned priority values.

26. A non-transitory computer-readable storage medium having stored thereon instructions that, when executed, cause one or more processors to: storing one or more priority values, each of the one or more priority values ​​being associated with a type of image object associated with the image data; associating image objects in the image data with one or more audio sources represented in one or more audio streams; spatially mapping the one or more audio sources represented in the one or more audio streams to associated image objects; storing position information for each of the one or more audio sources, the position information comprising a corresponding θ angle and φ angle; assigning a respective priority value to each of the one or more audio sources represented in the one or more audio streams; and The ambisonic reverberation coefficients are encoded and decoded based on the assigned priority values.

Images