Perform psychoacoustic audio encoding and decoding based on operating conditions

Abstract

In general, various aspects of the technology described in this disclosure relate to performing psychoacoustic audio encoding and decoding based on operating conditions. A device including a memory and one or more processors can be configured to perform these technologies. The memory can be configured to store encoded scene-based audio data. The one or more processors can be configured to obtain operating conditions of the device for decoding the encoded scene-based audio data, and based on the operating conditions, perform psychoacoustic audio decoding on the encoded scene-based audio data to obtain ambient sound transmission format audio data. The one or more processors can also be configured to perform spatial audio decoding on the ambient sound transmission format audio data to obtain scene-based audio data.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Patent Application No. 16 / 907,771, filed on June 22, 2020, entitled “PERFORMING PSYCHOACOUSTIC AUDIO CODING BASED ON OPERATING CONDITIONS,” which claims the benefit of U.S. Provisional Application No. 62 / 865,848, filed on June 24, 2019, entitled “PERFORMING PSYCHOACOUSTIC AUDIO CODING BASED ON OPERATING CONDITIONS,” the entire contents of which are incorporated herein by reference as if fully set forth herein. Technical Field

[0003] The present disclosure relates to audio data, and more particularly, to the coding of audio data. Background Art

[0004] Psychoacoustic audio codecs refer to the process of compressing audio data using psychoacoustic models. Psychoacoustic audio codecs can exploit limitations in the human auditory system to compress audio data, taking into account limitations that arise due to spatial masking (e.g., two audio sources in the same location, where one auditory source masks the other auditory source in loudness), temporal masking (e.g., where one audio source masks the other auditory source in loudness), etc. Psychoacoustic models can attempt to model the human auditory system to identify masked sound fields or other parts of the sound field that are redundant, masked, or otherwise imperceptible to the human auditory system. Psychoacoustic audio codecs can also perform lossless compression by entropy encoding the audio data. Summary of the invention

[0005] In general, techniques for performing psychoacoustic audio codecs based on operating conditions are described.

[0006] In one example, various aspects of the technology relate to a device configured to decode encoded scene-based audio data, the device comprising: a memory configured to store the encoded scene-based audio data; one or more processors configured to: obtain an operating condition of the device for decoding the encoded scene-based audio data; based on the operating condition, perform psychoacoustic audio decoding on the encoded scene-based audio data to obtain ambisonic transport format audio data; and perform spatial audio decoding on the ambisonic transport format audio data to obtain scene-based audio data.

[0007] In another example, various aspects of the technology relate to a method for decoding encoded scene-based audio data, the method comprising: obtaining an operating condition of a device configured to decode the encoded scene-based audio data; based on the operating condition, performing psychoacoustic audio decoding on the encoded scene-based audio data to obtain ambient sound transmission format audio data; and performing spatial audio decoding on the ambient sound transmission format audio data to obtain scene-based audio data.

[0008] In another example, various aspects of the technology relate to a device configured to decode encoded scene-based audio data, the device including: a component for obtaining an operating condition of the device configured to decode the encoded scene-based audio data; a component for performing psychoacoustic audio decoding on the encoded scene-based audio data based on the operating condition to obtain ambient sound transmission format audio data; and a component for performing spatial audio decoding on the ambient sound transmission format audio data to obtain scene-based audio data.

[0009] In another example, various aspects of the technology relate to a non-transitory computer-readable storage medium having instructions stored thereon, which, when executed, cause one or more processors to: obtain an operating condition of a device configured to decode encoded scene-based audio data; based on the operating condition, perform psychoacoustic audio decoding on the encoded scene-based audio data to obtain ambient sound transmission format audio data; and perform spatial audio decoding on the ambient sound transmission format audio data to obtain scene-based audio data.

[0010] The details of one or more aspects of these techniques are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of these techniques will become apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a diagram illustrating a system that can perform various aspects of the techniques described in this disclosure.

[0012] Figure 2 is a diagram illustrating another example of a system that can perform various aspects of the techniques described in this disclosure.

[0013] Figure 3A and Figure 3B is shown in more detail Figure 2 A block diagram of an example of an audio decoder.

[0014] Figure 4 is shown in more detail Figure 3A and Figure 3B A block diagram of an example of a multiplexer (mux) control unit shown in an example.

[0015] Figure 5 is shown in more detail Figure 3A and Figure 3B A block diagram of an example of a buffer control unit.

[0016] Figure 6 is a block diagram illustrating different versions of ATF audio data after processing by an audio decoder in accordance with various aspects of the present technique.

[0017] Figure 7 is a diagram showing various aspects of a system configured to perform the techniques described in this disclosure. Figure 2 Block diagram of an example of a psychoacoustic audio encoder.

[0018] Figure 8 is shown in more detail Figure 1-Figure 3B Block diagram of an embodiment of a psychoacoustic audio decoder.

[0019] Fig.9A and Fig. 9B is shown in more detail Figure 1-Figure 3B A block diagram of an additional example of a psychoacoustic audio encoder is shown in the example of FIG.

[0020] Fig. 10A and Fig. 10B is shown in more detail Figure 1-Figure 3B A block diagram of an additional example of a psychoacoustic audio decoder.

[0021] Fig.11 is a diagram showing an example of top-down quantization.

[0022] Fig.12 is a diagram showing an example of bottom-up quantization.

[0023] Fig.13 It is shown Figure 2 00106 00107 00108 00109 ...

[0024] Fig.14 It is shown Figure 2 A block diagram of exemplary components of a sink device shown in the example of FIG.

[0025] Fig.15 is a diagram showing various aspects of the technology described in this disclosure Figure 2 Flowchart of example operation of an audio decoder. DETAILED DESCRIPTION

[0026] There are different types of audio formats, including channel-based, object-based, and scene-based formats. Scene-based formats can use surround sound technology. Surround sound technology allows the sound field to be represented using a collection of layered elements that can be rendered to speaker feeds for most speaker configurations.

[0027] An example of a hierarchical element set is the set of spherical harmonic coefficients (SHCs). The following expression demonstrates the description or representation of a sound field using SHCs:

[0028]

[0029] This expression shows that at any point in the sound field at time t The pressure at p i Can be provided by SHC Here, c is the speed of sound (about 343 m / s), is the reference point (or observation point), j n (·) is a spherical Bessel function of order n, and is the spherical harmonic basis function of order n and suborder m (also called spherical basis function). It can be seen that the term in the square brackets is the frequency domain representation of the signal (i.e., ), which can be approximated by various time-frequency transforms, such as discrete Fourier transform (DFT), discrete cosine transform or wavelet transform. Other examples of hierarchical sets include wavelet transform coefficient sets and other coefficient sets of multiresolution basis functions.

[0030] SHC The SHCs (also referred to as surround sound coefficients) represent scene-based audio, where the SHCs can be input to an audio encoder to obtain encoded SHCs that can facilitate more efficient transmission or storage. For example, a method involving (1+4) can be used. 2 A fourth-order representation of 25 (so fourth-order) coefficients.

[0031] As described above, SHC can be derived from microphone recordings using a microphone array. Various examples of how SHC can be derived from a microphone array are described in Poletti, M., "Three-Dimensional Surround Sound Systems Based on Spherical Harmonics," J. Audio Eng. Soc., Vol. 53, No. 11, November 2005, pp. 1004-1025.

[0032] To illustrate how SHC can be derived from an object-based description, consider the following equation. The coefficients of the sound field corresponding to an individual audio object are It can be expressed as:

[0033]

[0034] Where i is is a spherical Hankel function (of the second kind) of order n, and is the position of the object. Knowing the source energy of the object g(ω) as a function of frequency (e.g., using time-frequency analysis techniques such as performing a Fast Fourier Transform on the PCM stream) allows us to convert each PCM object and the corresponding position into an SHC Furthermore, it can be shown (since the above is a linear and orthogonal decomposition) that for each object The coefficients are additive. In this way, multiple PCM objects (where a PCM object is an example of an audio object) can be represented by The coefficients are represented (e.g., as the sum of the coefficient vectors of the individual objects). Essentially, the coefficients contain information about the sound field (pressure as a function of the 3D coordinates), and the above represents the distance from the individual objects to the observation point The following figures are described in the context of SHC-based audio coding.

[0035] Figure 1 1 is a diagram illustrating a system 10 that can perform various aspects of the techniques described in this disclosure. Figure 1 , system 10 includes a content creator system 12 and a content consumer 14. Although described in the context of content creator system 12 and content consumer 14, the techniques may be implemented in any context in which SHC (which may also be referred to as surround sound coefficients) or any other layered representation of a sound field is encoded to form a bitstream representing audio data.

[0036] In addition, the content creator system 12 may represent a system that includes one or more of any form of computing device capable of implementing the techniques described in this disclosure, including a handheld device (or cellular phone, including a so-called "smartphone", or in other words, a mobile phone or handheld device), a tablet computer, a laptop computer, a desktop computer, an extended reality (XR) device (which may refer to any one or more of a virtual reality (VR) device, an augmented reality (AR) device, a mixed reality (MR) device, etc.), a gaming system, an optical disc player, a receiver (such as an audio / video (A / V) receiver), or dedicated hardware, to provide some examples.

[0037] Likewise, content consumer 14 may represent any form of computing device capable of implementing the techniques described in this disclosure, including a handset (or cellular telephone, including so-called “smartphones,” or in other words, a mobile collection or phone), an XR device, a tablet computer, a television (including so-called “smart TVs”), a set-top box, a laptop computer, a gaming system or console, a watch (including so-called smart watches), wireless headphones (including so-called “smart headphones”), or a desktop computer, to provide some examples.

[0038] Content creator system 12 may represent any entity that can generate audio content and possibly video content for consumption by content consumers, such as content consumer 14. Content creator system 12 may capture live audio data at an event, such as a sporting event, while also inserting various other types of additional audio data into the live audio content, such as commentary audio data, commercial audio data, intro or exit audio data, etc.

[0039] Content consumer 14 represents an individual who owns or has access to an audio playback system 16, which may refer to any form of audio playback system capable of rendering higher-order surround sound audio data (which includes higher-order audio coefficients, which may also be referred to as spherical harmonic coefficients) to a speaker feed for playback as audio content. Figure 1 In the example of FIG. 1 , content consumer 14 includes audio playback system 16 .

[0040] Surround sound audio data may be defined in the spherical harmonic domain and rendered or otherwise transformed from the spherical harmonic domain to the spatial domain, resulting in audio content in the form of one or more speaker feeds. Surround sound audio data may represent an example of "scene-based audio data", which describes an audio scene using surround sound coefficients. Scene-based audio data differs from object-based audio data in that the entire scene is described (in the spherical harmonic domain) rather than discrete objects (in the spatial domain) as is common in object-based audio data. Scene-based audio data differs from channel-based audio data in that scene-based audio data exists in the spherical harmonic domain, which is different from the spatial domain of channel-based audio data.

[0041] In any case, the content creator system 12 includes microphones 18 that record or otherwise obtain live audio in various formats (including directly as surround sound coefficients and audio objects). When the microphone array 18 (which may also be referred to as "microphones 18") obtains live audio directly as surround sound coefficients, the microphones 18 may include a transcoder, such as Figure 1 The transcoder 20 shown in the example.

[0042] In other words, while shown as being separate from the microphones 5, the surround sound transcoder 20 may be included as a separate instance within each microphone 5 in order to transcode the captured feed into the surround sound coefficients 21. However, when not included within the microphone 18, the surround sound transcoder 20 may transcode the real-time feed output from the microphone 18 into the surround sound coefficients 21. In this regard, the surround sound transcoder 20 may represent a unit configured to transcode the microphone feeds and / or audio objects into the surround sound coefficients 21. Thus, the content creator system 12 includes the surround sound transcoder 20 integrated with the microphone 18, as a transcoder separate from the microphone 18, or some combination thereof.

[0043] The content creator system 12 may also include an audio encoder 22 configured to compress the surround sound coefficients 21 to obtain a bitstream 31. The audio encoder 22 may include a spatial audio encoding device 24 and a psychoacoustic audio encoding device 26. The spatial audio encoding device 24 may represent a device capable of performing compression on the surround sound coefficients 21 to obtain intermediate formatted audio data 25 (which may also be referred to as "mezzanine formatted audio data 25" when the content creator system 12 represents a broadcast network as described in more detail below). The intermediate formatted audio data 25 may represent audio data that has been compressed using spatial audio compression but has not yet undergone psychoacoustic audio encoding (e.g., such as AptX or Advanced Audio Coding (AAC), or other similar types of psychoacoustic audio encoding, including various enhanced AAC (eAAC), such as High Efficiency AAC (HE-AAC) HE-AACv2, which is also referred to as eAAC+, etc.).

[0044] The spatial audio encoding device 24 may be configured to compress the surround sound coefficients 21. That is, the spatial audio encoding device 24 may use a decomposition involving application of a linear reversible transform (LIT) to compress the surround sound coefficients 21. An example of a linear reversible transform is called "singular value decomposition" ("SVD"), principal component analysis ("PCA"), or eigenvalue decomposition, which may represent different examples of linear reversible decomposition.

[0045] In this example, the spatial audio encoding device 24 may apply SVD to the surround sound coefficients 21 to determine a decomposed version of the surround sound coefficients 21. The decomposed version of the surround sound coefficients 21 may include one or more predominant audio signals and one or more corresponding spatial components describing spatial characteristics (e.g., direction, shape, and width) of the associated predominant audio signals. In this way, the spatial audio encoding device 24 may apply decomposition to the surround sound coefficients 21 to decouple the energy (as represented by the predominant audio signals) from the spatial characteristics (as represented by the spatial components).

[0046] The spatial audio encoding device 24 may analyze the decomposed version of the ambisonic sound coefficients 21 to identify various parameters, which may facilitate reordering of the decomposed version of the ambisonic sound coefficients 21. The spatial audio encoding device 24 may reorder the decomposed version of the ambisonic sound coefficients 21 based on the identified parameters, wherein such reordering may improve coding efficiency, assuming that the transform may reorder the ambisonic sound coefficients across a frame of ambisonic sound coefficients (where a frame typically includes M samples of the decomposed version of the ambisonic sound coefficients 21, and in some examples M is set to 1024).

[0047] After reordering the decomposed versions of the surround sound coefficients 21, the spatial audio encoding device 24 may select one or more decomposed versions of the surround sound coefficients 21 as foreground (or in other words, obvious, main or significant) components of the sound field. The spatial audio encoding device 24 may specify a decomposed version of the surround sound coefficients 21 representing a foreground component (which may also be referred to as a "primary sound signal", "primary audio signal" or "primary sound component") and associated directional information (which may also be referred to as a "spatial component" or, in some cases, a so-called "V vector" that identifies the spatial characteristics of the corresponding audio object). The spatial component may represent a vector having multiple different elements (which may be referred to as a "coefficient" in terms of a vector) and may therefore be referred to as a "multidimensional vector".

[0048] The spatial audio encoding device 24 may then perform a sound field analysis on the surround sound coefficients 21 to at least partially identify the surround sound coefficients 21 representing one or more background (or in other words, ambient) components of the sound field. Background components may also be referred to as "background audio signals" or "ambient audio signals". The spatial audio encoding device 24 may perform energy compensation on the background audio signals, assuming that in some examples, the background audio signals may only include a subset of any given sample of the surround sound coefficients 21 (e.g., such as those corresponding to zero-order and first-order spherical basis functions, rather than those corresponding to second-order or higher-order spherical basis functions). When performing order reduction, in other words, the spatial audio encoding device 24 may enhance the remaining background surround sound coefficients of the surround sound coefficients 21 (e.g., add energy to / subtract energy from them) to compensate for the change in total energy caused by performing order reduction.

[0049] The spatial audio encoding device 24 may then perform some form of interpolation on the foreground direction information (which is another way of referring to the spatial component), and then perform order reduction on the interpolated foreground direction information to generate reduced order foreground direction information. In some examples, the spatial audio encoding device 24 may further perform quantization on the reduced order foreground direction information, thereby outputting the encoded foreground direction information. In some cases, such quantization may include scalar / entropy quantization, which may take the form of vector quantization. The spatial audio encoding device 24 may then output the intermediate formatted audio data 25 as a background audio signal, a foreground audio signal, and quantized foreground direction information.

[0050] In any case, in some examples, the background audio signal and the foreground audio signal may include a transmission channel. That is, the spatial audio encoding device 24 may output a transmission channel for each frame of the surround sound coefficient 21 including a corresponding one of the background audio signals (e.g., M samples of one of the surround sound coefficients 21 corresponding to the zero or first order spherical basis function) and each frame of the foreground audio signal (e.g., M samples of the audio object decomposed from the surround sound coefficient 21). The spatial audio encoding device 24 may also output side information (which may also be referred to as "sideband information") including a quantized spatial component corresponding to each foreground audio signal.

[0051] In general, the transmission channel and side information can be Figure 1In the example of , the ATF audio data 25 is represented as surround sound transmission format (ATF) audio data 25 (which is another way to refer to intermediate formatted audio data). In other words, the AFT audio data 25 may include transmission channels and side information (which may also be referred to as "metadata"). As an example, the ATF audio data 25 may conform to the HOA (high-order surround sound) transmission format (HTF). More information about HTF can be found in the European Telecommunications Standards Institute (ETSI) technical specification (TS) entitled "High-Order Surround Sound (HOA) Transmission Format" (ETSI TS103589 V1.1.1, dated June 2018 (2018-06)). In this way, the ATF audio data 25 can be referred to as HTF audio data 25.

[0052] The spatial audio encoding device 24 may then send or otherwise output the ATF audio data 25 to a psychoacoustic audio encoding device 26. The psychoacoustic audio encoding device 26 may perform psychoacoustic audio encoding on the ATF audio data 25 to generate a bitstream 31. The psychoacoustic audio encoding device 26 may operate according to a standardized, open source, or proprietary audio codec process. For example, the psychoacoustic audio encoding device 26 can perform psychoacoustic audio encoding, such as the Unified Speech and Audio Codec denoted as “USAC” proposed by the Moving Picture Experts Group (MPEG), the MPEG-H 3D Audio Codec standard, the MPEG-I Immersive Audio standard, or a proprietary standard, such as AptX™ (including various versions of AptX, such as Enhanced AptX—E-AptX, AptX Live, AptX Stereo, and AptX High Definition—AptX-HD), Advanced Audio Codec (AAC), Audio Codec 3 (AC-3), Apple Lossless Audio Codec (ALAC), MPEG-4 Audio Lossless Streaming (ALS), Enhanced AC-3, Free Lossless Audio Codec (FLAC), Monkey's Audio, MPEG-1 Audio Layer II (MP2), MPEG-1 Audio Layer III (MP3), Opus, and Windows Media Audio (WMA).

[0053] In some examples, the psychoacoustic audio encoding device 26 may represent one or more instances of a psychoacoustic audio codec, each instance being used to encode a transmission channel of the ATF audio data 25. In some cases, the psychoacoustic audio encoding device 26 may represent one or more instances of an AptX encoding unit (as described above). In some cases, the psychoacoustic audio codec unit 26 may call an instance of the AptX encoding unit for each transmission channel of the ATF audio data 25.

[0054] In some examples, in order to generate different representations of the sound field using surround sound coefficients (which are also an example of audio data 21), the audio encoder 22 may use an encoding scheme for the surround sound representation of the sound field, known as mixed-order surround (MOA), as discussed in more detail in U.S. patent application serial number 15 / 672,058, entitled “MIXED-ORDER AMBISONICS (MOA) AUDIO DATA FOR COMPUTER-MEDIATED REALITY SYSTEMS,” filed on August 8, 2017, and published on January 3, 2019 as U.S. Patent Publication No. 2019 / 0007781.

[0055] To generate a particular MOA representation of a sound field, the audio encoder 22 may generate a partial subset of a complete set of surround sound coefficients. For example, each MOA representation generated by the audio encoder 22 may provide accuracy with respect to some areas of the sound field, but provide lower accuracy in other areas. In one example, the MOA representation of the sound field may include eight (8) uncompressed surround sound coefficients, while a third-order surround sound representation of the same sound field may include sixteen (16) uncompressed surround sound coefficients. In this way, each MOA representation of the sound field generated as a partial subset of the surround sound coefficients may be less storage-intensive and less bandwidth-intensive (if and when transmitted as part of the bitstream 31 over the transmission channel shown).

[0056] Although described with respect to MOA representation, the techniques of the present disclosure may also be performed for a full-order surround (FOA) representation, where all surround coefficients of a given order N are used to represent the sound field. In other words, the sound field representation generator 302 may use all surround coefficients of a given order N to represent the sound field, rather than using a partial, non-zero subset of the surround coefficients to represent the sound field, resulting in a total of (N+1) 2 The surround sound coefficient.

[0057] In this regard, higher-order surround sound audio data (which is another way of referring to surround sound coefficients in MOA representation or FOA representation) may include higher-order surround sound coefficients associated with spherical basis functions of order 1 or less (which may be referred to as "first-order surround sound audio data"), higher-order surround sound coefficients associated with spherical basis functions having mixed orders and sub-orders (which may be referred to as the "MOA representation" discussed above), or higher-order surround sound coefficients associated with spherical basis functions of order greater than 1 (referred to as the "FOA representation" above).

[0058] In addition, although Figure 131 is shown as being sent directly to the content consumer 14, but the content creator system 12 may output the bitstream 31 to an intermediate device located between the content creator system 12 and the content consumer 14. The intermediate device may store the bitstream 31 for later transmission to the content consumer 14 requesting the bitstream. The intermediate device may include a file server, a network server, a desktop computer, a laptop computer, a tablet computer, a mobile phone, a smart phone, or any other device capable of storing the bitstream 31 for later retrieval by an audio decoder. The intermediate device may exist in a content delivery network capable of streaming the bitstream 31 (and possibly in conjunction with transmitting a corresponding video data bitstream) to a subscriber (such as the content consumer 14) requesting the bitstream 31.

[0059] Alternatively, the content creator system 12 may store the bitstream 31 to a storage medium such as an optical disc, a digital video disc, a high-definition video disc, or other storage medium, most of which are capable of being read by a computer and may therefore be referred to as a computer-readable storage medium or a non-transitory computer-readable storage medium. In this context, a transmission channel may refer to those channels through which content stored to these media is sent (and may include retail stores and other store-based delivery mechanisms). In any case, the technology of the present disclosure should not be limited in this regard. Figure 1 .

[0060] like Figure 1 As further shown in the example of , the content consumer 14 includes an audio playback system 16. The audio playback system 16 can represent any audio playback system capable of playing multi-channel audio data. The audio playback system 16 can also include an audio decoding device 32. The audio decoding device 32 can represent a device configured to decode the surround sound coefficients 11' from the bitstream 31, wherein the surround sound coefficients 11' can be similar to the surround sound coefficients 11 but are different due to lossy operations (e.g., quantization) and / or transmission via a transmission channel.

[0061] The audio decoding device 32 may include a psychoacoustic audio decoding device 34 and a spatial audio decoding device 36. The psychoacoustic audio decoding device 34 may represent a unit configured to operate inversely to the psychoacoustic audio encoding device 26 to reconstruct the ATF audio data 25' from the bitstream 31. Likewise, the apostrophe with respect to the ATF audio data 25 output from the psychoacoustic audio decoding device 34 indicates that the ATF audio data 25' may differ slightly from the ATF audio data 25 due to lossy or other operations performed during compression of the ATF audio data 25. The psychoacoustic audio decoding device 34 may be configured to perform decompression according to a standardized, open source, or proprietary audio encoding process (such as AptX, a variant of AptX, AAC, a variant of AAC, etc., mentioned above).

[0062] Although described below primarily with respect to AptX, these techniques can also be applied to other psychoacoustic audio codecs. Examples of other psychoacoustic audio codecs include Audio Codec 3 (AC-3), Apple Lossless Audio Codec (ALAC), MPEG-4 Audio Lossless Streaming (ALS), Enhanced AC-3, Free Lossless Audio Codec (FLAC), Monkey's Audio, MPEG-1 Audio Layer II (MP2), MPEG-1 Audio Layer III (MP3), Opus, and Windows Media Audio (WMA).

[0063] In any case, the psychoacoustic audio decoding device 34 can perform psychoacoustic decoding on the foreground audio objects specified in the bitstream 31 and the encoded surround sound coefficients representing the background audio signals specified in the bitstream 31. In this way, the psychoacoustic audio decoding device 34 can obtain the ATF audio data 25' and output the ATF audio data 25' to the spatial audio decoding device 36.

[0064] The spatial audio decoding device 36 may represent a unit configured to operate inversely to the spatial audio encoding device 24. That is, the spatial audio decoding device 36 may dequantize the foreground direction information specified in the bitstream 31. The spatial audio decoding device 36 may also dequantize the quantized foreground direction information to obtain the decoded foreground direction information. The spatial audio decoding device 36 may then perform interpolation on the decoded foreground direction information and then determine the surround sound coefficients representing the foreground component based on the decoded foreground audio signal and the interpolated foreground direction information. The spatial audio decoding device 36 may then determine the surround sound coefficients 11' based on the determined surround sound coefficients representing the foreground audio signal and the decoded surround sound coefficients representing the background audio signal.

[0065] The audio playback system 16 may render the surround sound coefficients 11' to output speaker feeds 39 after decoding the bitstream 31 to obtain the surround sound coefficients 11'. The audio playback system 16 may include a plurality of different audio renderers 38. The audio renderers 38 may each provide different forms of rendering, wherein the different forms of rendering may include one or more of various ways of performing vector-based amplitude panning (VBAP), one or more of various ways of performing binaural rendering (e.g., head-related transfer function (HRTF), binaural room impulse response (BRIR), etc.), and / or one or more of various ways of performing sound field synthesis.

[0066] The audio playback system 16 may output a speaker feed 39 to one or more speakers 40. The speaker feed 39 may drive the speakers 40. The speakers 40 may represent loudspeakers (e.g., transducers placed in a cabinet or other enclosure), headphone speakers, or any other type of transducer capable of producing sound based on electrical signals.

[0067] In order to select an appropriate renderer or, in some cases, generate an appropriate renderer, the audio playback system 16 may obtain loudspeaker information 41 indicating the number of speakers 40 and / or the spatial geometry of the speakers 40. In some cases, the audio playback system 16 may dynamically determine the speaker information 41 by using a reference microphone and driving the speakers 40 to obtain the loudspeaker information 41. In other cases, or in conjunction with the dynamic determination of the speaker information 41, the audio playback system 16 may prompt the user to interact with the audio playback system 16 and input the speaker information 41.

[0068] The audio playback system 16 may select one of the audio renderers 38 based on the speaker information 41. In some cases, when none of the audio renderers 38 is within a certain threshold similarity metric (in terms of loudspeaker geometry) specified in the speaker information 41, the audio playback system 16 may generate one of the audio renderers 38 based on the speaker information 41. In some cases, the audio playback system 16 may generate one of the audio renderers 38 based on the speaker information 41 without first attempting to select an existing one of the audio renderers 38.

[0069] Although described with respect to speaker feed 39, the audio playback system 16 may render a headphone feed from the speaker feed 39 or directly from the surround sound coefficients 11', thereby outputting the headphone feed to the headphone speakers. The headphone feed may represent a binaural audio speaker feed, which the audio playback system 16 renders using a binaural audio renderer.

[0070] As described above, the psychoacoustic audio encoding device 26 may execute an instance of the psychoacoustic audio encoder for each transmission channel of the ATF audio data 25, effectively creating multiple bitstreams that are multiplexed to form the overall bitstream 31, where these multiple bitstreams may be denoted as “sub-bitstreams” to distinguish these sub-bitstreams from the bitstream 31. The psychoacoustic audio encoding device 26 may output the bitstreams to the audio playback system 16, which in some cases may consume power from a power-constrained power source (e.g., a battery), such as when the audio playback system 16 represents a mobile device.

[0071] Given a power-constrained power source, the psychoacoustic audio decoding device 34 of the audio playback system 16 can execute a single instance of the psychoacoustic audio decoder to decode the bitstream 31 to obtain the ATF audio data 25'. The audio playback system 16 can execute a single instance of the psychoacoustic audio decoder to avoid excessive energy consumption by multiple instances of the psychoacoustic audio decoder. However, when decoding the complete 3D sound field represented by the ATF audio data 25', even a single instance of the psychoacoustic audio decoder may consume a large amount of power, so that multiple sub-bitstreams of the bitstream 31 may prevent a mobile device or other power-constrained device from decoding and rendering the ATF audio data 25'.

[0072] According to various aspects of the techniques described in this disclosure, the psychoacoustic audio decoding device 25' can obtain operating conditions of the audio playback system 16 used to decode the bitstream 31 (which can also be referred to as "encoded scene-based audio data 31"). That is, the psychoacoustic audio decoding device 25' can monitor the audio playback system 16 to identify operating conditions based on battery power and / or communication link status (e.g., a communication link status of a link configured to operate via a transmission channel between the content creator system 12 and the audio playback system 16). The psychoacoustic audio decoding device 34 can adapt the application of a single instance of the psychoacoustic audio decoder to the bitstream 31 based on the operating conditions to potentially reduce power consumption.

[0073] For illustration, assume that the state of the communication link is an operating condition, and the psychoacoustic audio decoding device 34 determines the state of the communication link based on a signal-to-noise ratio (SNR). When the psychoacoustic audio decoding device 34 determines that the SNR is below a threshold, the psychoacoustic audio decoding device 34 may call a single instance of a psychoacoustic audio decoder to perform psychoacoustic audio decoding for the bitstream 31 to decode a subset of bits representing the ATF audio data 25' (which is an example of intermediate formatted scene-based audio data). The subset of bits may include less than the total number of bits used to represent the ATF audio data 25.

[0074] By decoding only a subset of the bits representing the ATF audio data 25', the psychoacoustic audio decoding device 34 can reduce the number of operations performed by the underlying hardware, thereby reducing the number of processor cycles spent decoding the bitstream 31. By reducing the number of processor cycles spent decoding the bitstream 31, the psychoacoustic audio decoding device 34 can reduce power consumption (by the processor and any bus used to communicate the ATF audio data 25', considering that fewer bits decoded result in less bus bandwidth consumption). In addition, by reducing the bit depth of the ATF audio data 25', components downstream of the psychoacoustic audio decoding device 34, such as the spatial audio decoding device 36 and the audio renderer 38, can consume less power because the subset of bits used to represent the ATF audio data 25' requires fewer processor cycles to process in order to obtain the speaker feeds 29. In addition, the reduction in bit depth may not overly affect the perceived quality during playback, because a low SNR indicates that a link error may have occurred, which may mask the reduction in quality of the ATF audio data 25'.

[0075] Figure 2 is a diagram illustrating another example of a system that can perform various aspects of the techniques described in this disclosure. Figure 2 The system 110 may represent Figure 1 An example of system 10 is shown in the example of FIG. Figure 2 As shown in the example of , system 110 includes a source device 112 and a sink device 114 , where source device 112 may represent an example of content creator system 12 and sink device 114 may represent an example of content consumer 14 and / or audio playback system 16 .

[0076] Although described with respect to source device 112 and sink device 114, in some cases, source device 112 may operate as a sink device, and in these and other cases, sink device 114 may operate as a source device. Figure 2 The example of system 110 shown in FIG. 1 is merely one example to illustrate various aspects of the techniques described in this disclosure.

[0077] In any case, as described above, source device 112 may represent any form of computing device capable of implementing the techniques described in this disclosure, including a handheld (or cellular phone, including so-called "smart phones"), a tablet computer, a so-called smart phone, a remote-controlled aircraft (such as a so-called "drone"), a robot, a desktop computer, a receiver (such as an audio / video (AV) receiver), a set-top box, a television (including so-called "smart TV"), a media player (such as a digital video disc player, a streaming media player, a Blu-ray Disc™ player, etc.), or any other device capable of wirelessly communicating audio data to a sink device via a personal area network (PAN). For purposes of illustration, it is assumed that source device 112 represents a smart phone.

[0078] The sink device 114 may represent any form of computing device capable of implementing the techniques described in this disclosure, including a handset (or in other words, a cellular phone, a mobile phone, a mobile handset, etc.), a tablet computer, a smart phone, a desktop computer, a wireless headset (which may include wireless headsets with or without a microphone, as well as so-called smart wireless headsets that include additional functions such as health monitoring, onboard music storage and / or playback, dedicated cellular functions, etc.), a wireless speaker (including so-called "smart speakers"), a watch (including so-called "smart watches"), or any other device capable of reproducing a sound field based on audio data wirelessly transmitted via a PAN. In addition, for the purpose of illustration, it is assumed that the sink device 114 represents a wireless headset.

[0079] like Figure 2 As shown in the example of FIG. 1 , source device 112 includes one or more application programs (“apps”) 118A-118N (“app 118”), a mixing unit 120, an audio encoder 122 (which includes a spatial audio encoding device (SAED) 124 and a psychoacoustic audio encoding device (PAED) 126), and a wireless connection manager 128. Although not shown in FIG. Figure 2 As shown in the example of , source device 112 may include multiple other elements that support the operation of app 118, including an operating system, various hardware and / or software interfaces (such as user interfaces, including graphical user interfaces), one or more processors (which may be represented as "processor(s)"), memory, storage devices, etc.

[0080] Each of the apps 118 represents software (such as a collection of instructions stored to a non-transitory computer-readable medium) that configures the system 110 to provide certain functionality when executed by one or more processors of the source device 112. To name a few examples, the apps 118 may provide messaging functionality (such as access to email, text messaging, and / or video messaging), voice calling functionality, video conferencing functionality, calendaring functionality, audio streaming functionality, directions functionality, mapping functionality, gaming functionality. The apps 118 may be first-party applications designed and developed by the same company that designs and sells the operating system executed by the source device 112 (and typically pre-installed on the source device 112) or third-party applications accessible via a so-called “app store” or possibly pre-installed on the source device 112. Each app 118, when executed, may output audio data 119A-119N (“audio data 119”), respectively.

[0081] In some examples, audio data 119 may be received from a microphone (not shown, but similar to a microphone) connected to source device 112. Figure 1 The audio data 119 may include the above-mentioned Figure 1 The surround sound coefficients of the surround sound audio data 21 discussed in the example of FIG. 1 are similar, wherein such surround sound audio data may be referred to as “scene-based audio data”. Thus, the audio data 119 may also be referred to as “scene-based audio data 119” or “surround sound audio data 119”.

[0082] Although described with respect to surround sound audio data, these techniques may be performed with respect to surround sound audio data that does not necessarily include coefficients corresponding to so-called "higher-order" spherical basis functions (e.g., spherical basis functions having an order greater than 1). Thus, these techniques may be performed with respect to surround sound audio data that includes coefficients corresponding only to zero-order spherical basis functions, or only to zero-order and first-order spherical basis functions.

[0083] The mixing unit 120 represents a unit configured to mix one or more audio data 119 output by the app 118 (as well as other audio data output by the operating system - such as alarms or other tones, including keyboard key sounds, ringtones, etc.) to generate mixed audio data 121. Audio mixing can refer to the process of combining multiple sounds (as set forth in the audio data 119) into one or more channels. During mixing, the mixing unit 120 can also manipulate and / or enhance the volume level (which can also be referred to as "gain level"), frequency content, and / or panoramic position of the surround sound audio data 119. In the context of streaming the surround sound audio data 119 over a wireless PAN session, the mixing unit 120 can output the mixed audio data 121 to the audio encoder 122.

[0084] The audio encoder 122 can be used with the above Figure 1 That is, the audio encoder 122 may represent a unit configured to encode the mixed audio data 121 to obtain encoded audio data in the form of a bitstream 131. In some examples, the audio encoder 122 may encode individual audio data in the audio data 119.

[0085] For illustration purposes, referring to an example of a PAN protocol, A number of different types of audio codecs (a word that is a combination of the words "encode" and "decode") are provided, and can be extended to include vendor-specific audio codecs. The Advanced Audio Distribution Profile (A2DP) of MPEG-1 indicates that support for A2DP requires support for the sub-band codecs specified in A2DP. A2DP also supports codecs described in MPEG-1 Part 3 (MP2), MPEG-2 Part 3 (MP3), MPEG-2 Part 7 (Advanced Audio Codec - AAC), MPEG-4 Part 3 (High Efficiency - AAC (HE-AAC)), and Adaptive Transform Acoustic Codec (ATRAC). In addition, as mentioned above, A2DP supports vendor-specific codecs such as aptXTM and various other versions of aptX (e.g., Enhanced aptX (E-aptX), aptX Live, and aptX High Definition (aptX-HD)).

[0086] The audio encoder 122 may operate in accordance with one or more of the audio codecs listed above and one or more of the audio codecs not listed above, but operates to encode the mixed audio data 121 to obtain encoded audio data 131 (which is another way to refer to the bitstream 131). The audio encoder 122 may first call the SAED 124, which may communicate with Figure 1 The SAED 124 may be similar (if not substantially similar) to the SAED 24 shown in the example of FIG. 124 may perform the above-described spatial audio compression on the mixed audio data to obtain ATF audio data 125 (which is similar to the Figure 1 The SAED 124 may output the ATF audio data 25 to the PAED 126.

[0087] PAED 126 can be used with Figure 1The PAED 126 may be similar (if not substantially similar) to the PAED 26 shown in the example of FIG. The PAED 126 may perform psychoacoustic audio encoding according to any of the aforementioned codecs (including AptX and its variants) to obtain a bitstream 131. The audio encoder 122 may output the encoded audio data 131 to one of the wireless communication units 130 (e.g., the wireless communication unit 130A) managed by the wireless connection manager 128.

[0088] The wireless connection manager 128 may represent a unit configured to allocate bandwidth within certain frequencies of the available spectrum to different wireless communication units in the wireless communication unit 130. For example, The communication protocol operates in the 2.5 GHz spectrum range, which overlaps with the spectrum range used by various WLAN communication protocols. The wireless connection manager 128 can allocate certain portions of the bandwidth to the wireless devices during a given time. The wireless connection manager 128 may be configured to control the bandwidth allocation and other aspects of the wireless communication protocol 129 to achieve a specified quality of service (QoS). In other words, the wireless connection manager 128 may provide an API to adjust the scheme 129 by which the operation of the wireless communication unit 130 is controlled to achieve a specified QoS.

[0089] In other words, the wireless connection manager 128 can manage the coexistence of multiple wireless communication units 130 operating within the same spectrum, such as some WLAN communication protocols and some PAN protocols as described above. The wireless connection manager 128 may include a coexistence scheme 129 (in Figure 2 129 ), which indicates when (eg, intervals) each of the wireless communication units 130 can send packets and how many packets to send, the size of the packets sent, and so on.

[0090] The wireless communication units 130 may each represent a wireless communication unit 130 operating according to one or more communication protocols to communicate the bitstream 131 to the sink device 114 via a transmission channel. Figure 2 In the example of FIG. 1 , for the purpose of illustration, it is assumed that the wireless communication unit 130A is based on Assume further that the wireless communication unit 130A operates according to A2DP to establish a PAN link (via a transport channel) to allow the bitstream 131 to be transmitted from the source device 112 to the sink device 114 .

[0091] related For more information about the Bluetooth communication protocol suite, see the document entitled “Bluetooth Core Specification v 5.0” dated December 6, 2016, which is available at: www.bluetooth.org / en-us / specification / adopted-specifications. For more information about A2DP, see the document entitled “Advanced Audio Distribution Profile Specification” (version 1.3.1) dated July 14, 2015. Although assuming that the The wireless communication unit 130A operates using a suite of communication protocols, but the wireless communication unit 130A may operate using any form of wireless communication, including WiFi™, cellular communications (including so-called 3G, 4G and 5G cellular data communications), or any other form of wireless communication.

[0092] The wireless communication unit 130A may output the bit stream 131 to the sink device 114 via a transmission channel, which is assumed to be a wireless channel in the example of Bluetooth. Figure 2 131 is shown as being sent directly to sink device 114, but source device 112 may output bitstream 131 to an intermediate device located between source device 112 and sink device 114. The intermediate device may store bitstream 131 for later transmission to sink device 114, which may request bitstream 131. The intermediate device may include a file server, a network server, a desktop computer, a laptop computer, a tablet computer, a mobile phone, a smart phone, or any other device capable of storing bitstream 131 for later retrieval by an audio decoder. The intermediate device may be present in a content delivery network capable of streaming bitstream 131 (and possibly in conjunction with transmitting a corresponding video data bitstream) to a subscriber (such as sink device 114) requesting bitstream 131.

[0093] Alternatively, source device 112 may store bitstream 131 to a storage medium such as an optical disc, digital video disc, high-definition video disc, or other storage medium, most of which are capable of being read by a computer and may therefore be referred to as computer-readable storage medium or non-transitory computer-readable storage medium. In this context, transmission channels may refer to those channels through which content stored to these media is sent (and may include retail stores and other store-based delivery mechanisms). In any case, the technology of the present disclosure should not be limited in this regard. Figure 2 .

[0094] like Figure 2As further shown in the example of FIG. 1 , the sink device 114 includes a wireless connection manager 150 that manages one or more of wireless communication units 152A-152N (“wireless communication units 152”) according to a scheme 151, an audio decoder 132 (including a psychoacoustic audio decoding device (PADD) 134 and a spatial audio decoding device (SADD) 136), and one or more speakers 140A-140N (“speakers 140”, which may be similar to Figure 1 40 as shown in the example of FIG. 1 ). The wireless connection manager 150 may operate in a manner similar to that described above with respect to the wireless connection manager 128, exposing an API to coordinate a scheme 151 by which the operation of the wireless communication unit 152 achieves a specified QoS.

[0095] Wireless communication unit 152 may be similar in operation to wireless communication unit 130, except that wireless communication unit 152 operates inversely to wireless communication unit 130 to receive bit stream 131 via a transmission channel. Assume that one of wireless communication units 152 (e.g., wireless communication unit 152A) receives bit stream 131 according to The wireless communication unit 152A may output the bit stream 131 to the audio decoder 132.

[0096] The audio decoder 132 may operate in a reverse manner to the audio encoder 122. The audio decoder 132 may operate in accordance with one or more of the audio codecs listed above and one or more of the audio codecs not listed above, but operates to decode the encoded audio data 131 to obtain mixed audio data 121'. Again, the primes with respect to the "mixed audio data 121" indicate that there may be some losses due to quantization operations or other lossy operations that occurred during encoding by the audio encoder 122.

[0097] The audio decoder 132 may call the PADD 134 to perform psychoacoustic audio decoding on the bitstream 131 to obtain the ATF audio data 125', which the PADD 134 may output to the SADD 136. The SADD 136 may perform spatial audio decoding to obtain the mixed audio data 121'. Although for the purpose of illustration, Figure 2 The example does not show a renderer (similar to Figure 1 38), but the audio decoder 132 may render the mixed audio data 121' to the speaker feed (using any renderer, such as described above with respect to Figure 1 ) and outputs the speaker feed to one or more speakers 140.

[0098] Each speaker 140 represents a transducer configured to reproduce a sound field fed from the speaker. Figure 2 As shown in the example of , the transducer can be integrated within the sink device 114, or can be communicatively coupled to the sink device 114 (via wired or wireless). Speaker 140 can represent any form of speaker, such as a loudspeaker, an earphone speaker, or a speaker in an earbud. In addition, although described with respect to a transducer, speaker 140 can represent other forms of speakers, such as a "speaker" used in bone conduction headphones, which sends vibrations to the upper jaw, which induces sound in the human auditory system.

[0099] As described above, PADD 134 may monitor operating conditions of sink device 114 and thereby obtain operating conditions in terms of battery charge and / or status (as examples) of a communication link over a transmission channel between source device 112 and sink device 114. Based on the operating conditions, PADD 134 may adapt the operation of a single instance of a psychoacoustic audio decoder to potentially reduce power consumption.

[0100] To illustrate, consider another example in which the mixed audio data 121 includes ambient sound audio data defined according to ambient sound coefficients corresponding to spherical basis functions having a zero order up to a highest order (e.g., a second order, a third order, or a fourth order). Given the relationship with the spherical basis functions, the ambient sound audio data is defined in a spherical harmonic domain. In addition, even after undergoing a transformation using a linear reversible transform during spatial audio encoding, the resulting foreground audio signal and the corresponding spatial components exist in the spherical harmonic domain because the linear reversible transform does not transform the foreground audio signal and the spatial components between domains.

[0101] In any case, the PADD 134 can instantiate a single instance of the psychoacoustic audio decoder to perform psychoacoustic audio decoding on the bitstream 31 to decode the ATF audio data 25' associated with a subset of orders when the battery level, the state of the communication link, or some combination thereof is below a threshold. In some examples, the PADD 134 decodes only the ATF audio data 25' associated with a subset of orders, where the subset of orders is less than all orders of the spherical basis functions to which the ATF audio data 25' corresponds. The PADD 134 can output the reduced order ATF audio data 25' to the SADD 136, which can perform spatial audio decoding on the reduced order ATF audio data 25' to reconstruct or otherwise obtain the mixed audio data 121'.

[0102] By decoding only a subset of the orders of the spherical basis functions corresponding to the ATF audio data 25', the PADD 134 can again reduce the number of operations performed (and the associated processor cycles, bus bandwidth, etc.). In this way, due to the smaller number of operations performed, the PADD 134 can reduce power consumption, which can contribute to longer battery life, thereby trading quality for extended playback. Figure 3A and Figure 3B More information about the audio decoder 132 is described.

[0103] Figure 3A and Figure 3B is shown in more detail Figure 2 A block diagram of an example of an audio decoder. Figure 3A The audio decoder 132 shown in the example of FIG. 1 represents both the PADD 134 and the SADD 136, wherein the PADD 134 has been integrated with the SADD 136, which is Figure 3A In the example of , the renderer 38 is shown as a reconstruction unit 516. Furthermore, the renderer 38 is also shown as being integrated into the audio decoder 132.

[0104] like Figure 3A As further shown in the example of , the audio decoder 132 includes a monitoring unit 500, a performance mapper 502, a multiplexer ("mux") control unit 504, a multiplexer 506, an ATF metadata decoder 508, an AptX decoder 510, a buffer control unit 512, a buffer 514, a synchronization unit 516, and a reconstruction unit 518. The monitoring unit 500 represents a unit configured to monitor the audio playback device 16 for operating conditions 501A and / or 501B.

[0105] As an example, the monitoring unit 500 may monitor a communication link such as a Bluetooth (BT) link 499 or any other form of wireless and / or wired communication link to obtain operating conditions 501A, which may include an indication of signal quality (e.g., the above-mentioned SNR), bandwidth, loss, noise, or any other indication of the communication link. As another example, the monitoring unit 500 can monitor the battery or other power source of the audio playback device 16 to obtain operating conditions 501B, which may include an indication of battery charge (e.g., a percentage of remaining battery life), an expected battery duration (e.g., in units of time), and any other indication of the battery.

[0106] Performance mapper 502 may represent a unit configured to convert operating conditions 501A and 501B ("operating conditions 501") into priority levels 503. Performance mapper 502 may also obtain as input a latency requirement 505, which takes into account the conversion of operating conditions 501 into priority levels 503, increasing decoding priority (and possibly reducing quality by choosing to decode fewer bits or decoding ambient sound audio data associated with spherical basis functions having only a subset of all orders) when latency requirement 505 is low, and reducing decoding priority (and possibly improving quality by decoding a larger number of bits or more ambient sound audio data associated with spherical basis functions having a subset of orders and additional orders) when latency requirement 505 is high.

[0107] The performance mapper 502 may interface with a consumer or other user of the audio playback device 16 (eg, via one of the apps 118) to obtain the latency requirements 505 (in Figure 3A 505 ) or retrieve a predefined delay requirement 505 (which may be statically defined by one of the apps 118 as an example). The performance mapper 502 may output the priority level 503 to the multiplexer control unit 504 and the buffer control unit 512 .

[0108] The buffer control unit 512 may represent a unit configured to control storage of data to the buffer 514. The buffer control unit 512 may also determine which sub-bitstreams 509 of the bitstream 131 to decode from the bitstream 131 based on the priority level 503 in order to reconstruct the ATF audio data 25'. The buffer control unit 512 may output an indication 513 to each of the multiplexer control unit 504, the ATF metadata decoder 508, the AptX decoder 510, and the buffer 514, wherein the indication 513 identifies which sub-bitstream 509 to decode and buffer. The indication 513 identifies the order of the spherical basis function corresponding to the ambient sound coefficient representing the background audio signal, and one or more foreground audio signals to be obtained from the bitstream 131.

[0109] The multiplexer control unit 504 may represent a unit configured to control the multiplexer 506. The multiplexer control unit 504 may output a control signal 507 to control the multiplexer 506. The multiplexer control unit 504 may determine the control signal 507 based on the priority level 503, wherein the control signal 507 indicates which of the sub-bitstream-1 to sub-bitstream-N is output to the AptX decoder 510 (which may represent a decoder configured to decode each sub-bitstream according to the AptX decoding algorithm) as the sub-bitstream 509. The control signal 507 may also identify a number of bits (or in other words, a bit depth) for each sample of the background audio signal and the aforementioned audio signal identified by the indication 513 to be parsed from the bitstream 131.

[0110] Each sub-bitstream 509 may represent a separate encoded transmission channel of ATF audio data 25', which may include background ambient sound coefficients (or more generally, background audio signals representing background components of a sound field) or foreground audio signals (representing foreground components of a sound field). The multiplexer control unit 504 may also output a control signal 507 that configures the multiplexer 506 to output ATF metadata 511 (which may define spatial components for each foreground audio signal) to the ATF metadata decoder 508.

[0111] The ATF metadata decoder 508 may represent a unit configured to process the ATF metadata 511 to output the spatial component 515. The ATF metadata decoder 508 may perform the above-described Figure 1 The various operations of the spatial audio decoding device 36 are discussed in more detail in order to decode the spatial components 515 before outputting them to the synchronization unit 516.

[0112] As described above, the AptX decoder 510 may represent a unit configured to decode each sub-bitstream 509 to obtain a transmission channel 517. The AptX decoder 510 may implement a single instance of an AptX decompression algorithm that complies with one or more of the AptX codecs listed above, including E-AptX, AptX Live, AptX-HD, AptX Stereo, and the like.

[0113] In some examples, the AptX decoder 510 may execute a single instance of the AptX decompression algorithm to obtain a quantization gain and a quantization shape representing a transmission channel specified by the sub-bitstream 509. The AptX decoder 510 may obtain the number of coarse bits used during quantization of the gain (or in other words, energy) of the ambient sound transmission format audio data, which in this case is represented by the sub-bitstream 509. The AptX decoder 510 may also determine the number of fine bits used during quantization of the gain (or in other words, energy) of each sub-bitstream 509. The AptX decoder 510 may perform psychoacoustic audio decoding on the encoded scene-based audio data (which is represented by the sub-bitstream 509) to decode the number of coarse bits and the number of fine bits representing the energy (or in other words, gain) of each sub-bitstream 509, thereby obtaining the decompressed energy.

[0114] The single instance of the AptX decompression algorithm executed by the AptX decoder 510 may also dequantize the quantized shape. In some examples, the AptX decoder 510 may perform vector dequantization on the quantized shape to decompress the quantized shape and obtain a decompressed shape. The AptX decoder 510 may perform gain / shape synthesis on the decompressed energy and the decompressed shape to reconstruct the transmission channel 517. The AptX decoder 510 may output the transmission channel 517 to the buffer 514.

[0115] The synchronization unit 516 can receive the spatial components 515 and synchronize the output of the spatial components to align the spatial components 515 with the corresponding primary audio signal stored in the buffer 514. The synchronization unit 516 can coordinate the delivery of the spatial components 515 to the reconstruction unit 518 so that the spatial components 515 are correctly (in time) aligned with the AFT audio data 25', because the buffer 514 injects some delay into the decoding, which the synchronization unit 516 can correct by injecting delay in the spatial component encoding path. The synchronization unit 516 can output the spatial components 515 to the reconstruction unit 518 in time, and the buffer 514 delivers different parts of the transmission channel represented by the AFT audio data 25'.

[0116] The reconstruction unit 518 may represent a unit configured to reconstruct the mixed audio data 121' based on the spatial components 515 and the AFT audio data 25', as described in more detail above. The reconstruction unit 518 may output the mixed audio data 121' to the renderer 38, which may render the mixed audio data 121' to one or more speaker feeds. The renderer 38 may output the speaker feeds to the one or more speakers 103, which may reproduce the sound field represented by the mixed audio data 121' based on the speaker feeds.

[0117] Next reference Figure 3B For example, the audio decoding device 132' is similar to Figure 3A , except that the audio decoding device 132′ includes a generic decoder 550 that replaces the AptX decoder 510. The decoder 550 may represent any type of psychoacoustic audio decoder, such as the Unified Speech and Audio Coder denoted “USAC” proposed by the Moving Picture Experts Group (MPEG), the MPEG-H 3D Audio Codec Standard, the MPEG-I Immersive Audio Standard, or a proprietary standard, such as AptX™ (including various versions of AptX, such as Enhanced AptX—E-AptX, AptX Live, AptX Stereo, and AptX High Definition—AptX-HD), Advanced Audio Codec (AAC), Audio Codec 3 (AC-3), Apple Lossless Audio Codec (ALAC), MPEG-4 Audio Lossless Streaming (ALS), Enhanced AC-3, Free Lossless Audio Codec (FLAC), Monkey's Audio, MPEG-1 Audio Layer II (MP2), MPEG-1 Audio Layer III (MP3), Opus, and Windows Media Audio (WMA).

[0118] Figure 4 is shown in more detail Figure 3A and Figure 3B A block diagram of an example of a multiplexer control unit shown in the example of FIG. Figure 3A As described in the example of , the performance mapper 502 can obtain the operating conditions 501 and the delay requirements 505, and determine the priority level 503 based on one or more of the operating conditions 501 and the delay requirements 505. The performance mapper 502 may include a table or other data structure that receives one or more of the operating conditions 501 and the delay requirements 505 as one or more inputs. The table may include a lookup table that outputs the priority level 503 based on the one or more inputs. The performance mapper 502 can perform machine learning on the lookup table to develop the lookup table over time. As described above, the performance mapper 502 can output the priority level 503 to the multiplexer control unit 504.

[0119] like Figure 4As shown in the example of , the multiplexing control unit 504 may include a parameter mapping unit 600 and an adaptive segmentation unit 602. The parameter mapping unit 600 may represent a unit configured to map the priority level 503 to one or more parameters 601. The parameters 601 may include the number of bits to be retained from the sub-bitstream 509, the number of spherical basis function orders corresponding to the background audio signal, and the number of foreground audio signals to be considered. The parameter mapping unit 600 may output the parameters 601 to the adaptive segmentation unit 602.

[0120] The adaptive segmentation unit 602 may process the parameters 601 to obtain the control signal 507. That is, the adaptive segmentation unit 602 may output the control signal 507, which effectively indicates that when the priority level 503 (which indicates one or more of the operating condition 501 and the delay requirement 505) is below a threshold, only a subset of bits representing the ATF audio data 25 will be decoded (where the subset of bits is less than the total number of bits used to represent the AFT audio data 25). The multiplexer 506 may then output only the indicated subset of bits, causing the AptX decoder 510 to perform psychoacoustic audio decoding on the sub-bitstream 509 (which indicates the encoded scene-based audio data) to decode only the subset of bits representing the ATF audio data 25'.

[0121] In some examples, the parameter mapping unit 600 may determine the number of bits to be reserved from the sub-bitstream 509 as coarse bits for representing energy (or in other words, gain). When the priority level 503 is below a threshold, the parameter mapping unit 600 may output a parameter 601 to indicate that only coarse bits (instead of fine bits) are parsed from the sub-bitstream 509, thereby causing the AptX decoder 510 to perform psychoacoustic audio decoding only on the coarse bits (instead of fine bits) of the sub-bitstream 509. When the priority level 503 is greater than or equal to the threshold, the parameter mapping unit 600 may output a parameter 601 to indicate that both coarse bits and fine bits are to be parsed from the sub-bitstream 509, causing the AptX decoder 510 to perform psychoacoustic audio decoding on both coarse bits and fine bits (which are an example of additional bits) of the sub-bitstream 509.

[0122] In this regard, the audio decoder 132 can decode more or fewer bits by utilizing the layered structure of AptX (e.g., coarse and fine energy). When the priority level 503 is below a threshold (or in other words, "low"), the audio decoder 132 decodes only the coarse bits of energy. When the priority level is greater than or equal to the threshold (or in other words, "high"), the audio decoder 132 can decode both the coarse and fine bits of energy. In this way, the audio decoder 132 can decode the entire sound field by adjusting the decoding quality (where the quality is represented by the bit resolution).

[0123] In some cases, when the priority level 503 is less than a threshold, the audio decoder 132 may decode all bits of the selected subset of the ATF audio data 25' instead of decoding a subset of the bits. The parameter mapping unit 600 may identify the selected subset of the ATF audio data 25' and the subset of the foreground audio signal based on the order (e.g., zero or one) of the spherical basis functions corresponding to the ambient sound coefficients, thereby outputting a control signal 507 and one or more and possibly all of the foreground audio signals, the control signal 507 specifying that only a subset of the sub-bitstream 509 will be provided to the AptX decoder 510 (such as the first-order ambient sound coefficients represented as W, X, Y, and Z). In this case, when the operating condition is below the threshold, the AptX decoder 510 may perform psychoacoustic audio decoding on the sub-bitstream 509 to decode high-order ambient sound audio data corresponding to the spherical basis functions having a subset of orders that are less than all orders of the spherical basis functions corresponding to the sub-bitstream 509.

[0124] When the priority level 503 is greater than or equal to a threshold, the parameter mapping unit 600 may indicate that a sub-bitstream 509 corresponding to a spherical basis function having a subset of orders and at least one additional order (and possibly all orders) is to be output to the AptX decoder 510 for psychoacoustic audio decoding. In this regard, when the priority level 503 (which at least partially indicates the operating condition 501) is above the threshold, the AptX decoder 510 may perform psychoacoustic audio decoding on the sub-bitstream 509 to reconstruct ATF audio data 25' corresponding to a spherical basis function having a subset of orders and at least one additional order, the subset of orders being less than all orders of the spherical basis functions to which the ATF audio data 25 corresponds.

[0125] In this way, the audio decoder 132 can decode certain ambient sound orders and / or background / foreground audio signals by utilizing the hierarchical structure of the ambient sound (e.g., first, second, third, ..., Nth order). When the priority level 503 is lower than the threshold (or in other words, "low"), the audio decoder 132 only decodes the lower orders (e.g., first and second orders). When the priority level is greater than or equal to the threshold (or in other words, "high"), the audio decoder 132 can decode higher orders (e.g., third and fourth orders). In this way, the audio decoder 132 can decode the sound field by adjusting the spatial accuracy of the decoding while decoding all bits (sacrificing spatial resolution but maintaining quality).

[0126] In some cases, the parameter mapping unit 600 may generate parameters 601 indicating some combination of the two aforementioned different aspects of the technique. In this way, the AptX decoder 510 may perform psychoacoustic audio decoding on the sub-bitstream 509 to decode high-order ambient sound audio data corresponding to all orders up to the defined order while reconstructing a subset of bits or both a subset of bits and additional bits.

[0127] Figure 5 is shown in more detail Figure 3A and Figure 3B A block diagram of an example of a buffer control unit. Figure 3A As described in the example of , the performance mapper 502 can obtain the operating conditions 501 and the delay requirements 505, and determine the priority level 503 based on one or more of the operating conditions 501 and the delay requirements 505. The performance mapper 502 may include a table or other data structure that receives one or more of the operating conditions 501 and the delay requirements 505 as one or more inputs. The table may include a lookup table that outputs the priority level 503 based on the one or more inputs. The performance mapper 502 can perform machine learning on the lookup table to develop the lookup table over time. As described above, the performance mapper 502 can output the priority level 503 to the buffer control unit 512.

[0128] like Figure 5 As shown in the example of , the buffer control unit 512 may include a parameter mapping unit 700 and an adaptive segmentation unit 702. The parameter mapping unit 700 may represent a unit configured to map the priority level 503 to one or more parameters 701. The parameters 701 may include the number of spherical basis function orders corresponding to the background audio signal and the number of foreground audio signals to be considered. The parameter mapping unit 701 may output the parameters as an indication 513 to the adaptive segmentation unit 702 (although in Figure 5 , but is output to the multiplexer control unit 504, the AptX decoder 510, and the ATF metadata decoder 508).

[0129] The adaptive segmentation unit 702 may represent a unit that configures the buffer 514 to perform adaptive segmentation (or in other words, segmentation) of the transmission channel 517 according to the parameters 513. As such, various aspects of the technique may be performed during parsing of the bitstream 131 before AptX decoding, after AptX decoding, or some combination thereof.

[0130] With respect to the delay requirement 505, the buffer control unit 512 may configure the buffer 514 to reduce the delay so as to meet the delay requirement 505. To reduce the delay, the buffer control unit 512 may reduce the size of the buffer 514, which may result in buffering only low ambient sound levels (or, in other words, reducing the number of signals to be decoded / rendered). The delay requirement 505 may facilitate applications that require lower latency, such as in the context of virtual reality (or other types of extended reality, such as augmented reality or mixed reality) involving a head-mounted display. In some cases, the audio decoder 132 may perform time scaling to seamlessly transition from high latency to low latency.

[0131] Figure 6 is a block diagram illustrating different versions of ATF audio data after being processed by an audio decoder in accordance with various aspects of the present technology. Figure 6 In the example of , ATF audio data 25A' represents a version of the ATF audio data 25' of full quality or maximum quality, in which the audio decoder 132 decodes all bits of each sub-bitstream 509. ATF audio data 25B' represents a version of the ATF audio data 25' in which the audio decoder 132 decodes only the ambient sound coefficients corresponding to the spherical basis functions of the first order and below. ATF audio data 25C' represents a version of the ATF audio data 25' in which the audio decoder 132 decodes only a subset of the bits (e.g., coarse bits of energy) but decodes all sub-bitstreams 509 (which are represented by "SBS1c" to "SBSNc"). ATF audio data 25D' represents a version of the ATF audio data 25' in which the audio decoder 132 decodes only two of three or more foreground audio signals.

[0132] Figure 7 is a diagram showing various aspects of a system configured to perform the techniques described in this disclosure. Figure 2 The audio encoder 1000A may represent an example of a PAED 126, which may be configured to encode audio data for transmission over a personal area network or "PAN" (e.g., ) for transmission. However, the techniques of the present disclosure performed by the audio encoder 1000A may be used in any context where compression of audio data is desired. In some examples, the audio encoder 1000A may be configured to encode the audio data 17 according to an aptX™ audio codec (including, for example, enhanced aptX—E-APTx, aptX Live, and APTx HD).

[0133] exist Figure 7 In an example of , the audio encoder 1000A can be configured to encode the audio data 25 using a gain-shape vector quantization encoding process, which includes encoding and decoding the residual vector using a compact map. In the gain-shape vector quantization encoding process, the audio encoder 1000A is configured to encode both the gain (e.g., energy level) and shape (e.g., residual vector defined by transform coefficients) of a subband of frequency domain audio data. Each subband of the frequency domain audio data represents a certain frequency range of a particular frame of the audio data 25.

[0134] The audio data 25 can be sampled at a specific sampling frequency. Although any desired sampling frequency can be used, the example sampling frequency can include 48kHz or 44.1kHz. Each digital sample of the audio data 25 can be defined by a specific input bit depth, for example, 16 bits or 24 bits. In one example, the audio encoder 1000A can be configured to operate on a single channel (channel) of the audio data 21 (for example, mono audio). In another example, the audio encoder 1000A can be configured to encode two or more channels of the audio data 25 independently. For example, the audio data 17 can include a left channel and a right channel of stereo audio. In this example, the audio encoder 1000A can be configured to encode the left and right audio channels independently in a dual mono mode. In other examples, the audio encoder 1000A can be configured to encode two or more channels of the audio data 25 together (for example, in a joint stereo mode). For example, the audio encoder 1000A can perform certain compression operations by predicting one channel of the audio data 25 with another channel of the audio data 25.

[0135] Regardless of how the channels of the audio data 25 are arranged, the audio encoder 1000A obtains the audio data 25 and sends the audio data 25 to the transform unit 1100. The transform unit 1100 is configured to transform the frames of the audio data 25 from the time domain to the frequency domain to generate frequency domain audio data 1112. The frames of the audio data 25 can be represented by a predetermined number of audio data samples. In one example, the frames of the audio data 25 can be 1024 samples wide. Different frame widths can be selected based on the frequency transform used and the desired compression amount. The frequency domain audio data 1112 can be represented as transform coefficients, wherein the value of each transform coefficient represents the energy of the frequency domain audio data 1112 at a specific frequency.

[0136] In one example, the transform unit 1100 can be configured to transform the audio data 25 into frequency domain audio data 1112 using a modified discrete cosine transform (MDCT). MDCT is a "lapped" transform based on a type IV discrete cosine transform. MDCT is considered "lapped" because it processes data from multiple frames. That is, in order to perform a transform using MDCT, the transform unit 1100 can include a fifty percent overlapping window into subsequent frames of audio data. The overlapping nature of MDCT may be useful for data compression techniques such as audio coding because it can reduce artifacts of encoding and decoding at frame boundaries. The transform unit 1100 need not be limited to using MDCT, but can use other frequency domain transform techniques to transform the audio data 17 into frequency domain audio data 1112.

[0137] The subband filter 1102 separates the frequency domain audio data 1112 into subbands 1114. Each subband 1114 includes transform coefficients of the frequency domain audio data 1112 within a specific frequency range. For example, the subband filter 1102 can separate the frequency domain audio data 1112 into twenty different subbands. In some examples, the subband filter 1102 can be configured to separate the frequency domain audio data 1112 into subbands 1114 of uniform frequency ranges. In other examples, the subband filter 1102 can be configured to separate the frequency domain audio data 1112 into subbands 1114 of non-uniform frequency ranges.

[0138] For example, the subband filter 1102 can be configured to separate the frequency domain audio data 1112 into subbands 1114 according to a Bark scale. In general, the subbands of the Bark scale have frequency ranges that are perceptually equal in distance. That is, the subbands of the Bark scale are not equal in frequency range, but are equal in human auditory perception. In general, the subbands of lower frequencies have fewer transform coefficients because lower frequencies are more easily perceived by the human auditory system. Thus, compared with higher frequency subbands, the frequency domain audio data 1112 in the lower frequency subbands of the subband 1114 is less compressed by the audio encoder 1000A. Similarly, the higher frequency subbands of the subband 1114 can include more transform coefficients because higher frequencies are more difficult to be perceived by the human auditory system. Thus, compared with lower frequency subbands, the frequency domain audio 1112 in the data in the higher frequency subbands of the subband 1114 can be more compressed by the audio encoder 1000A.

[0139] The audio encoder 1000A can be configured to process each subband 1114 using a subband processing unit 1128. That is, the subband processing unit 1128 can be configured to process each subband separately. According to the technology of the present disclosure, the subband processing unit 1128 can be configured to perform a gain-shape vector quantization process using an extended range of coarse-fine quantization.

[0140] The gain-shape analysis unit 1104 may receive subbands 1114 as input. For each subband 1114, the gain-shape analysis unit 1104 may determine an energy level 1116 for each subband 1114. That is, each subband 1114 has an associated energy level 1116. The energy level 1116 is a scalar value in decibels (dB) that represents the total amount of energy (also referred to as gain) in the transform coefficients of a particular subband in the subbands 1114. The gain-shape analysis unit 1104 may separate the energy level 1116 of one of the subbands 1114 from the transform coefficients of the subband to produce a residual vector 1118. The residual vector 1118 represents the so-called "shape" of the subband. The shape of the subband may also be referred to as the spectrum of the subband.

[0141] The vector quantizer 1108 may be configured to quantize the residual vector 1118. In one example, the vector quantizer 1108 may quantize the residual vector using a quantization process to generate a residual ID 1124. Instead of quantizing each sample individually (e.g., scalar quantization), the vector quantizer 1108 may be configured to quantize a block of samples included in the residual vector 1118 (e.g., a shape vector). Any vector quantization technique approach may be used with the extended range coarse-fine energy quantization process.

[0142] In some examples, the audio encoder 1000A can dynamically allocate bits for encoding and decoding the energy level 1116 and the residual vector 1118. That is, for each subband 1114, the audio encoder 1000A can determine the number of bits allocated for energy quantization (e.g., by the energy quantizer 1106) and the number of bits allocated for vector quantization (e.g., by the vector quantizer 1108). The total number of bits allocated for energy quantization can be referred to as energy allocation bits. These energy allocation bits can then be allocated between the coarse quantization process and the fine quantization process.

[0143] The energy quantizer 1106 may receive the energy level 1116 of the subband 1114 and quantize the energy level 1116 of the subband 1114 into a coarse energy 1120 and a fine energy 1122 (which may represent one or more quantized fine residuals). This disclosure will describe the quantization process for one subband, but it should be understood that the energy quantizer 1106 may perform energy quantization on one or more of the subbands 1114 (including each subband 1114).

[0144] In general, the energy quantizer 1106 can perform a recursive two-step quantization process. The energy quantizer 1106 can first quantize the energy level 1116 with a first number of bits for a coarse quantization process to generate a coarse energy 1120. The energy quantizer 1106 can generate the coarse energy using a predetermined range of energy levels for quantization (e.g., a range defined by a maximum and minimum energy level). The coarse energy 1120 is close to the value of the energy level 1116.

[0145] The energy quantizer 1106 may then determine the difference between the coarse energy 1120 and the energy level 1116. This difference is sometimes referred to as a quantization error. The energy quantizer 1106 may then quantize the quantization error using a second number of bits in a fine quantization process to produce a fine energy 1122. The number of bits used for the fine quantization bits is determined by the total number of energy allocation bits minus the number of bits used for the coarse quantization process. When added together, the coarse energy 1120 and the fine energy 1122 represent the total quantized value of the energy level 1116. The energy quantizer 1106 may continue in this manner to produce one or more fine energies 1122.

[0146] The audio encoder 1000A may also be configured to encode the coarse energy 1120, the fine energy 1122, and the residual ID 1124 using a bitstream encoder 1110 to create the encoded audio data 31 (which is another way of referring to the bitstream 31). The bitstream encoder 1110 may be configured to use one or more entropy encoding processes to further compress the coarse energy 1120, the fine energy 1122, and the residual ID 1124. The entropy encoding process may include Huffman coding, arithmetic coding, context-adaptive binary arithmetic coding (CABAC), and other similar coding techniques.

[0147] In one example of the present disclosure, the quantization performed by the energy quantizer 1106 is uniform quantization. That is, the step size (also referred to as "resolution") of each quantization is equal. In some examples, the step size can be in decibels (dB). The step sizes of coarse quantization and fine quantization can be determined according to a predetermined range of quantized energy values ​​and the number of bits allocated for quantization, respectively. In one example, the energy quantizer 1106 performs uniform quantization on both coarse quantization (e.g., to generate coarse energy 1120) and fine quantization (e.g., to generate fine energy 1122).

[0148] Performing a two-step uniform quantization process is equivalent to performing a single uniform quantization process. However, by dividing the uniform quantization into two parts, the bits allocated to coarse quantization and fine quantization can be controlled independently. This can allow greater flexibility in bit allocation across energy and vector quantization, and can improve compression efficiency. Consider an M-level uniform quantizer, where M defines the number of levels (e.g., in dB) into which the energy level can be divided. M can be determined by the number of bits allocated for quantization. For example, the energy quantizer 1106 can use M1 levels for coarse quantization and M2 levels for fine quantization. This is equivalent to a single uniform quantizer using M1*M2 levels.

[0149] Figure 8 is shown in more detail Figure 1 - A block diagram of an embodiment of a psychoacoustic audio decoder of FIG. 3. Audio decoder 1002A may represent an example of an AptX decoder 510, which may be configured to decode audio signals received via a PAN (e.g., ) is used to decode audio data received by the audio decoder 1002A. However, the techniques of the present disclosure performed by the audio decoder 1002A can be used in any context where compression of audio data is desired. In some examples, the audio decoder 1002A can be configured to decode the audio data 21 according to an aptXTM audio codec (including, for example, enhanced aptX-E-aptX, APTx Live, and APTx HD). However, the techniques of the present disclosure can be used in any audio codec that is configured to perform quantization of audio data. According to the techniques of the present disclosure, the audio decoder 1002A can be configured to perform various aspects of the quantization process using tight mapping.

[0150] In general, the audio decoder 1002A can operate in an inverse manner relative to the audio encoder 1000A. As such, the same process used in the encoder for quality / bitrate scalable cooperative PVQ can be used in the audio decoder 1002A. Decoding is based on the same principles, using the reverse of the operations performed in the decoder so that audio data can be reconstructed from the encoded bitstream received from the encoder. Each quantizer has an associated inverse quantizer counterpart. For example, Figure 8 As shown, the inverse transform unit 1100', the inverse subband filter 1102', the gain-shape synthesis unit 1104', the energy dequantizer 1106', the vector dequantizer 1108' and the bitstream decoder 1110' can be respectively configured to perform relative to Figure 7 The inverse operations of the transform unit 1100, subband filter 1102, gain-shape analysis unit 1104, energy quantizer 1106, vector quantizer 1108 and bitstream encoder 1110.

[0151] Specifically, the gain-shape synthesis unit 1104' reconstructs the frequency domain audio data, which has a reconstructed residual vector and a reconstructed energy level. The inverse subband filter 1102' and the inverse transform unit 1100' output the reconstructed audio data 25'. In an example where the encoding is lossless, the reconstructed audio data 25' may perfectly match the audio data 25. In an example where the encoding is lossy, the reconstructed audio data 25' may not perfectly match the audio data 25.

[0152] Fig.9A and Fig. 9B is shown in more detail Figure 1-Figure 3B A block diagram of an additional example of a psychoacoustic audio encoder is shown in the example of FIG. Fig.9A For example, the audio encoder 1000B may be configured to encode the audio data for transmission via a PAN (eg, ) for transmission. However, likewise, the techniques of the present disclosure performed by the audio encoder 1000B may be used in any context where compression of audio data is desired. In some examples, the audio encoder 1000B may be configured to encode the audio data 25 according to an aptX audio codec TM (including, for example, enhanced aptX - E-APTx, aptX live, and APTx HD). However, the techniques of the present disclosure may be used in any audio codec. As will be explained in more detail below, the audio encoder 1000B may be configured to perform various aspects of perceptual audio coding and decoding according to various aspects of the techniques described in the present disclosure.

[0153] exist Fig.9A In an example of , the audio encoder 1000B can be configured to encode the audio data 25 using a gain-shape vector quantization encoding process. In the gain-shape vector quantization encoding process, the audio encoder 1000B is configured to encode both the gain (e.g., energy level) and shape (e.g., residual vector defined by transform coefficients) of a subband of frequency domain audio data. Each subband of the frequency domain audio data represents a certain frequency range of a particular frame of the audio data 25. In general, in the present disclosure, the term "subband" refers to a frequency range, a frequency band, etc.

[0154] The audio encoder 1000B invokes the transform unit 1100 to process the audio data 25. The transform unit 1100 is configured to process the audio data 25 at least in part by applying a transform to frames of the audio data 25 and thereby transform the audio data 25 from the time domain to the frequency domain to generate frequency domain audio data 1112.

[0155] A frame of audio data 25 may be represented by a predetermined number of audio data samples. In one example, a frame of audio data 25 may be 1024 samples wide. Different frame widths may be selected based on the frequency transform used and the desired amount of compression. The frequency domain audio data 1112 may be represented as transform coefficients, where the value of each transform coefficient represents the energy of the frequency domain audio data 1112 at a particular frequency.

[0156] In one example, the transform unit 1100 can be configured to transform the audio data 25 into frequency domain audio data 1112 using a modified discrete cosine transform (MDCT). MDCT is an "overlapped" transform based on a type IV discrete cosine transform. MDCT is considered "overlapped" because it processes data from multiple frames. That is, in order to perform the transform using MDCT, the transform unit 1100 can include a fifty percent overlapping window into subsequent frames of audio data. The overlapping nature of MDCT may be useful for data compression techniques such as audio coding because it can reduce artifacts of encoding and decoding at frame boundaries. The transform unit 1100 need not be limited to using MDCT, but can use other frequency domain transform techniques to transform the audio data 25 into frequency domain audio data 1112.

[0157] The subband filter 1102 separates the frequency domain audio data 1112 into subbands 1114. Each subband 1114 includes transform coefficients of the frequency domain audio data 1112 within a specific frequency range. For example, the subband filter 1102 can separate the frequency domain audio data 1112 into twenty different subbands. In some examples, the subband filter 1102 can be configured to separate the frequency domain audio data 1112 into subbands 1114 of uniform frequency ranges. In other examples, the subband filter 1102 can be configured to separate the frequency domain audio data 1112 into subbands 1114 of non-uniform frequency ranges.

[0158] For example, the subband filter 1102 can be configured to separate the frequency domain audio data 1112 into subbands 1114 according to the Bark scale. In general, the subbands of the Bark scale have frequency ranges that are perceptually equal. That is, the subbands of the Bark scale are not equal in terms of frequency range, but are equal in human auditory perception. In general, lower frequency subbands have fewer transform coefficients because lower frequencies are more easily perceived by the human auditory system.

[0159] As such, the frequency domain audio data 1112 in the lower frequency sub-bands of the sub-band 1114 may be less compressed by the audio encoder 1000B than the higher frequency sub-bands. Likewise, the higher frequency sub-bands of the sub-band 1114 may include more transform coefficients because higher frequencies are more difficult for the human auditory system to perceive. As such, the frequency domain audio 1112 in the data in the higher frequency sub-bands of the sub-band 1114 may be more compressed by the audio encoder 1000B than the lower frequency sub-bands.

[0160] The audio encoder 1000B may be configured to process each subband 1114 using a subband processing unit 1128. That is, the subband processing unit 1128 may be configured to process each subband individually. The subband processing unit 1128 may be configured to perform a gain-shape vector quantization process.

[0161] The gain-shape analysis unit 1104 may receive subbands 1114 as input. For each subband 1114, the gain-shape analysis unit 1104 may determine an energy level 1116 for each subband 1114. That is, each subband 1114 has an associated energy level 1116. The energy level 1116 is a scalar value in decibels (dB) that represents the total amount of energy (also referred to as gain) in the transform coefficients of a particular one of the subbands 1114. The gain-shape analysis unit 1104 may separate the energy level 1116 of one of the subbands 1114 from the transform coefficients of the subband to produce a residual vector 1118. The residual vector 1118 represents the so-called "shape" of the subband. The shape of the subband may also be referred to as the spectrum of the subband.

[0162] The vector quantizer 1108 may be configured to quantize the residual vector 1118. In one example, the vector quantizer 1108 may quantize the residual vector using a quantization process to produce a residual ID 1124. Instead of quantizing each sample individually (e.g., scalar quantization), the vector quantizer 1108 may be configured to quantize a block of samples included in the residual vector 1118 (e.g., a shape vector).

[0163] In some examples, the audio encoder 1000B can dynamically allocate bits for encoding and decoding the energy level 1116 and the residual vector 1118. That is, for each subband 1114, the audio encoder 1000B can determine the number of bits allocated for energy quantization (e.g., by the energy quantizer 1106) and the number of bits allocated for vector quantization (e.g., by the vector quantizer 1108). The total number of bits allocated for energy quantization can be referred to as energy allocation bits. These energy allocation bits can then be allocated between the coarse quantization process and the fine quantization process.

[0164] The energy quantizer 1106 may receive the energy level 1116 of the subband 1114 and quantize the energy level 1116 of the subband 1114 into a coarse energy 1120 and a fine energy 1122. This disclosure will describe the quantization process for one subband, but it should be understood that the energy quantizer 1106 may perform energy quantization on one or more of the subbands 1114 (including each subband 1114).

[0165] like Fig.9AAs shown in the example of , the energy quantizer 1106 may include a prediction / difference (“P / D”) unit 1130, a coarse quantization (“CQ”) unit 1132, a summation unit 1134, and a fine quantization (“FQ”) unit 1136. The P / D unit 1130 may predict or otherwise identify a difference between an energy level 1116 of one of the sub-bands 1114 and another of the sub-bands 1114 representing the same frame of audio data (which may refer to a spatial (in the frequency domain) prediction) or a difference between energy levels 1116 of the same (or possibly different) one of the sub-bands 1114 from different frames (which may refer to a temporal prediction). The P / D unit 1130 may analyze the energy levels 1116 in this manner to obtain a predicted energy level 1131 (“PEL 1131”) for each sub-band 1114. The P / D unit 1130 may output the predicted energy level 1131 to the coarse quantization unit 1132.

[0166] The coarse quantization unit 1132 may represent a unit configured to perform coarse quantization on the predicted energy level 1131 to obtain the coarse energy 1120. The coarse quantization unit 1132 may output the coarse energy 1120 to the bitstream encoder 1110 and the summing unit 1134. The summing unit 1134 may represent a unit configured to obtain the difference between the coarse quantization unit 1134 and the predicted energy level 1131. The summing unit 1134 may output the difference as an error 1135 (which may also be referred to as a "residual 1135") to the fine quantization unit 1135.

[0167] The fine quantization unit 1132 may represent a unit configured to perform fine quantization on the error 1135. The fine quantization may be considered "fine" relative to the coarse quantization performed by the coarse quantization unit 1132. That is, the fine quantization unit 1132 may further quantize the error 1135 by quantizing according to a step size having a higher resolution than the step size used when performing the coarse quantization. As a result of performing the fine quantization on the error 1135, the fine quantization unit 1136 may obtain a fine energy 1122 for each subband 1122. The fine quantization unit 1136 may output the fine energy 1122 to the bitstream encoder 1110.

[0168] In general, the energy quantizer 1106 can perform a multi-step quantization process. The energy quantizer 1106 can first quantize the energy level 1116 with a first number of bits for a coarse quantization process to generate a coarse energy 1120. The energy quantizer 1106 can generate the coarse energy using a predetermined range of energy levels for quantization (e.g., a range defined by a maximum and minimum energy level). The coarse energy 1120 is close to the value of the energy level 1116.

[0169] The energy quantizer 1106 can then determine the difference between the coarse energy 1120 and the energy level 1116. This difference is sometimes referred to as a quantization error (or residual). The energy quantizer 1106 can then use a second number of bits to quantize the quantization error in a fine quantization process to produce a fine energy 1122. The number of bits used for the fine quantization bits is determined by the total number of energy allocation bits minus the number of bits used for the coarse quantization process. When added together, the coarse energy 1120 and the fine energy 1122 represent the total quantized value of the energy level 1116.

[0170] The audio encoder 1000B may also be configured to encode the coarse energy 1120, the fine energy 1122, and the residual ID 1124 using the bitstream encoder 1110 to create the encoded audio data 21. The bitstream encoder 1110 may be configured to further compress the coarse energy 1120, the fine energy 1122, and the residual ID 1124 using one or more of the above-described entropy encoding processes.

[0171] According to aspects of the present disclosure, energy quantizer 1106 (and / or components thereof, such as fine quantization unit 1136) can implement a hierarchical rate control mechanism to provide a greater degree of scalability and enable seamless or substantially seamless real-time streaming. For example, according to aspects of the present disclosure, fine quantization unit 1136 can implement a hierarchical fine quantization scheme. In some examples, fine quantization unit 1136 calls multiplexer (or "MUX") 1137 to implement a selection operation of hierarchical rate control.

[0172] The term "coarse quantization" refers to the combined operation of the above two-step coarse-fine quantization process. According to various aspects of the present disclosure, the fine quantization unit 1136 can perform one or more additional iterations of fine quantization on the error 1135 received from the summing unit 1134. The fine quantization unit 1136 can use a multiplexer 1137 to switch between and traverse various fine (finer) energy levels.

[0173] Hierarchical rate control may refer to a tree-based fine quantization structure or a cascaded fine quantization structure. When viewed as a tree-based structure, the existing two-step quantization operation forms the root node of the tree, and the root node is described as having a resolution depth of one (1). In accordance with the techniques of the present disclosure, the multiplexer 1137 may select additional level(s) of fine quantization, depending on the availability of bits for further fine quantization. Relative to the tree-based structure representing the multi-level fine quantization techniques of the present disclosure, any such subsequent fine quantization levels selected by the multiplexer 1137 represent two (2), three (3), etc., resolution depths.

[0174] The fine quantization unit 1136 can provide improved scalability and control for seamless real-time streaming scenarios in wireless PANs. For example, the fine quantization unit 1136 can replicate the hierarchical fine quantization scheme and quantization multiplexing tree at a higher level, seeding at the coarse quantization point of the more general decision tree. In addition, the fine quantization unit 1136 can enable the audio encoder 1000B to achieve seamless or substantially seamless real-time compression and streaming navigation. For example, the fine quantization unit 1136 can implement a multi-root hierarchical decision structure for multiple levels of fine quantization, thereby enabling the energy quantizer 1106 to potentially implement several iterations of fine quantization using the total available bits.

[0175] The fine quantization unit 1136 can implement the hierarchical rate control process in a variety of ways. The fine quantization unit 1136 can call the multiplexer 1137 on a per-subband basis to independently multiplex (and thereby select a corresponding tree-based quantization scheme) the error information 1135 associated with each of the subbands 1114. That is, in these examples, the fine quantization unit 1136 performs a multiplexing-based hierarchical quantization scheme selection for each corresponding subband 1114 independently of the quantization scheme selection for any other subband 1114. In these examples, the fine quantization unit 1136 quantizes each subband 1114 according to a target bit rate specified only for the corresponding subband 1114. In these examples, the audio encoder 1000B can signal the details of the specific hierarchical quantization scheme for each subband 1114 as part of the encoded audio data 21.

[0176] In other examples, the fine quantization unit 1136 may call the multiplexer 1137 only once and thereby select a single multiplexing-based quantization scheme for the error 1135 information associated with all subbands 1114. That is, in these examples, the fine quantization unit 1136 quantizes the error information 1135 associated with all subbands 1114 according to the same target bit rate, which is selected once and uniformly defined for all subbands 1114. In these examples, the audio encoder 1000B may signal the details of the single hierarchical quantization scheme applied across all subbands 1114 as part of the encoded audio data 21.

[0177] Next reference Fig. 9B For example, the audio encoder 1000C may represent Figure 1 and Figure 2 Another example of a psychoacoustic audio encoding device 26 and / or 126 is shown in the example of FIG. The audio encoder 1000C is similar to Fig.9AIn the example shown in FIG. 1 , the audio encoder 1000B includes a general analysis unit 1148, a quantization controller unit 1150, a general quantizer 1156, and a cognitive / perceptual / auditory / psychoacoustic (CPHP) quantizer 1160, and the general analysis unit 1148 can perform gain synthesis analysis or any other type of analysis to output a level 1149 and a residual 1151.

[0178] The general analysis unit 1148 may receive the subbands 1114 and perform any type of analysis to generate levels 1149 and residuals 1151. The general analysis unit 1148 may output the levels 1149 to the quantization controller unit 1150 and the residuals 1151 to the CPHP quantizer 1160.

[0179] Quantization controller unit 1150 may receive level 1149. Fig. 9B As shown in the example of , the quantization controller unit 1150 may include a layer specification unit 1152 and a specification control (SC) manager unit 1154. In response to receiving the level 1149, the quantization controller unit 1150 may call the layer specification unit 1152, which may perform top-down or bottom-up layer specification. Fig.11 is a diagram showing an example of top-down quantization. Fig.12 1 is a diagram illustrating an example of bottom-up quantization. That is, the hierarchical normalization unit 1152 may switch back and forth between coarse quantization and fine quantization on a frame-by-frame basis to implement a requantization mechanism that may make any given quantization coarser or finer.

[0180] From a coarse state to a finer state, the transition may occur by requantizing the previous quantization error. Alternatively, quantization may occur such that adjacent quantization points are grouped together into a single quantization point (moving from a fine state to a coarse state). Such an implementation may use a sequence data structure, such as a linked list or a richer structure, such as a tree or graph. Thus, the hierarchical specification unit 1152 may determine whether to switch from fine quantization to coarse quantization or from coarse quantization to fine quantization, providing the hierarchical space 1153 (which is the set of quantization points for the current frame) to the SC manager unit 1154. The hierarchical specification unit 1152 may determine whether to switch between finer or coarser quantization based on any information used to perform the fine or coarse quantization specified above (e.g., temporal or spatial priority information).

[0181] The SC manager unit 1154 may receive the layered space 1153 and generate canonical metadata 1155, passing an indication 1159 of the layered space 1153 along with the canonical metadata 1155 to the bitstream encoder 1110. The SC manager unit 1154 may also output the layered specification 1159 to the quantizer 1156, which may perform quantization according to the layered space 1159 for the levels 1149 to obtain quantized levels 1157. The quantizer 1156 may output the quantized levels 1157 to the bitstream encoder 1110, which may operate as described above to form the encoded audio data 31.

[0182] The CPHP quantizer 1160 may perform one or more of cognitive, perceptual, auditory, psychoacoustic coding on the residual 1151 to obtain a residual ID 1161. The CPHP quantizer 1160 may output the residual ID 1161 to the bitstream encoder 1110, which may operate as described above to form the encoded audio data 31.

[0183] Fig. 10A and Fig. 10B is shown in more detail Figure 1-Figure 3B A block diagram of an additional example of a psychoacoustic audio decoder is provided. Fig. 10A In the example of Figure 3A 1203. The audio decoder 1002B includes an extraction unit 1232, a subband reconstruction unit 1234, and a reconstruction unit 1236. The extraction unit 1232 may represent a unit configured to extract the coarse energy 1120, the fine energy 1122, and the residual ID 1124 from the encoded audio data 31. The extraction unit 1232 may extract one or more of the coarse energy 1120, the fine energy 1122, and the residual ID 1124 based on the energy bit allocation 1203. The extraction unit 1232 may output the coarse energy 1120, the fine energy 1122, and the residual ID 1124 to the subband reconstruction unit 1234.

[0184] The subband reconstruction unit 1234 may be configured to Fig.9A The subband reconstruction unit 1234 is a unit that operates in a manner opposite to the operation of the subband processing unit 1128 of the audio encoder 1000B shown in the example of FIG. In other words, the subband reconstruction unit 1234 can reconstruct the subband from the coarse energy 1120, the fine energy 1122, and the residual ID 1124. The subband reconstruction unit 1234 may include an energy dequantizer 1238, a vector dequantizer 1240, and a subband synthesizer 1242.

[0185] The energy dequantizer 1238 may be represented as being configured to Fig.9A A unit that performs inverse quantization in the opposite manner to the quantization performed by the energy quantizer 1106 shown. The energy inverse quantizer 1238 may perform inverse quantization on the coarse energy 1122 and the fine energy 1122 to obtain a prediction / difference energy level, and the energy inverse quantizer 1238 may perform an inverse prediction or difference calculation to obtain the energy level 1116. The energy inverse quantizer 1238 may output the energy level 1116 to the subband synthesizer 1242.

[0186] If the encoded audio data 31 includes a syntax element set to a value indicating that the fine energies 1122 are hierarchically quantized, the energy dequantizer 1238 may hierarchically dequantize the fine energies 1122. In some examples, the encoded audio data 31 may include a syntax element indicating whether the hierarchically quantized fine energies 1122 are formed using the same hierarchical quantization structure across all subbands 1114, or whether the corresponding hierarchical quantization structure is determined separately for each subband 1114. Based on the value of the syntax element, the energy dequantizer 1238 may apply the same hierarchical dequantization structure across all subbands 1114 represented by the fine energies 1122, or may update the hierarchical dequantization structure on a per-subband basis when dequantizing the fine energies 1122.

[0187] The vector dequantizer 1240 may represent a unit configured to perform vector dequantization in a manner opposite to the vector quantization performed by the vector quantizer 1108. The vector dequantizer 1240 may perform vector dequantization on the residual ID 1124 to obtain the residual vector 1118. The vector dequantizer 1240 may output the residual vector 1118 to the subband synthesizer 1242.

[0188] The subband synthesizer 1242 may represent a unit configured to operate in an inverse manner to the gain-shape analysis unit 1104. As such, the subband synthesizer 1242 may perform an inverse gain-shape analysis on the energy levels 1116 and the residual vectors 1118 to obtain the subbands 1114. The subband synthesizer 1242 may output the subbands 1114 to the reconstruction unit 1236.

[0189] The reconstruction unit 1236 may represent a unit configured to reconstruct the audio data 25' based on the subbands 1114. In other words, the reconstruction unit 1236 may perform inverse subband filtering in an inverse manner to the subband filtering applied by the subband filters 1102 to obtain the frequency domain audio data 1112. The reconstruction unit 1236 may then perform an inverse transform in an inverse manner to the transform applied by the transform unit 1100 to obtain the audio data 25'.

[0190] Next reference Fig. 10B For example, the audio decoder 1002C may represent Figure 1 and / or Figure 2 1002C may be similar to the audio decoder 1002B, except that the audio decoder 1002C may include an abstraction control manager 1250, a hierarchical abstraction unit 1252, an inverse quantizer 1254, and a CPHP inverse quantizer 1256.

[0191] The abstract control manager 1250 and the hierarchical abstraction unit 1252 may form an inverse quantizer controller 1249 that controls the operation of the inverse quantizer 1254, operating inversely to the quantizer controller 1150. As such, the abstract control manager 1250 may operate inversely to the SC manager unit 1154, receiving the metadata 1155 and the hierarchical specification 1159. The abstract control manager 1250 processes the metadata 1155 and the hierarchical specification 1159 to obtain the hierarchical space 1153, which the abstract control manager 1250 outputs to the hierarchical abstraction unit 1252. The hierarchical abstraction unit 1252 may operate inversely to the hierarchical specification unit 1152, processing the hierarchical space 1153 to output an indication 1159 of the hierarchical space 1153 to the inverse quantizer 1254.

[0192] The inverse quantizer 1254 may operate inversely to the quantizer 1156, where the inverse quantizer 1254 may inverse quantize the quantized levels 1157 using the indication 1159 of the hierarchical space 1153 to obtain the inverse quantized levels 1149. The inverse quantizer 1254 may output the inverse quantized levels 1149 to the subband synthesizer 1242.

[0193] The extraction unit 1232 may output the residual ID 1161 to a CPHP dequantizer 1256, which may operate inversely to the CPHP quantizer 1160. The CPHP dequantizer 1256 may process the residual ID 1161 to dequantize the residual ID 1161 and obtain the residual 1161. The CPHP dequantizer 1256 may output the residual to a subband synthesizer 1242, which may process the residual 1151 and the dequantized level 1254 to output the subband 1114. The reconstruction unit 1236 may operate as described above, converting the subband 1114 into audio data 25' by applying an inverse subband filter to the subband 1114 and then applying an inverse transform to the output of the inverse subband filter.

[0194] Fig.13 It is shown Figure 2 A block diagram of example components of a source device is shown in the example of FIG. Fig.13, source device 112 includes processor 412 (which may be referred to as "one or more processors" or "processor(s)"), graphics processing unit (GPU) 414, system memory 416, display processor 418, one or more integrated speakers 140, display 103, user interface 420, antenna 421, and transceiver module 422. In examples where source device 112 is a mobile device, display processor 418 is a mobile display processor (MDP). In some examples, such as examples where source device 112 is a mobile device, processor 412, GPU 414, and display processor 418 may be formed as an integrated circuit (IC).

[0195] For example, an IC may be considered a processing chip within a chip package and may be a system on chip (SoC). In some examples, two of processor 412, GPU 414, and display processor 418 may be housed together in the same IC, and the other may be housed in a different integrated circuit (i.e., a different chip package), or all three may be housed in different ICs or on the same IC. However, in an example where source device 12 is a mobile device, processor 412, GPU 414, and display processor 418 may all be housed in different integrated circuits.

[0196] Examples of processor 412, GPU 414, and display processor 418 include, but are not limited to, one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Processor 412 may be a central processing unit (CPU) of source device 12. In some examples, GPU 414 may be dedicated hardware that includes integrated and / or discrete logic circuits that provide GPU 414 with massively parallel processing capabilities suitable for graphics processing. In some cases, GPU 414 may also include general-purpose processing capabilities and may be referred to as a general-purpose GPU (GPGPU) when performing general-purpose processing tasks (i.e., non-graphics-related tasks). Display processor 418 may also be specialized integrated circuit hardware that is designed to retrieve image content from system memory 416, combine the image content into image frames, and output the image frames to display 103.

[0197] The processor 412 may execute various types of applications 20. Examples of applications 20 include web browsers, email applications, spreadsheets, video games, other applications that generate visual objects for display, or any of the types of applications listed in more detail above. The system memory 416 may store instructions for executing the applications 20. Executing one of the applications 20 on the processor 412 causes the processor 412 to generate graphics data for image content to be displayed and audio data 21 to be played (possibly via the integrated speakers 105). The processor 412 may transmit the graphics data for the image content to the GPU 414 and for further processing based on instructions or commands sent by the processor 412 to the GPU 414.

[0198] Processor 412 may communicate with GPU 414 according to a particular application processing interface (API). Examples of such APIs include of Khronos Group or OpenGL and OpenCL TM However, aspects of the present disclosure are not limited to the APIs of DirectX, OpenGL or OpenCL, and can be extended to other types of APIs. In addition, the techniques described in the present disclosure do not need to work according to the API, and the processor 412 and the GPU 414 can communicate using any technology.

[0199] System memory 416 may be memory for source device 12. System memory 416 may include one or more computer-readable storage media. Examples of system memory 416 include, but are not limited to, random access memory (RAM), electrically erasable programmable read-only memory (EEPROM), flash memory, or other media that may be used to carry or store desired program code in the form of instructions and / or data structures that may be accessed by a computer or processor.

[0200] In some examples, system memory 416 may include instructions that cause processor 412, GPU 414, and / or display processor 418 to perform the functions attributed in this disclosure to processor 412, GPU 414, and / or display processor 418. Thus, system memory 416 may be a computer-readable storage medium having stored thereon instructions that, when executed, cause one or more processors (e.g., processor 412, GPU 414, and / or display processor 418) to perform various functions.

[0201] System memory 416 may include non-transitory storage media. The term "non-transitory" means that the storage medium is embodied in a carrier wave or propagating signal. However, the term "non-transitory" should not be interpreted as meaning that system memory 416 is non-removable or that its contents are static. As an example, system memory 416 may be removed from source device 12 and moved to another device. As another example, a memory (substantially similar to system memory 416) may be inserted into source device 12. In some examples, non-transitory storage media may store data that may change over time (e.g., in RAM).

[0202] The user interface 420 may represent one or more hardware or virtual (meaning a combination of hardware and software) user interfaces through which a user may interact with the source device 12. The user interface 420 may include physical buttons, switches, toggles, lights, or virtual versions thereof. The user interface 420 may also include a physical or virtual keyboard, a touch interface (such as a touch screen), tactile feedback, and the like.

[0203] The processor 412 may include one or more hardware units (including so-called "processing cores") configured to perform all or part of the operations discussed above with respect to one or more of the mixing unit 120, the audio encoder 122, the wireless connection manager 128, and the wireless communication unit 130. The antenna 421 and the transceiver module 422 may represent a unit configured to establish and maintain a wireless connection between the source device 12 and the sink device 114. The antenna 421 and the transceiver module 422 may represent one or more receivers and / or one or more transmitters capable of wireless communication according to one or more wireless communication protocols. That is, the transceiver module 422 may represent a separate transmitter, a separate receiver, a separate transmitter and a separate receiver, or a combined transmitter and receiver. The antenna 421 and the transceiver 422 may be configured to receive encoded audio data that has been encoded according to the techniques of the present disclosure. Similarly, the antenna 421 and the transceiver 422 may be configured to send encoded audio data that has been encoded according to the techniques of the present disclosure. The transceiver module 422 may perform all or part of the operations of one or more of the wireless connection manager 128 and the wireless communication unit 130 .

[0204] Fig.14 It is shown Figure 2 Although sink device 114 may include components similar to those described above with respect to Fig.13 The examples of source device 112 and source device 114 are discussed in greater detail, but in some cases, sink device 14 may include only a subset of the components discussed above with respect to source device 112.

[0205] exist Fig.14 In the example of , sink device 114 includes one or more speakers 802, processor 812, system memory 816, user interface 820, antenna 821, and transceiver module 822. Processor 812 can be similar to or substantially similar to processor 812. In some cases, processor 812 can be different from processor 412 in terms of total processing power or can be tailored for low power consumption. System memory 816 can be similar to or substantially similar to system memory 416. Speaker 140, user interface 820, antenna 821, and transceiver module 822 can be similar to or substantially similar to corresponding speaker 440, user interface 420, and transceiver module 422. Sink device 114 can also optionally include display 800, although display 800 can represent a low-power, low-resolution (perhaps black and white LED) display through which limited information is conveyed, which can be driven directly by processor 812.

[0206] The processor 812 may include one or more hardware units (including so-called "processing cores") configured to perform all or part of the operations discussed above with respect to one or more of the wireless connection manager 150, the wireless communication unit 152, and the audio decoder 132. The antenna 821 and the transceiver module 822 may represent units configured to establish and maintain a wireless connection between the source device 112 and the sink device 114. The antenna 821 and the transceiver module 822 may represent one or more receivers and one or more transmitters capable of wireless communication according to one or more wireless communication protocols. The antenna 821 and the transceiver 822 may be configured to receive encoded audio data that has been encoded according to the techniques of the present disclosure. Similarly, the antenna 821 and the transceiver 822 may be configured to send encoded audio data that has been encoded according to the techniques of the present disclosure. The transceiver module 822 may perform all or part of the operations of one or more of the wireless connection manager 150 and the wireless communication unit 152.

[0207] Fig.15 is a diagram showing various aspects of the technology described in this disclosure Figure 2 Flowchart of example operations of an audio decoder of the present invention. The audio decoder 132 may first obtain an operating condition of a sink device 114 configured to decode encoded scene-based audio data (represented as a bitstream 131) (900). The audio decoder 132 may then, based on the operating condition, perform psychoacoustic audio decoding on the encoded scene-based audio data 131 to obtain ambient sound transmission format audio data 125' (902). The audio decoder 132 may then perform spatial audio decoding on the ambient sound transmission format audio data 125' to obtain scene-based audio data 121' (904).

[0208] The foregoing aspects of these techniques may be implemented in accordance with the following clauses.

[0209] Item 1A. A device configured to decode encoded scene-based audio data, the device comprising: a memory configured to store the encoded scene-based audio data; and one or more processors configured to: obtain an operating condition of the device for decoding the encoded scene-based audio data; based on the operating condition, perform psychoacoustic audio decoding on the encoded scene-based audio data to obtain ambient sound transmission format audio data; and perform spatial audio decoding on the ambient sound transmission format audio data to obtain scene-based audio data.

[0210] Clause 2A. The device of clause 1A, wherein the one or more processors are configured to: based on operating conditions, perform psychoacoustic audio decoding according to an AptX compression algorithm on the encoded scene-based audio data to obtain ambient sound transmission format audio data.

[0211] Clause 3A. The apparatus of any combination of clauses 1A and 2A, wherein the one or more processors perform a single instance of psychoacoustic audio decoding on the encoded scene-based audio data to obtain ambient sound transport format audio data.

[0212] Clause 4A. A device as described in any combination of clauses 1A-3A, wherein the one or more processors are further configured to: obtain a delay requirement for decoding the encoded scene-based audio data; and wherein the one or more processors are configured to perform psychoacoustic audio decoding on the encoded scene-based audio data based on operating conditions and the delay requirement to obtain ambient sound transmission format audio data.

[0213] Clause 5A. An apparatus as described in any combination of clauses 1A-4A, wherein the one or more processors are further configured to: establish a communication link through which the encoded scene-based audio data is received, and wherein the one or more processors are configured to obtain a status of the communication link as an operating condition.

[0214] Clause 6A. A device as described in any combination of clauses 1A-5A, wherein one or more processors are configured to: when operating conditions are below a threshold, perform psychoacoustic audio decoding on the encoded scene-based audio data to decode a subset of bits representing the ambient sound transmission format audio data, wherein the subset of bits is less than the total number of bits used to represent the ambient sound transmission format audio data.

[0215] Clause 7A. A device as described in clause 6A, wherein one or more processors are configured to: determine a number of coarse bits used during quantization of energy of ambient sound transmission format audio data; and perform psychoacoustic audio decoding on the encoded scene-based audio data to decode the number of coarse bits representing the ambient sound transmission format audio data.

[0216] Clause 8A. A device according to any combination of clauses 1A-5A, wherein one or more processors are configured to: when an operating condition is greater than or equal to a threshold, perform psychoacoustic audio decoding on the encoded scene-based audio data to decode a subset of bits representing ambient sound transmission format audio data and additional bits representing the ambient sound transmission format audio data, wherein the subset of bits is less than the total number of bits used to represent the ambient sound transmission format audio data.

[0217] Clause 9A. A device as described in any combination of clauses 1A-5A, wherein one or more processors are configured to: determine a number of coarse bits used during quantization of energy of ambient sound transmission format audio data; determine a number of fine bits used during quantization of energy of ambient sound transmission format audio data; and based on an operating condition being greater than or equal to a threshold, perform psychoacoustic audio decoding on the encoded scene-based audio data to decode the number of coarse bits and the number of fine bits representing the ambient sound transmission format audio data.

[0218] Clause 10A. A device as described in any combination of clauses 6A-9A, wherein one or more processors are configured to: perform psychoacoustic audio decoding on the encoded scene-based audio data to decode ambient sound audio data corresponding to all orders up to a defined order when reconstructing a subset of bits or a subset of bits and additional bits.

[0219] Clause 11A. A device according to any combination of clauses 1A-5A, wherein one or more processors are configured to: when an operating condition is below a threshold, perform psychoacoustic audio decoding on the encoded scene-based audio data to decode ambient sound audio data corresponding to spherical basis functions having a subset of orders that are less than all orders of the spherical basis functions to which the ambient sound audio data corresponds.

[0220] Clause 12A. A device according to any combination of clauses 1A-5A, wherein one or more processors are configured to: when an operating condition is above a threshold, perform psychoacoustic audio decoding on the encoded scene-based audio data to reconstruct ambient sound audio data corresponding to spherical basis functions having a subset of orders and at least one additional order, wherein the subset of orders is less than all orders of the spherical basis functions corresponding to the ambient sound audio data.

[0221] Clause 13A. A device according to any combination of clauses 11A and 12A, wherein one or more processors are configured to: perform psychoacoustic audio decoding on the encoded scene-based audio data to decode all bits of ambient sound audio data corresponding to spherical basis functions having a subset of orders or having both a subset of orders and at least one additional order.

[0222] Clause 14A. A device according to any combination of clauses 1A-13A, wherein the one or more processors are further configured to: render scene-based audio data to one or more speaker feeds; and output the speaker feeds to one or more speakers to reproduce a sound field represented by the scene-based audio data based on the speaker feeds.

[0223] Clause 15A. A device as described in any combination of clauses 1A-13A, wherein the one or more processors are further configured to: render scene-based audio data to one or more speaker feeds, and wherein the device includes one or more speakers configured to reproduce a sound field represented by the scene-based audio data based on the speaker feeds.

[0224] Clause 16A. The device of any combination of clauses 1A-15A, wherein the one or more processors are configured to obtain battery power as an operating condition.

[0225] Clause 17A. The device of any combination of clauses 1A-16A, wherein the foreground audio signal comprises a foreground ambient sound audio signal.

[0226] Clause 18A. A method for decoding encoded scene-based audio data, the method comprising: obtaining operating conditions of a device configured to decode encoded scene-based audio data; based on the operating conditions, performing psychoacoustic audio decoding on the encoded scene-based audio data to obtain ambient sound transmission format audio data; and performing spatial audio decoding on the ambient sound transmission format audio data to obtain scene-based audio data.

[0227] Clause 19A. The method of clause 18A, wherein performing psychoacoustic audio decoding comprises: performing psychoacoustic audio decoding according to an AptX compression algorithm on the encoded scene-based audio data based on operating conditions to obtain ambient sound transmission format audio data.

[0228] Clause 20A. The method of any combination of clauses 18A and 19A, wherein performing psychoacoustic audio decoding comprises: performing a single instance of psychoacoustic audio decoding on the encoded scene-based audio data to obtain ambient sound transmission format audio data.

[0229] Clause 21A. The method of any combination of clauses 18A-20A, further comprising obtaining a delay requirement for decoding encoded scene-based audio data, and wherein performing psychoacoustic audio decoding comprises: based on operating conditions and the delay requirement, performing psychoacoustic audio decoding on the encoded scene-based audio data to obtain ambient sound transmission format audio data.

[0230] Clause 22A. The method of any combination of clauses 18A-21A, further comprising establishing a communication link through which the encoded scene-based audio data is received, and wherein obtaining the operating condition comprises obtaining a state of the communication link as the operating condition.

[0231] Clause 23A. A method according to any combination of clauses 18A-22A, wherein performing psychoacoustic audio decoding includes: when an operating condition is below a threshold, performing psychoacoustic audio decoding on the encoded scene-based audio data to decode a subset of bits representing ambient sound transmission format audio data, wherein the subset of bits is less than the total number of bits used to represent the ambient sound transmission format audio data.

[0232] Clause 24A. A method according to clause 23A, wherein obtaining psychoacoustic audio decoding includes: determining a number of coarse bits used during quantization of energy of ambient sound transmission format audio data; and performing psychoacoustic audio decoding on the encoded scene-based audio data to decode the number of coarse bits representing the ambient sound transmission format audio data.

[0233] Clause 25A. A method according to any combination of clauses 18A-22A, wherein performing psychoacoustic audio decoding includes: when an operating condition is greater than or equal to a threshold, performing psychoacoustic audio decoding on the encoded scene-based audio data to decode a subset of bits representing ambient sound transmission format audio data and additional bits representing the ambient sound transmission format audio data, the subset of bits being less than the total number of bits used to represent the ambient sound transmission format audio data.

[0234] Clause 26A. A method according to any combination of clauses 18A-22A, wherein performing psychoacoustic audio decoding includes: determining a number of coarse bits used during quantization of energy of ambient sound transmission format audio data; determining a number of fine bits used during quantization of energy of ambient sound transmission format audio data; and performing psychoacoustic audio decoding on the encoded scene-based audio data to decode the number of coarse bits and the number of fine bits representing the ambient sound transmission format audio data based on an operating condition greater than or equal to a threshold.

[0235] Clause 27A. A method according to any combination of clauses 23A-26A, wherein performing psychoacoustic audio decoding includes: performing psychoacoustic audio decoding on the encoded scene-based audio data to decode ambient sound audio data corresponding to all orders up to a defined order when reconstructing a subset of bits or both a subset of bits and additional bits.

[0236] Clause 28A. A method according to any combination of clauses 18A-22A, wherein performing psychoacoustic audio decoding includes: when an operating condition is below a threshold, performing psychoacoustic audio decoding on the encoded scene-based audio data to decode ambient sound audio data corresponding to spherical basis functions having a subset of orders, wherein the subset of orders is less than all orders of the spherical basis functions corresponding to the ambient sound audio data.

[0237] Clause 29A. A method according to any combination of clauses 18A-22A, wherein performing psychoacoustic audio decoding includes: when an operating condition is above a threshold, performing psychoacoustic audio decoding on the encoded scene-based audio data to reconstruct ambient sound audio data corresponding to spherical basis functions having a subset of orders and at least one additional order, wherein the subset of orders is less than all orders of the spherical basis functions corresponding to the ambient sound audio data.

[0238] Clause 30A. A method according to any combination of clauses 28A and 29A, wherein performing psychoacoustic audio decoding includes: performing psychoacoustic audio decoding on the encoded scene-based audio data to decode all bits representing ambient sound audio data corresponding to spherical basis functions having a subset of orders or having a subset of orders and at least one additional order.

[0239] Clause 31A. The method of any combination of clauses 18A-30A, further comprising: rendering the scene-based audio data to one or more speaker feeds, and outputting the one or more speaker feeds to one or more speakers.

[0240] Clause 32A. The method of any combination of clauses 18A-32A, wherein obtaining the operating condition comprises obtaining battery power as the operating condition.

[0241] Clause 33A. The method of any combination of clauses 18A-33A, wherein the foreground audio signal comprises a foreground ambient sound audio signal.

[0242] Clause 34A. A device configured to decode encoded scene-based audio data, the device comprising: a component for obtaining an operating condition of the device configured to decode encoded scene-based audio data; a component for performing psychoacoustic audio decoding on the encoded scene-based audio data based on the operating condition to obtain ambient sound transmission format audio data; and a component for performing spatial audio decoding on the ambient sound transmission format audio data to obtain scene-based audio data.

[0243] Clause 35A. The apparatus of clause 34A, wherein the means for performing psychoacoustic audio decoding comprises means for performing psychoacoustic audio decoding according to an AptX compression algorithm on the encoded scene-based audio data based on operating conditions to obtain ambient sound transmission format audio data.

[0244] Clause 36A. An apparatus as described in any combination of clauses 34A and 35A, wherein the means for performing psychoacoustic audio decoding includes: means for performing a single instance of psychoacoustic audio decoding on the encoded scene-based audio data to obtain ambient sound transmission format audio data.

[0245] Clause 37A. An apparatus according to any combination of clauses 34A-36A, further comprising a component for obtaining a delay requirement for decoding encoded scene-based audio data, and wherein the component for performing psychoacoustic audio decoding includes: a component for performing psychoacoustic audio decoding on the encoded scene-based audio data based on operating conditions and the delay requirement to obtain ambient sound transmission format audio data.

[0246] Clause 38A. An apparatus according to any combination of clauses 34A-37A, further comprising means for establishing a communication link through which the encoded scene-based audio data is received, and wherein the means for obtaining the operating condition includes means for obtaining a state of the communication link as the operating condition.

[0247] Clause 39A. An apparatus as described in any combination of clauses 34A-38A, wherein the component for performing psychoacoustic audio decoding includes: a component for performing psychoacoustic audio decoding on the encoded scene-based audio data to decode a subset of bits representing the ambient sound transmission format audio data when the operating condition is below a threshold, wherein the subset of bits is less than the total number of bits used to represent the ambient sound transmission format audio data.

[0248] Clause 40A. An apparatus as described in clause 39A, wherein the component for obtaining psychoacoustic audio decoding includes: a component for determining a number of coarse bits used during quantization of energy of ambient sound transmission format audio data; and a component for performing psychoacoustic audio decoding on the encoded scene-based audio data to decode the number of coarse bits representing the ambient sound transmission format audio data.

[0249] Clause 41A. An apparatus as described in any combination of clauses 34A-38A, wherein the component for performing psychoacoustic audio decoding includes: a component for performing psychoacoustic audio decoding on the encoded scene-based audio data to decode a subset of bits representing ambient sound transmission format audio data and additional bits representing the ambient sound transmission format audio data when an operating condition is greater than or equal to a threshold, the subset of bits being less than the total number of bits used to represent the ambient sound transmission format audio data.

[0250] Clause 42A. An apparatus as described in any combination of clauses 34A-38A, wherein the components for performing psychoacoustic audio decoding include: components for determining a number of coarse bits used during quantization of energy of ambient sound transmission format audio data; components for determining a number of fine bits used during quantization of energy of ambient sound transmission format audio data; components for performing psychoacoustic audio decoding on encoded scene-based audio data to decode the number of coarse bits and the number of fine bits representing the ambient sound transmission format audio data based on an operating condition greater than or equal to a threshold.

[0251] Clause 43A. An apparatus as described in any combination of clauses 39A-42A, wherein the means for performing psychoacoustic audio decoding includes: means for performing psychoacoustic audio decoding on encoded scene-based audio data to decode ambient sound audio data corresponding to all orders up to a defined order when reconstructing a subset of bits or a subset of bits and additional bits.

[0252] Clause 44A. An apparatus as described in any combination of clauses 34A-38A, wherein the component for performing psychoacoustic audio decoding includes: a component for performing psychoacoustic audio decoding on the encoded scene-based audio data to decode ambient sound audio data corresponding to spherical basis functions having a subset of orders, wherein the subset of orders is less than all orders of the spherical basis functions corresponding to the ambient sound audio data when an operating condition is below a threshold.

[0253] Clause 45A. An apparatus as described in any combination of clauses 34A-38A, wherein the component for performing psychoacoustic audio decoding includes: a component for performing psychoacoustic audio decoding on the encoded scene-based audio data when the operating condition is above a threshold to decode ambient sound audio data corresponding to spherical basis functions having a subset of orders and at least one additional order, wherein the subset of orders is less than all orders of the spherical basis functions corresponding to the ambient sound audio data.

[0254] Clause 46A. An apparatus as described in any combination of clauses 44A and 45A, wherein the component for performing psychoacoustic audio decoding includes: a component for performing psychoacoustic audio decoding on the encoded scene-based audio data to decode all bits representing ambient sound audio data corresponding to spherical basis functions having a subset of orders or having a subset of orders and at least one additional order.

[0255] Clause 47A. The device of any combination of clauses 34A-46A, further comprising: means for rendering scene-based audio data to one or more speaker feeds, and means for outputting the one or more speaker feeds to one or more speakers.

[0256] Clause 48A. The apparatus of any combination of clauses 34A-48A, wherein the means for obtaining the operating condition comprises means for obtaining battery power as the operating condition.

[0257] Clause 49A. The device of any combination of clauses 34A-49A, wherein the foreground audio signal comprises a foreground ambient sound audio signal.

[0258] Item 50A. A non-temporary computer-readable storage medium having instructions stored thereon, which, when executed, cause one or more processors to: obtain operating conditions of a device configured to decode encoded scene-based audio data; based on the operating conditions, perform psychoacoustic audio decoding on the encoded scene-based audio data to obtain ambient sound transmission format audio data; and perform spatial audio decoding on the ambient sound transmission format audio data to obtain scene-based audio data.

[0259] In some contexts, such as broadcast contexts, the audio encoding device can be divided into a spatial audio encoder and a psychoacoustic audio encoder 26, where the spatial audio encoder performs some form of intermediate compression for the surround sound representation, including gain control, and the psychoacoustic audio encoder (which is also called the "perceptual audio encoder 26") performs perceptual audio compression to reduce data redundancy between transmission channels for gain normalization.

[0260] Furthermore, the above techniques may be performed for any number of different contexts and audio ecosystems, and should not be limited to any of the above contexts or audio ecosystems. Several exemplary contexts are described below, although the techniques should be limited to the exemplary contexts. An exemplary audio ecosystem may include audio content, a movie studio, a music studio, a game audio studio, channel-based audio content, an encoding engine, game audio stems, a game audio encoding / rendering engine, and a transport system.

[0261] Movie studios, music studios, and game audio studios can receive audio content. In some examples, audio content can represent the output of the collection. Movie studios can output channel-based audio content (e.g., 2.0, 5.1, and 7.1), such as by using a digital audio workstation (DAW). Music studios can output channel-based audio content (e.g., 2.0 and 5.1), such as by using a DAW. In either case, the encoding engine can receive and encode channel-based audio content based on one or more codecs (e.g., AAC, AC3, Dolby True HD, Dolby Digital Plus, and DTS Body Audio) for transmission system output. Game audio studios can output one or more game audio stems, such as by using a DAW. Game audio encoding / rendering engines can encode and / or render audio stems into channel-based audio content for transmission system output. Another exemplary context in which these techniques can be performed includes an audio ecosystem, which can include broadcast recording audio objects, professional audio systems, capture on consumer devices, surround sound audio formats, on-device rendering, consumer audio, televisions and accessories, and car audio systems.

[0262] Broadcast recorded audio objects, professional audio systems, and capture on consumer devices can all use surround sound audio formats to encode their output. In this way, audio content can be encoded into a single representation using surround sound audio formats that can be played using on-device rendering, consumer audio, televisions and accessories, and car audio systems. In other words, a single representation of audio content can be played in a universal audio playback system (i.e., as opposed to requiring a specific configuration such as 5.1, 7.1, etc.), such as audio playback system 16.

[0263] Other examples of contexts in which these techniques may be performed include an audio ecosystem that may include acquisition elements and playback elements. Acquisition elements may include wired and / or wireless acquisition devices (e.g., intrinsic (Eigen) microphones), on-device surround sound capture, and mobile devices (e.g., smartphones and tablets). In some examples, the wired and / or wireless acquisition devices may be coupled to the mobile device via (multiple) wired and / or wireless communication channels.

[0264] According to one or more techniques of the present disclosure, a mobile device can be used to obtain a sound field. For example, a mobile device can obtain a sound field via a wired and / or wireless acquisition device and / or surround sound capture on a device (e.g., multiple microphones integrated into a mobile device). The mobile device can then encode the obtained sound field into surround sound coefficients for one or more playback elements. For example, a user of a mobile device can record (obtain its sound field) a live event (e.g., a meeting, conference, play, concert, etc.) and encode the recording into surround sound coefficients.

[0265] The mobile device can also play the surround sound encoded sound field using one or more playback elements. For example, the mobile device can decode the surround sound encoded sound field and output a signal to one or more playback elements, which causes one or more of the playback elements to recreate the sound field. As an example, the mobile device can output the signal to one or more speakers (e.g., speaker arrays, sound bars, etc.) using wireless and / or wireless communication channels. As another example, the mobile device can output the signal to one or more docking stations and / or one or more docking speakers (e.g., smart cars and / or home audio systems) using a docking solution. As another example, the mobile device can output the signal to a set of headphones using headphone rendering, for example, to create realistic binaural sound.

[0266] In some examples, a particular mobile device can both acquire a 3D sound field and later play the same 3D sound field. In some examples, a mobile device can acquire a 3D sound field, encode the 3D sound field into an HOA, and send the encoded 3D sound field to one or more other devices (e.g., other mobile devices and / or other non-mobile devices) for playback.

[0267] Another context in which these techniques may be performed includes an audio ecosystem that may include audio content, a game studio, encoded audio content, a rendering engine, and a transmission system. In some examples, a game studio may include one or more DAWs that may support editing of surround sound signals. For example, one or more DAWs may include surround sound plug-ins and / or tools that may be configured to operate (e.g., work together) with one or more game audio systems. In some examples, a game studio may output a new stem format that supports HOA. In any case, the game studio may output the encoded audio content to a rendering engine that may render the sound field for playback by a transmission system.

[0268] These techniques may also be performed for an exemplary audio acquisition device. For example, these techniques may be performed with respect to an intrinsic microphone, which may include multiple microphones that are collectively configured to record a 3D sound field. In some examples, multiple microphones of the intrinsic microphone may be located on the surface of a substantially spherical ball with a radius of approximately 4 cm. In some examples, the audio encoding device 20 may be integrated into the intrinsic microphone so as to output a bitstream 21 directly from the microphone.

[0269] Another exemplary audio acquisition context may include a production truck that may be configured to receive signals from one or more microphones, such as one or more intrinsic microphones. The production truck may also include an audio encoder, such as Figure 1 A spatial audio encoding device 24.

[0270] In some cases, the mobile device may also include multiple microphones that are collectively configured to record a 3D sound field. In other words, the multiple microphones may have X, Y, Z diversity. In some examples, the mobile device may include a microphone that can be rotated to provide X, Y, Z diversity for one or more other microphones of the mobile device. The mobile device may also include an audio encoder, such as Figure 1 Audio encoder 22.

[0271] The enhanced video capture device can be further configured to record a 3D sound field. In some examples, the enhanced video capture device can be attached to a helmet of a user engaging in an activity. For example, the enhanced video capture device can be attached to a helmet of a user whitewater rafting. In this way, the enhanced video capture device can capture a 3D sound field representing the action around the user (e.g., water crashing behind the user, another raftsman talking in front of the user, etc.).

[0272] These techniques can also be performed for an accessory-enhanced mobile device, which can be configured to record a 3D sound field. In some examples, the mobile device can be similar to the mobile device discussed above, but with one or more accessories added. For example, an intrinsic microphone can be attached to the above-mentioned mobile device to form an accessory-enhanced mobile device. In this way, the accessory-enhanced mobile device can capture a higher quality version of the 3D sound field than using only the sound capture component integrated with the accessory-enhanced mobile device.

[0273] An exemplary audio playback device that can perform various aspects of the techniques described in this disclosure is discussed further below. According to one or more techniques of this disclosure, the speakers and / or soundbar can be arranged in any arbitrary configuration while still playing a 3D sound field. In addition, in some examples, the headphone playback device can be coupled to the decoder 32 (which refers to the decoder 32) via a wired or wireless connection. Figure 1 According to one or more techniques of the present disclosure, a single generic representation of a sound field can be used to render the sound field on any combination of speaker, soundbar, and headphone playback devices.

[0274] Many different exemplary audio playback environments may also be suitable for performing various aspects of the techniques described in this disclosure. For example, a 5.1 speaker playback environment, a 2.0 (e.g., stereo) speaker playback environment, a 9.1 speaker playback environment with full-height front speakers, a 22.2 speaker playback environment, a 16.0 speaker playback environment, a car speaker playback environment, and a mobile device with an earbud playback environment may be suitable environments for performing various aspects of the techniques described in this disclosure.

[0275] According to one or more techniques of the present disclosure, a single universal representation of a sound field can be used to render the sound field on any of the aforementioned playback environments. In addition, the techniques of the present disclosure enable the renderer to render the sound field from the universal representation for playback in playback environments other than the aforementioned environments. For example, if design considerations do not permit proper placement of the speakers according to a 7.1 speaker playback environment (e.g., if placement of a right surround speaker is not possible), the techniques of the present disclosure enable the renderer to compensate with the other 6 speakers, thereby enabling playback in a 6.1 speaker playback environment.

[0276] In addition, the user may be wearing headphones to watch a sports game. According to one or more techniques of the present disclosure, a 3D sound field of a sports game can be obtained (for example, one or more intrinsic microphones can be placed in and / or around a baseball field), surround sound coefficients corresponding to the 3D sound field can be obtained and sent to a decoder, the decoder can reconstruct the 3D sound field based on the surround sound coefficients and output the reconstructed 3D sound field to a renderer, the renderer can obtain an indication of the type of playback environment (for example, headphones) and render the reconstructed 3D sound field into a signal that causes the headphones to output a 3D sound field representation of the sports game.

[0277] In each of the various examples described above, it should be understood that the audio encoding device 22 may perform a method or otherwise include components for performing each step of the method that the audio encoding device 22 is configured to perform. In some instances, the components may include one or more processors. In some instances, one or more processors may represent a dedicated processor configured by instructions stored in a non-transitory computer-readable storage medium. In other words, various aspects of the technology in each set of encoding examples may provide a non-transitory computer-readable storage medium having instructions stored thereon, which, when executed, cause one or more processors to perform the method that the audio encoding device 20 has been configured to perform.

[0278] In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored as one or more instructions or codes on or sent through a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media, which corresponds to tangible media such as data storage media. Data storage media may be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, codes, and / or data structures for implementing the techniques described in the present disclosure. A computer program product may include a computer-readable medium.

[0279] By way of example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory, or any other medium that can be used to store the desired program code in the form of instructions or data structures and can be accessed by a computer. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals or other temporary media, but rather involve non-temporary, tangible storage media. As used herein, disks and optical disks include compact disks (CDs), laser optical disks, optical optical disks, digital versatile disks (DVDs), floppy disks, and Blu-ray disks, where disks typically reproduce data magnetically, while optical disks reproduce data optically using lasers. The above combinations should also be included within the scope of computer-readable media.

[0280] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Therefore, the term "processor" as used herein may refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. In addition, in some aspects, the functionality described herein may be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into a combined codec. In addition, these techniques may be fully implemented in one or more circuits or logic elements.

[0281] The techniques of the present disclosure may be implemented in a variety of devices or apparatuses, including wireless handsets, integrated circuits (ICs), or a set of ICs (e.g., chipsets). Various components, modules, or units are described in this disclosure to emphasize functional aspects of devices configured to perform the disclosed techniques, but do not necessarily need to be implemented by different hardware units. Instead, as described above, the various units may be combined in a codec hardware unit or provided by a collection of interoperable hardware units including one or more processors as described above in combination with appropriate software and / or firmware.

[0282] Furthermore, as used herein, "A and / or B" means "A or B", or "A and B".

[0283] Various aspects of the technology have been described. These and other aspects of the technology are within the scope of the following claims.

Claims

1. A device configured to decode encoded scene-based audio data, the device comprising: a memory configured to store the encoded scene-based audio data; as well as One or more processors configured to: obtaining an operating condition of a device for decoding encoded scene-based audio data; Based on the operating condition, performing psychoacoustic audio decoding on the encoded scene-based audio data to obtain ambient sound transmission format audio data, wherein to perform the psychoacoustic audio decoding, the one or more processors are configured to: When the operating condition is below a threshold, performing psychoacoustic audio decoding on the encoded scene-based audio data to decode a subset of bits representing the ambient sound transmission format audio data, the subset of bits being less than a total number of bits used to represent the ambient sound transmission format audio data, and performing psychoacoustic audio decoding on the encoded scene-based audio data to decode a number of coarse bits used during quantization of energy of the ambient sound transmission format audio data, the subset of bits comprising the coarse bits; as well as Spatial audio decoding is performed on the ambient sound transmission format audio data to obtain scene-based audio data.

2. The device according to claim 1, wherein: The one or more processors perform a single instance of psychoacoustic audio decoding on the encoded scene-based audio data to obtain the ambient sound transmission format audio data.

3. The device according to claim 1, wherein: The one or more processors are configured to, based on the operating condition, perform psychoacoustic audio decoding according to a compression algorithm on the encoded scene-based audio data to obtain the ambient sound transmission format audio data.

4. The device according to claim 1, in, The one or more processors are further configured to obtain a delay requirement for decoding the encoded scene-based audio data; and The one or more processors are configured to perform psychoacoustic audio decoding on the encoded scene-based audio data based on the operating conditions and the delay requirements to obtain the ambient sound transmission format audio data.

5. The device according to claim 1, in, The one or more processors are further configured to establish a communication link over which the encoded scene-based audio data is received, and The one or more processors are configured to obtain a status of the communication link as the operating condition.

6. The device according to claim 1, wherein: The one or more processors are configured to, when the operating condition is greater than or equal to a threshold, perform psychoacoustic audio decoding on the encoded scene-based audio data to decode a subset of the bits representing the ambient sound transmission format audio data that is less than a total number of bits used to represent the ambient sound transmission format audio data and additional bits representing the ambient sound transmission format audio data.

7. The device according to claim 1, wherein: The one or more processors are configured to: determining a coarse number of bits to use during quantization of energy of the ambient sound transmission format audio data; determining a number of fine bits to use during quantization of energy of the ambient sound transmission format audio data; Based on the operating condition being greater than or equal to a threshold, psychoacoustic audio decoding is performed on the encoded scene-based audio data to decode the number of coarse bits and the number of fine bits representing the ambient sound transmission format audio data.

8. The device according to claim 1, wherein: The one or more processors are configured to: Psychoacoustic audio decoding is performed on the encoded scene-based audio data to decode ambient sound audio data corresponding to all orders up to a defined order while reconstructing the subset of bits or both the subset of bits and additional bits.

9. The device according to claim 1, wherein: The one or more processors are configured to, when the operating condition is below a threshold, perform psychoacoustic audio decoding on the encoded scene-based audio data to decode ambient sound audio data corresponding to spherical basis functions having a subset of orders that are less than all orders of the spherical basis functions to which the ambient sound audio data corresponds.

10. The device according to claim 1, wherein: The one or more processors are configured to, when the operating condition is above a threshold, perform psychoacoustic audio decoding on the encoded scene-based audio data to reconstruct ambient sound audio data corresponding to spherical basis functions having a subset of orders and at least one additional order, wherein the subset of orders is less than all orders of the spherical basis functions to which the ambient sound audio data corresponds.

11. The device according to claim 1, wherein: The one or more processors are further configured to: rendering the scene-based audio data to one or more speaker feeds; as well as The speaker feeds are output to one or more speakers to reproduce a sound field represented by the scene based audio data based on the speaker feeds.

12. The device according to claim 1, in, The one or more processors are further configured to render the scene-based audio data to one or more speaker feeds, and Therein, the device comprises one or more speakers configured to reproduce a sound field represented by the scene-based audio data based on the speaker feeds.

13. The apparatus according to claim 1, wherein: The one or more processors are configured to obtain battery power as the operating condition.

14. The apparatus according to claim 1, wherein: The foreground audio signal includes a foreground ambient sound audio signal.

15. A method of decoding encoded scene-based audio data, the method comprising: obtaining an operating condition of a device configured to decode the encoded scene-based audio data; Based on the operating condition, performing psychoacoustic audio decoding on the encoded scene-based audio data to obtain ambient sound transmission format audio data, wherein performing the psychoacoustic audio decoding comprises: When the operating condition is below a threshold, performing psychoacoustic audio decoding on the encoded scene-based audio data to decode a subset of bits representing the ambient sound transmission format audio data, the subset of bits being less than a total number of bits used to represent the ambient sound transmission format audio data, and performing psychoacoustic audio decoding on the encoded scene-based audio data to decode a number of coarse bits used during quantization of energy of the ambient sound transmission format audio data, the subset of bits comprising the coarse bits; and Spatial audio decoding is performed on the ambient sound transmission format audio data to obtain scene-based audio data.

16. The method according to claim 15, wherein: Performing psychoacoustic audio decoding includes performing a single instance of psychoacoustic audio decoding on the encoded scene-based audio data to obtain the ambient sound transmission format audio data.

17. The method according to claim 15, wherein: Performing psychoacoustic audio decoding includes: performing psychoacoustic audio decoding according to a compression algorithm on the encoded scene-based audio data based on the operating condition to obtain the ambient sound transmission format audio data.

18. The method of claim 15, further comprising obtaining a delay requirement for decoding the encoded scene-based audio data, and in, Performing psychoacoustic audio decoding includes performing psychoacoustic audio decoding on the encoded scene-based audio data based on the operating condition and the delay requirement to obtain the ambient sound transmission format audio data.

19. The method of claim 15, further comprising establishing a communication link through which the encoded scene-based audio data is received, and in, Obtaining the operating condition includes obtaining a status of the communication link as the operating condition.

20. The method according to claim 15, wherein: Performing psychoacoustic audio decoding includes: when the operating condition is greater than or equal to a threshold, performing psychoacoustic audio decoding on the encoded scene-based audio data to decode a subset of the bits representing the ambient sound transmission format audio data that is less than a total number of bits used to represent the ambient sound transmission format audio data and additional bits representing the ambient sound transmission format audio data.

21. The method according to claim 15, wherein: Performing psychoacoustic audio decoding involves: determining a coarse number of bits to use during quantization of energy of the ambient sound transmission format audio data; determining a number of fine bits to use during quantization of energy of the ambient sound transmission format audio data; Based on the operating condition being greater than or equal to a threshold, psychoacoustic audio decoding is performed on the encoded scene-based audio data to decode the number of coarse bits and the number of fine bits representing the ambient sound transmission format audio data.

22. The method according to claim 15, wherein: Performing psychoacoustic audio decoding includes performing psychoacoustic audio decoding on the encoded scene-based audio data to decode ambient sound audio data corresponding to all orders up to a defined order when reconstructing a subset of the bits or reconstructing both the subset of bits and additional bits.

23. The method according to claim 15, wherein: Performing psychoacoustic audio decoding includes: when the operating condition is below a threshold, performing psychoacoustic audio decoding on the encoded scene-based audio data to decode ambient sound audio data corresponding to spherical basis functions having a subset of orders, wherein the subset of orders is less than all orders of the spherical basis functions corresponding to the ambient sound audio data.

24. A device configured to decode encoded scene-based audio data, the device comprising: means for obtaining an operating condition of a device configured to decode the encoded scene-based audio data; A means for performing psychoacoustic audio decoding on the encoded scene-based audio data based on the operating condition to obtain ambient sound transmission format audio data, wherein the means for performing the psychoacoustic audio decoding comprises: means for performing psychoacoustic audio decoding on the encoded scene-based audio data to decode a subset of bits representing the ambient sound transmission format audio data when the operating condition is below a threshold, the subset of bits being less than a total number of bits used to represent the ambient sound transmission format audio data, and means for performing psychoacoustic audio decoding on the encoded scene-based audio data to decode a number of coarse bits used during quantization of energy of the ambient sound transmission format audio data, the subset of bits comprising the coarse bits; and means for performing spatial audio decoding on the ambient sound transmission format audio data to obtain scene-based audio data.

25. A non-transitory computer-readable storage medium having stored thereon instructions that, when executed, cause one or more processors to: obtaining an operating condition of a device configured to decode encoded scene-based audio data; Based on the operating condition, psychoacoustic audio decoding is performed on the encoded scene-based audio data to obtain ambient sound transmission format audio data, wherein, The psychoacoustic audio decoding includes: When the operating condition is below a threshold, performing psychoacoustic audio decoding on the encoded scene-based audio data to decode a subset of bits representing the ambient sound transmission format audio data, the subset of bits being less than a total number of bits used to represent the ambient sound transmission format audio data, and performing psychoacoustic audio decoding on the encoded scene-based audio data to decode a number of coarse bits used during quantization of energy of the ambient sound transmission format audio data, the subset of bits comprising the coarse bits; as well as Spatial audio decoding is performed on the ambient sound transmission format audio data to obtain scene-based audio data.

Images