Device and method for encoding and decoding scene-based audio data
Patent Information
- Application Number
- CN202080044574.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2020-06-22
- Filing Date
- 2020-06-23
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2040-06-23
Smart Images

Figure CN113994425B_ABST
Abstract
Description
[0001] This application claims priority to U.S. Patent Application No. 16 / 907,934, filed on June 22, 2020, entitled “QUANTIZING SPATIAL COMPONENTS BASED ON BIT ALLOCATIONS DETERMINED FOR PSYCHOACOUSTIC AUDIO CODING,” which claims the benefit of U.S. Provisional Application No. 62 / 865,853, filed on June 24, 2019, entitled “QUANTIZING SPATIAL COMPONENTS BASED ON BIT ALLOCATIONS DETERMINED FOR PSYCHOACOUSTIC AUDIO CODING,” the entire contents of which are incorporated herein by reference as if set forth in full. Technical Field
[0002] The present disclosure relates to audio data, and more particularly, to the coding of audio data. Background Art
[0003] Psychoacoustic audio codecs refer to the process of compressing audio data using psychoacoustic models. Psychoacoustic audio codecs can exploit limitations in the human auditory system to compress audio data, taking into account limitations that arise due to spatial masking (e.g., two audio sources in the same location, where one auditory source masks the other auditory source in loudness), temporal masking (e.g., where one audio source masks the other auditory source in loudness), etc. Psychoacoustic models can attempt to model the human auditory system to identify masked sound fields or other parts of the sound field that are redundant, masked, or otherwise imperceptible to the human auditory system. Psychoacoustic audio codecs can also perform lossless compression by entropy encoding the audio data. Summary of the invention
[0004] In general, techniques are described for quantizing spatial components based on a bit allocation determined for a psychoacoustic audio codec.
[0005] In one example, various aspects of the technology relate to a device configured to encode scene-based audio data, the device comprising: a memory configured to store the scene-based audio data; and one or more processors configured to: perform spatial audio encoding with respect to the scene-based audio data to obtain a foreground audio signal and a corresponding spatial component, the spatial component defining a spatial characteristic of the foreground audio signal; perform psychoacoustic audio encoding on the foreground audio signal to obtain an encoded foreground audio signal; when performing psychoacoustic audio encoding on the foreground audio signal, determine a first bit allocation of the foreground audio signal; determine a second bit allocation of the spatial component based on the first bit allocation of the foreground audio signal; quantize the spatial component based on the second bit allocation of the spatial component to obtain a quantized spatial component; and specify the encoded foreground audio signal and the quantized spatial component in a bitstream.
[0006] In another example, various aspects of the technology relate to a method for encoding scene-based audio data, the method comprising: performing spatial audio encoding on the scene-based audio data to obtain a foreground audio signal and a corresponding spatial component, the spatial component defining a spatial characteristic of the foreground audio signal; performing psychoacoustic audio encoding on the foreground audio signal to obtain an encoded foreground audio signal; when performing psychoacoustic audio encoding on the foreground audio signal, determining a first bit allocation of the foreground audio signal; determining a second bit allocation of the spatial component based on the first bit allocation of the foreground audio signal; quantizing the spatial component based on the second bit allocation of the spatial component to obtain a quantized spatial component; and specifying the encoded foreground audio signal and the quantized spatial component in a bitstream.
[0007] In another example, various aspects of the technology relate to a device configured to encode scene-based audio data, the device comprising: a component for performing spatial audio encoding on the scene-based audio data to obtain a foreground audio signal and a corresponding spatial component, the spatial component defining a spatial characteristic of the foreground audio signal; a component for performing psychoacoustic audio encoding on the foreground audio signal to obtain an encoded foreground audio signal; a component for determining a first bit allocation of the foreground audio signal when performing psychoacoustic audio encoding on the foreground audio signal; a component for determining a second bit allocation of the spatial component based on the first bit allocation of the foreground audio signal; a component for quantizing the spatial component based on the second bit allocation of the spatial component to obtain a quantized spatial component; and a component for specifying the foreground audio signal and the quantized spatial component in a bitstream.
[0008] In another example, various aspects of the technology are directed to a non-transitory computer-readable storage medium having instructions stored thereon, which instructions, when executed, cause one or more processors to: perform spatial audio encoding on scene-based audio data to obtain a foreground audio signal and a corresponding spatial component, which defines a spatial characteristic of the foreground audio signal; perform psychoacoustic audio encoding on the foreground audio signal to obtain an encoded foreground audio signal; when performing psychoacoustic audio encoding on the foreground audio signal, determine a first bit allocation for the foreground audio signal; determine a second bit allocation for the spatial component based on the first bit allocation for the foreground audio signal; quantize the spatial component based on the second bit allocation for the spatial component to obtain a quantized spatial component; and specify the foreground audio signal and the quantized spatial component in a bitstream.
[0009] In another example, various aspects of the technology relate to a device configured to decode a bitstream representing encoded scene-based audio data, the device comprising: a memory configured to store the bitstream, the bitstream comprising an encoded foreground audio signal and a corresponding quantized spatial component, the spatial component defining a spatial characteristic of the foreground audio signal; and one or more processors, the one or more processors configured to: perform psychoacoustic audio decoding on the encoded foreground audio signal to obtain the foreground audio signal; when performing psychoacoustic audio decoding on the encoded foreground audio signal, determine a first bit allocation of the encoded foreground audio signal; determine a second bit allocation of the quantized spatial component based on the first bit allocation of the encoded foreground audio signal; dequantize the quantized spatial component based on the second bit allocation of the quantized spatial component to obtain the spatial component; and reconstruct the scene-based audio data based on the foreground audio signal and the spatial component.
[0010] In another example, various aspects of the technology relate to a method for decoding a bitstream representing scene-based audio data, the method comprising: obtaining an encoded foreground audio signal and a corresponding quantized spatial component from the bitstream, the quantized spatial component defining a spatial characteristic of the encoded foreground audio signal; performing psychoacoustic audio decoding on the encoded foreground audio signal to obtain the foreground audio signal; when performing psychoacoustic audio decoding on the encoded foreground audio signal, determining a first bit allocation of the encoded foreground audio signal; determining a second bit allocation of the quantized spatial component based on the first bit allocation of the encoded foreground audio signal; dequantizing the quantized spatial component based on the second bit allocation of the quantized spatial component to obtain the spatial component; and reconstructing the scene-based audio data based on the foreground audio signal and the spatial component.
[0011] In another example, various aspects of the technology relate to a device configured to decode a bitstream representing encoded scene-based audio data, the device comprising: a component for obtaining an encoded foreground audio signal and a corresponding quantized spatial component from the bitstream, the quantized spatial component defining a spatial characteristic of the encoded foreground audio signal; a component for performing psychoacoustic audio decoding on the encoded foreground audio signal to obtain the foreground audio signal; a component for determining a first bit allocation of the encoded foreground audio signal when performing psychoacoustic audio decoding on the encoded foreground audio signal; a component for determining a second bit allocation of the quantized spatial component based on the first bit allocation of the encoded foreground audio signal; a component for dequantizing the quantized spatial component based on the second bit allocation of the quantized spatial component to obtain the spatial component; and a component for reconstructing the scene-based audio data based on the foreground audio signal and the spatial component.
[0012] In another example, various aspects of the technology relate to a non-transitory computer-readable storage medium having instructions stored thereon, which instructions, when executed, cause one or more processors to: obtain an encoded foreground audio signal and a corresponding quantized spatial component from a bit stream representing scene-based audio data, the quantized spatial component defining a spatial feature of the encoded foreground audio signal; perform psychoacoustic audio decoding on the encoded foreground audio signal to obtain the foreground audio signal; when performing psychoacoustic audio decoding on the encoded foreground audio signal, determine a first bit allocation of the encoded foreground audio signal; determine a second bit allocation of the quantized spatial component based on the first bit allocation of the encoded foreground audio signal; dequantize the quantized spatial component based on the second bit allocation of the quantized spatial component to obtain the spatial component; and reconstruct the scene-based audio data based on the foreground audio signal and the spatial component.
[0013] The details of one or more aspects of these techniques are set forth in the accompanying drawings and the following description. Other features, objects, and advantages of these techniques will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] Figure 1 is a diagram illustrating a system that can perform various aspects of the techniques described in this disclosure.
[0015] Figure 2 is a diagram illustrating another example of a system that can perform various aspects of the techniques described in this disclosure.
[0016] Figure 3A and Figure 3B is shown in more detail Figure 1 and Figure 2A block diagram of an example of a psychoacoustic audio encoder is shown in the example of FIG.
[0017] Figure 4A and Figure 4B is shown in more detail Figure 1 and Figure 2 A block diagram of an example of a psychoacoustic audio decoder.
[0018] Figure 5 It is shown Figure 2 00106 00107 00108 00109 ...
[0019] Figure 6 It is shown Figure 2 00140] A block diagram of exemplary components of a sink device as shown in the example of FIG.
[0020] Figure 7 It is shown Figure 1 Flowchart of exemplary operation of an audio encoder shown in FIG. 1 in performing various aspects of the techniques described in this disclosure.
[0021] Figure 8 It is shown Figure 1 Flowchart of exemplary operation of an audio decoder shown in FIG. 1 in performing various aspects of the techniques described in this disclosure. DETAILED DESCRIPTION
[0022] There are different types of audio formats, including channel-based, object-based, and scene-based formats. Scene-based formats can use surround sound technology. Surround sound technology allows the sound field to be represented using a collection of layered elements that can be rendered to speaker feeds for most speaker configurations.
[0023] An example of a hierarchical element set is the set of spherical harmonic coefficients (SHCs). The following expression demonstrates the description or representation of a sound field using SHCs:
[0024]
[0025] This expression shows that at any point in the sound field at time t The pressure at p i Can be provided by SHC Here, c is the speed of sound (about 343 m / s), is the reference point (or observation point), j n (·) is a spherical Bessel function of order n, and is the spherical harmonic basis function of order n and suborder m (also called spherical basis function). It can be seen that the term in the square brackets is the frequency domain representation of the signal (i.e., ), which can be approximated by various time-frequency transforms, such as discrete Fourier transform (DFT), discrete cosine transform or wavelet transform. Other examples of hierarchical sets include wavelet transform coefficient sets and other coefficient sets of multiresolution basis functions.
[0026] SHC The SHCs (also referred to as surround sound coefficients) represent scene-based audio, where the SHCs can be input to an audio encoder to obtain encoded SHCs that can facilitate more efficient transmission or storage. For example, a method involving (1+4) can be used. 2 A fourth-order representation of 25 (so fourth-order) coefficients.
[0027] As described above, SHC can be derived from microphone recordings using a microphone array. Various examples of how SHC can be derived from a microphone array are described in Poletti, M., "Three-Dimensional Surround Sound Systems Based on Spherical Harmonics," J. Audio Eng. Soc., Vol. 53, No. 11, November 2005, pp. 1004-1025.
[0028] To illustrate how SHC can be derived from an object-based description, consider the following equation. The coefficients of the sound field corresponding to an individual audio object are It can be expressed as:
[0029]
[0030] Where i is is a spherical Hankel function (of the second kind) of order n, and is the position of the object. Knowing the source energy of the object g(ω) as a function of frequency (e.g., using time-frequency analysis techniques such as performing a Fast Fourier Transform on the PCM stream) allows us to convert each PCM object and the corresponding position into an SHC Furthermore, it can be shown (since the above is a linear and orthogonal decomposition) that for each object The coefficients are additive. In this way, multiple PCM objects (where a PCM object is an example of an audio object) can be represented by The coefficients are represented (e.g., as the sum of the coefficient vectors of the individual objects). Essentially, the coefficients contain information about the sound field (pressure as a function of the 3D coordinates), and the above represents the distance from the individual objects to the observation point The following figures are described in the context of SHC-based audio coding.
[0031] Figure 1 1 is a diagram illustrating a system 10 that can perform various aspects of the techniques described in this disclosure. Figure 1 , system 10 includes a content creator system 12 and a content consumer 14. Although described in the context of content creator system 12 and content consumer 14, the techniques may be implemented in any context in which SHC (which may also be referred to as surround sound coefficients) or any other layered representation of a sound field is encoded to form a bitstream representing audio data.
[0032] In addition, the content creator system 12 may represent a system that includes one or more of any form of computing device capable of implementing the techniques described in this disclosure, including a handheld device (or cellular phone, including a so-called "smartphone", or in other words, a mobile phone or handheld device), a tablet computer, a laptop computer, a desktop computer, an extended reality (XR) device (which may refer to any one or more of a virtual reality (VR) device, an augmented reality (AR) device, a mixed reality (MR) device, etc.), a gaming system, an optical disc player, a receiver (such as an audio / video (A / V) receiver), or dedicated hardware, to provide some examples.
[0033] Likewise, content consumer 14 may represent any form of computing device capable of implementing the techniques described in this disclosure, including a handset (or cellular telephone, including so-called “smartphones,” or in other words, a mobile collection or phone), an XR device, a tablet computer, a television (including so-called “smart TVs”), a set-top box, a laptop computer, a gaming system or console, a watch (including so-called smart watches), wireless headphones (including so-called “smart headphones”), or a desktop computer, to provide some examples.
[0034] Content creator system 12 may represent any entity that can generate audio content and possibly video content for consumption by content consumers, such as content consumer 14. Content creator system 12 may capture live audio data at an event, such as a sporting event, while also inserting various other types of additional audio data into the live audio content, such as commentary audio data, commercial audio data, intro or exit audio data, etc.
[0035] Content consumer 14 represents an individual who owns or has access to an audio playback system 16, which may refer to any form of audio playback system capable of rendering higher-order surround sound audio data (which includes higher-order audio coefficients, which may also be referred to as spherical harmonic coefficients) to a speaker feed for playback as audio content. Figure 1 In the example of FIG. 1 , content consumer 14 includes audio playback system 16 .
[0036] Surround sound audio data may be defined in the spherical harmonic domain and rendered or otherwise transformed from the spherical harmonic domain to the spatial domain, resulting in audio content in the form of one or more speaker feeds. Surround sound audio data may represent an example of "scene-based audio data", which describes an audio scene using surround sound coefficients. Scene-based audio data differs from object-based audio data in that the entire scene is described (in the spherical harmonic domain) rather than discrete objects (in the spatial domain) as is common in object-based audio data. Scene-based audio data differs from channel-based audio data in that scene-based audio data exists in the spherical harmonic domain, which is different from the spatial domain of channel-based audio data.
[0037] In any case, the content creator system 12 includes microphones 18 that record or otherwise obtain live audio in various formats (including directly as surround sound coefficients and audio objects). When the microphone array 18 (which may also be referred to as "microphones 18") obtains live audio directly as surround sound coefficients, the microphones 18 may include a transcoder, such as Figure 1 The surround sound transcoder 20 is shown in the example of FIG.
[0038] In other words, while shown as being separate from the microphones 5, the surround sound transcoder 20 may be included as a separate instance within each microphone 5 in order to transcode the captured feed into the surround sound coefficients 21. However, when not included within the microphone 18, the surround sound transcoder 20 may transcode the real-time feed output from the microphone 18 into the surround sound coefficients 21. In this regard, the surround sound transcoder 20 may represent a unit configured to transcode the microphone feeds and / or audio objects into the surround sound coefficients 21. Thus, the content creator system 12 includes the surround sound transcoder 20 integrated with the microphone 18, as a transcoder separate from the microphone 18, or some combination thereof.
[0039] The content creator system 12 may also include an audio encoder 22 configured to compress the surround sound coefficients 21 to obtain a bitstream 31. The audio encoder 22 may include a spatial audio encoding device 24 and a psychoacoustic audio encoding device 26. The spatial audio encoding device 24 may represent a device capable of performing compression on the surround sound coefficients 21 to obtain intermediate formatted audio data 25 (which may also be referred to as "mezzanine formatted audio data 25" when the content creator system 12 represents a broadcast network as described in more detail below). The intermediate formatted audio data 25 may represent audio data that has been compressed using spatial audio compression but has not yet undergone psychoacoustic audio encoding (e.g., such as AptX or Advanced Audio Coding (AAC), or other similar types of psychoacoustic audio encoding, including various enhanced AAC (eAAC), such as High Efficiency AAC (HE-AAC) HE-AACv2, which is also referred to as eAAC+, etc.).
[0040] The spatial audio encoding device 24 may be configured to compress the surround sound coefficients 21. That is, the spatial audio encoding device 24 may use a decomposition involving application of a linear reversible transform (LIT) to compress the surround sound coefficients 21. An example of a linear reversible transform is called "singular value decomposition" ("SVD"), principal component analysis ("PCA"), or eigenvalue decomposition, which may represent different examples of linear reversible decomposition.
[0041] In this example, the spatial audio encoding device 24 may apply SVD to the surround sound coefficients 21 to determine a decomposed version of the surround sound coefficients 21. The decomposed version of the surround sound coefficients 21 may include one or more predominant audio signals and one or more corresponding spatial components describing spatial characteristics (e.g., direction, shape, and width) of the associated predominant audio signals. In this way, the spatial audio encoding device 24 may apply decomposition to the surround sound coefficients 21 to decouple the energy (as represented by the predominant audio signals) from the spatial characteristics (as represented by the spatial components).
[0042] The spatial audio encoding device 24 may analyze the decomposed version of the ambisonic sound coefficients 21 to identify various parameters, which may facilitate reordering of the decomposed version of the ambisonic sound coefficients 21. The spatial audio encoding device 24 may reorder the decomposed version of the ambisonic sound coefficients 21 based on the identified parameters, wherein such reordering may improve coding efficiency, assuming that the transform may reorder the ambisonic sound coefficients across a frame of ambisonic sound coefficients (where a frame typically includes M samples of the decomposed version of the ambisonic sound coefficients 21, and in some examples M is set to 1024).
[0043] After reordering the decomposed versions of the surround sound coefficients 21, the spatial audio encoding device 24 may select one or more decomposed versions of the surround sound coefficients 21 as foreground (or in other words, obvious, main or significant) components of the sound field. The spatial audio encoding device 24 may specify a decomposed version of the surround sound coefficients 21 representing a foreground component (which may also be referred to as a "primary sound signal", "primary audio signal" or "primary sound component") and associated directional information (which may also be referred to as a "spatial component" or, in some cases, a so-called "V vector" that identifies the spatial characteristics of the corresponding audio object). The spatial component may represent a vector having multiple different elements (which may be referred to as a "coefficient" in terms of a vector) and may therefore be referred to as a "multidimensional vector".
[0044] The spatial audio encoding device 24 may then perform a sound field analysis on the surround sound coefficients 21 to at least partially identify the surround sound coefficients 21 representing one or more background (or in other words, ambient) components of the sound field. Background components may also be referred to as "background audio signals" or "ambient audio signals". The spatial audio encoding device 24 may perform energy compensation on the background audio signals, assuming that in some examples, the background audio signals may only include a subset of any given sample of the surround sound coefficients 21 (e.g., such as those corresponding to zero-order and first-order spherical basis functions, rather than those corresponding to second-order or higher-order spherical basis functions). When performing order reduction, in other words, the spatial audio encoding device 24 may enhance the remaining background surround sound coefficients of the surround sound coefficients 21 (e.g., add energy to / subtract energy from them) to compensate for the change in total energy caused by performing order reduction.
[0045] The spatial audio encoding device 24 may then perform some form of interpolation on the foreground direction information (which is another way of referring to the spatial component), and then perform order reduction on the interpolated foreground direction information to generate reduced order foreground direction information. In some examples, the spatial audio encoding device 24 may further perform quantization on the reduced order foreground direction information, thereby outputting the encoded foreground direction information. In some cases, such quantization may include scalar / entropy quantization, which may take the form of vector quantization. The spatial audio encoding device 24 may then output the intermediate formatted audio data 25 as a background audio signal, a foreground audio signal, and quantized foreground direction information.
[0046] In any case, in some examples, the background audio signal and the foreground audio signal may include a transmission channel. That is, the spatial audio encoding device 24 may output a transmission channel for each frame of the surround sound coefficient 21 including a corresponding one of the background audio signals (e.g., M samples of one of the surround sound coefficients 21 corresponding to the zero or first order spherical basis function) and each frame of the foreground audio signal (e.g., M samples of the audio object decomposed from the surround sound coefficient 21). The spatial audio encoding device 24 may also output side information (which may also be referred to as "sideband information") including a quantized spatial component corresponding to each foreground audio signal.
[0047] In general, the transmission channel and side information can be Figure 1 In the example of , the ATF audio data 25 is represented as surround sound transmission format (ATF) audio data 25 (which is another way to refer to intermediate formatted audio data). In other words, the AFT audio data 25 may include transmission channels and side information (which may also be referred to as "metadata"). As an example, the ATF audio data 25 may conform to the HOA (high-order surround sound) transmission format (HTF). More information about HTF can be found in the European Telecommunications Standards Institute (ETSI) technical specification (TS) entitled "High-Order Surround Sound (HOA) Transmission Format" (ETSI TS103589 V1.1.1, dated June 2018 (2018-06)). In this way, the ATF audio data 25 can be referred to as HTF audio data 25.
[0048] The spatial audio encoding device 24 may then send or otherwise output the ATF audio data 25 to a psychoacoustic audio encoding device 26. The psychoacoustic audio encoding device 26 may perform psychoacoustic audio encoding on the ATF audio data 25 to generate a bitstream 31. The psychoacoustic audio encoding device 26 may operate according to a standardized, open source, or proprietary audio encoding process. For example, the psychoacoustic audio encoding device 26 may operate according to AptX TM , various other versions of AptX (e.g., Enhanced AptX (E-AptX), AptX live, AptX Stereo, and AptX High Definition (AptX-HD)), or Advanced Audio Coding (AAC) and its derivatives to perform psychoacoustic audio encoding. The content creator system 12 can then transmit the bitstream 31 to the content consumer 14 via a transmission channel.
[0049] In some examples, the psychoacoustic audio encoding device 26 may represent one or more instances of a psychoacoustic audio codec, each instance being used to encode a transmission channel of the ATF audio data 25. In some cases, the psychoacoustic audio encoding device 26 may represent one or more instances of an AptX encoding unit (as described above). In some cases, the psychoacoustic audio codec unit 26 may call an instance of the AptX encoding unit for each transmission channel of the ATF audio data 25.
[0050] In some examples, in order to generate different representations of the sound field using surround sound coefficients (which are also an example of audio data 21), the audio encoder 22 may use an encoding scheme for the surround sound representation of the sound field, known as mixed-order surround (MOA), as discussed in more detail in U.S. patent application serial number 15 / 672,058, entitled “MIXED-ORDER AMBISONICS (MOA) AUDIO DATA FOR COMPUTER-MEDIATED REALITY SYSTEMS,” filed on August 8, 2017, and published on January 3, 2019 as U.S. Patent Publication No. 2019 / 0007781.
[0051] To generate a particular MOA representation of a sound field, the audio encoder 22 may generate a partial subset of a complete set of surround sound coefficients. For example, each MOA representation generated by the audio encoder 22 may provide accuracy with respect to some areas of the sound field, but provide lower accuracy in other areas. In one example, the MOA representation of the sound field may include eight (8) uncompressed surround sound coefficients, while a third-order surround sound representation of the same sound field may include sixteen (16) uncompressed surround sound coefficients. In this way, each MOA representation of the sound field generated as a partial subset of the surround sound coefficients may be less storage-intensive and less bandwidth-intensive (if and when transmitted as part of the bitstream 31 over the transmission channel shown).
[0052] Although described with respect to MOA representation, the techniques of the present disclosure may also be performed for a full-order surround (FOA) representation, where all surround coefficients of a given order N are used to represent the sound field. In other words, the sound field representation generator 302 may use all surround coefficients of a given order N to represent the sound field, rather than using a partial, non-zero subset of the surround coefficients to represent the sound field, resulting in a total of (N+1) 2 The surround sound coefficient.
[0053] In this regard, higher-order surround sound audio data (which is another way of referring to surround sound coefficients in an MOA representation or a FOA representation) may include higher-order surround sound coefficients associated with spherical basis functions having an order of 1 or less (which may be referred to as "first-order surround sound audio data"), higher-order surround sound coefficients associated with spherical basis functions having mixed orders and sub-orders (which may be referred to as the "MOA representation" discussed above), or higher-order surround sound coefficients associated with spherical basis functions having an order greater than 1 (referred to as the "FOA representation" above).
[0054] In addition, although Figure 1 31 is shown as being sent directly to the content consumer 14, but the content creator system 12 may output the bitstream 31 to an intermediate device located between the content creator system 12 and the content consumer 14. The intermediate device may store the bitstream 31 for later transmission to the content consumer 14 requesting the bitstream. The intermediate device may include a file server, a network server, a desktop computer, a laptop computer, a tablet computer, a mobile phone, a smart phone, or any other device capable of storing the bitstream 31 for later retrieval by an audio decoder. The intermediate device may exist in a content delivery network capable of streaming the bitstream 31 (and possibly in conjunction with transmitting a corresponding video data bitstream) to a subscriber (such as the content consumer 14) requesting the bitstream 31.
[0055] Alternatively, the content creator system 12 may store the bitstream 31 to a storage medium such as an optical disc, a digital video disc, a high-definition video disc, or other storage medium, most of which are capable of being read by a computer and may therefore be referred to as a computer-readable storage medium or a non-transitory computer-readable storage medium. In this context, a transmission channel may refer to those channels through which content stored to these media is sent (and may include retail stores and other store-based delivery mechanisms). In any case, the technology of the present disclosure should not be limited in this regard. Figure 1 .
[0056] like Figure 1 As further shown in the example of , the content consumer 14 includes an audio playback system 16. The audio playback system 16 can represent any audio playback system capable of playing multi-channel audio data. The audio playback system 16 can also include an audio decoding device 32. The audio decoding device 32 can represent a device configured to decode the surround sound coefficients 11' from the bitstream 31, wherein the surround sound coefficients 11' can be similar to the surround sound coefficients 11 but are different due to lossy operations (e.g., quantization) and / or transmission via a transmission channel.
[0057] The audio decoding device 32 may include a psychoacoustic audio decoding device 34 and a spatial audio decoding device 36. The psychoacoustic audio decoding device 34 may represent a unit configured to operate inversely to the psychoacoustic audio encoding device 26 to reconstruct the ATF audio data 25' from the bitstream 31. Likewise, the apostrophe with respect to the ATF audio data 25 output from the psychoacoustic audio decoding device 34 indicates that the ATF audio data 25' may differ slightly from the ATF audio data 25 due to lossy or other operations performed during compression of the ATF audio data 25. The psychoacoustic audio decoding device 34 may be configured to perform decompression according to a standardized, open source, or proprietary audio encoding process (such as AptX, a variant of AptX, AAC, a variant of AAC, etc., mentioned above).
[0058] Although described below primarily with respect to AptX, these techniques can also be applied to other psychoacoustic audio codecs. Examples of other psychoacoustic audio codecs include Audio Codec 3 (AC-3), Apple Lossless Audio Codec (ALAC), MPEG-4 Audio Lossless Streaming (ALS), Enhanced AC-3, Free Lossless Audio Codec (FLAC), Monkey's Audio, MPEG-1 Audio Layer II (MP2), MPEG-1 Audio Layer III (MP3), Opus, and Windows Media Audio (WMA).
[0059] In any case, the psychoacoustic audio decoding device 34 can perform psychoacoustic decoding on the foreground audio objects specified in the bitstream 31 and the encoded surround sound coefficients representing the background audio signals specified in the bitstream 31. In this way, the psychoacoustic audio decoding device 34 can obtain the ATF audio data 25' and output the ATF audio data 25' to the spatial audio decoding device 36.
[0060] The spatial audio decoding device 36 may represent a unit configured to operate inversely to the spatial audio encoding device 24. That is, the spatial audio decoding device 36 may dequantize the foreground direction information specified in the bitstream 31. The spatial audio decoding device 36 may also dequantize the quantized foreground direction information to obtain the decoded foreground direction information. The spatial audio decoding device 36 may then perform interpolation on the decoded foreground direction information and then determine the surround sound coefficients representing the foreground component based on the decoded foreground audio signal and the interpolated foreground direction information. The spatial audio decoding device 36 may then determine the surround sound coefficients 11' based on the determined surround sound coefficients representing the foreground audio signal and the decoded surround sound coefficients representing the background audio signal.
[0061] The audio playback system 16 may render the surround sound coefficients 11' to output speaker feeds 39 after decoding the bitstream 31 to obtain the surround sound coefficients 11'. The audio playback system 16 may include a plurality of different audio renderers 38. The audio renderers 38 may each provide different forms of rendering, wherein the different forms of rendering may include one or more of various ways of performing vector-based amplitude panning (VBAP), one or more of various ways of performing binaural rendering (e.g., head-related transfer function (HRTF), binaural room impulse response (BRIR), etc.), and / or one or more of various ways of performing sound field synthesis.
[0062] The audio playback system 16 may output a speaker feed 39 to one or more speakers 40. The speaker feed 39 may drive the speakers 40. The speakers 40 may represent loudspeakers (e.g., transducers placed in a cabinet or other enclosure), headphone speakers, or any other type of transducer capable of producing sound based on electrical signals.
[0063] In order to select an appropriate renderer or, in some cases, generate an appropriate renderer, the audio playback system 16 may obtain loudspeaker information 41 indicating the number of speakers 40 and / or the spatial geometry of the speakers 40. In some cases, the audio playback system 16 may dynamically determine the speaker information 41 by using a reference microphone and driving the speakers 40 to obtain the loudspeaker information 41. In other cases, or in conjunction with the dynamic determination of the speaker information 41, the audio playback system 16 may prompt the user to interact with the audio playback system 16 and input the speaker information 41.
[0064] The audio playback system 16 may select one of the audio renderers 38 based on the speaker information 41. In some cases, when none of the audio renderers 38 is within a certain threshold similarity metric (in terms of loudspeaker geometry) specified in the speaker information 41, the audio playback system 16 may generate one of the audio renderers 38 based on the speaker information 41. In some cases, the audio playback system 16 may generate one of the audio renderers 38 based on the speaker information 41 without first attempting to select an existing one of the audio renderers 38.
[0065] Although described with respect to speaker feed 39, the audio playback system 16 may render a headphone feed from the speaker feed 39 or directly from the surround sound coefficients 11', thereby outputting the headphone feed to the headphone speakers. The headphone feed may represent a binaural audio speaker feed, which the audio playback system 16 renders using a binaural audio renderer.
[0066] As described above, the audio encoder 22 may call the spatial audio encoding device 24 to perform spatial audio encoding (or otherwise compress) the surround sound audio data 21, thereby obtaining the ATF audio data 25. During the application of spatial audio encoding to the surround sound audio data 21, the spatial audio encoding device 24 may obtain a foreground audio signal and a corresponding spatial component, which are respectively specified as transmission channels and accompanying metadata (or side information) in an encoded form.
[0067] As described above, the spatial audio encoding device 24 may apply vector quantization with respect to the spatial components and prior to specifying the spatial components as metadata in the AFT audio data 25. The psychoacoustic audio encoding device 26 may quantize each transmission channel of the ATF audio data 25 independently of the quantization of the spatial components performed by the spatial audio encoding device 24. Because the spatial components provide spatial characteristics of the corresponding foreground audio signals, independent quantization may result in different errors between the spatial components and the foreground audio signals, which may result in audio artifacts during playback, such as incorrect positioning of the above-mentioned audio signals within the reconstructed sound field, poor spatial resolution of the higher quality foreground audio signals, and other anomalies that may cause interference or apparent inaccuracies during reproduction of the sound field.
[0068] According to various aspects of the techniques described in this disclosure, the spatial audio encoding device 24 and the psychoacoustic audio encoding device 26 are integrated in that the psychoacoustic audio encoding device 26 may incorporate a spatial component quantizer (SCQ) 46, thereby offloading quantization from the spatial audio encoding device 24. The SCQ 46 may perform quantization for spatial components based on the bit allocation specified for the transmission channel, thereby more tightly integrating the spatial audio encoding device 24 and the psychoacoustic audio encoding device 26. Aligning or otherwise integrating the two devices 24 and 26 may allow for more uniform quantization, which may reduce or otherwise eliminate the above-mentioned audio artifacts, thereby improving the performance of the audio encoder 22 itself.
[0069] In operation, the spatial audio encoding device 24 may perform spatial audio encoding on the scene-based audio data 21 to obtain a foreground audio signal and a corresponding spatial component. However, the spatial audio encoding performed by the spatial audio encoding device 24 omits the above-mentioned quantization of the spatial component, because the quantization is again offloaded to the psychoacoustic audio encoding device 26. The spatial audio encoding device 24 may output the ATF audio data 25 to the psychoacoustic audio encoding device 26.
[0070] The audio encoder 22 invokes the psychoacoustic audio encoding device 26 to perform psychoacoustic audio encoding on the foreground audio signal to obtain an encoded foreground audio signal. In some examples, the psychoacoustic audio encoding device 26 may perform psychoacoustic audio encoding according to an AptX compression algorithm, including any of the various versions of AptX listed above. The AptX compression algorithm is generally related to Figures 5 to 8 .
[0071] The psychoacoustic audio encoding device 26 can determine a first bit allocation for the foreground audio signal when performing psychoacoustic audio encoding for the foreground audio signal. The psychoacoustic audio encoding device 26 can determine a second bit allocation for the spatial component based on the first bit allocation for the foreground audio signal. Because the second bit allocation is determined based on the first bit allocation, the psychoacoustic audio encoding device 26 more fully integrates quantization across the foreground audio signal and the spatial component. The psychoacoustic audio encoding device 26 can call the SCQ 46 and pass the second bit allocation to the SCQ 46. The SCQ 46 can apply quantization (such as vector quantization) to achieve a bit allocation for the spatial component that meets or exceeds the second bit allocation. The psychoacoustic audio encoding device 26 can then specify the encoded foreground audio signal and the quantized spatial component in the bitstream 31.
[0072] As described above, the audio decoder 32 can operate inversely to the audio encoder 22. In this manner, the audio decoder 32 can obtain the bitstream 31 and invoke the psychoacoustic audio decoding device 34 to perform psychoacoustic audio decoding on the encoded foreground audio signal to obtain the foreground audio signal. As described above, the psychoacoustic audio decoding device 34 can perform psychoacoustic audio decoding according to the AptX decompression algorithm. Again, more information about the AptX decompression algorithm is provided below with respect to Figures 5 to 8 .
[0073] In any case, when psychoacoustic audio encoding is performed on the foreground audio signal, the psychoacoustic audio decoding device 34 can determine a first bit allocation of the encoded foreground audio signal. The psychoacoustic audio decoding device 34 can also determine a second bit allocation of the quantized spatial components based on the first bit allocation of the encoded foreground audio signal. The psychoacoustic audio decoding device 34 can call a spatial component dequantizer (SCD) 54, which can dequantize the quantized spatial components based on the second bit allocation of the spatial components to obtain the spatial components. The psychoacoustic audio decoding device 34 can reconstruct the ATF audio data 25' based on the foreground audio signal and the spatial components. The spatial audio decoding device 36 can then reconstruct the scene-based audio data 21' based on the foreground audio signal and the spatial components.
[0074] Figure 2 is a diagram illustrating another example of a system that can perform various aspects of the techniques described in this disclosure. Figure 2 The system 110 may represent Figure 1 An example of system 10 is shown in the example of FIG. Figure 2 As shown in the example of , system 110 includes a source device 112 and a sink device 114 , where source device 112 may represent an example of content creator system 12 and sink device 114 may represent an example of content consumer 14 and / or audio playback system 16 .
[0075] Although described with respect to source device 112 and sink device 114, in some cases, source device 112 may operate as a sink device, and in these and other cases, sink device 114 may operate as a source device. Figure 2 The example of system 110 shown in FIG. 1 is merely one example to illustrate various aspects of the techniques described in this disclosure.
[0076] In any case, as described above, source device 112 may represent any form of computing device capable of implementing the techniques described in this disclosure, including handheld devices (or cellular phones, including so-called "smart phones"), tablet computers, so-called smart phones, remote-controlled aircraft (such as so-called "drones"), robots, desktop computers, receivers (such as audio / video (AV) receivers), set-top boxes, televisions (including so-called "smart TVs"), media players (such as digital video disc players, streaming media players, Blu-ray disc players, etc.), and other devices. TM Player, etc.) or any other device capable of wirelessly communicating audio data to a sink device via a personal area network (PAN). For purposes of illustration, it is assumed that source device 112 represents a smart phone.
[0077] The sink device 114 may represent any form of computing device capable of implementing the techniques described in this disclosure, including a handset (or in other words, a cellular phone, a mobile phone, a mobile handset, etc.), a tablet computer, a smart phone, a desktop computer, a wireless headset (which may include wireless headsets with or without a microphone, as well as so-called smart wireless headsets that include additional functions such as health monitoring, onboard music storage and / or playback, dedicated cellular functions, etc.), a wireless speaker (including so-called "smart speakers"), a watch (including so-called "smart watches"), or any other device capable of reproducing a sound field based on audio data wirelessly transmitted via a PAN. In addition, for the purpose of illustration, it is assumed that the sink device 114 represents a wireless headset.
[0078] like Figure 2As shown in the example of , source device 112 includes one or more applications ("apps") 118A-118N ("app 118"), a mixing unit 120, an audio encoder 122 (which includes a spatial audio encoding device (SAED) 124 and a psychoacoustic audio encoding device (PAED) 126), and a wireless connection manager 128. Although not shown in Figure 2 As shown in the example of , source device 112 may include multiple other elements that support the operation of app 118, including an operating system, various hardware and / or software interfaces (such as user interfaces, including graphical user interfaces), one or more processors, memory, storage devices, etc.
[0079] Each of the apps 118 represents software (such as a collection of instructions stored to a non-transitory computer-readable medium) that configures the system 110 to provide certain functionality when executed by one or more processors of the source device 112. To name a few examples, the apps 118 may provide messaging functionality (such as access to email, text messaging, and / or video messaging), voice calling functionality, video conferencing functionality, calendaring functionality, audio streaming functionality, directions functionality, mapping functionality, gaming functionality. The apps 118 may be first-party applications designed and developed by the same company that designs and sells the operating system executed by the source device 112 (and are typically pre-installed on the source device 112) or third-party applications that are accessible via a so-called “app store” or may be pre-installed on the source device 112. Each app 118, when executed, may output audio data 119A-119N (“audio data 119”), respectively.
[0080] In some examples, audio data 119 may be received from a microphone (not shown, but similar to a microphone) connected to source device 112. Figure 1 The audio data 119 may include the above-mentioned Figure 1 The surround sound coefficients of the surround sound audio data 21 discussed in the example of FIG. 1 are similar, wherein such surround sound audio data may be referred to as “scene-based audio data”. Thus, the audio data 119 may also be referred to as “scene-based audio data 119” or “surround sound audio data 119”.
[0081] Although described with respect to surround sound audio data, these techniques may be performed with respect to surround sound audio data that does not necessarily include coefficients corresponding to so-called "higher-order" spherical basis functions (e.g., spherical basis functions having an order greater than 1). Thus, these techniques may be performed with respect to surround sound audio data that includes coefficients corresponding only to zero-order spherical basis functions, or only to zero-order and first-order spherical basis functions.
[0082] The mixing unit 120 represents a unit configured to mix one or more audio data 119 output by the app 118 (as well as other audio data output by the operating system - such as alarms or other tones, including keyboard key sounds, ringtones, etc.) to generate mixed audio data 121. Audio mixing can refer to the process of combining multiple sounds (as set forth in the audio data 119) into one or more channels. During mixing, the mixing unit 120 can also manipulate and / or enhance the volume level (which can also be referred to as "gain level"), frequency content, and / or panoramic position of the surround sound audio data 119. In the context of streaming the surround sound audio data 119 over a wireless PAN session, the mixing unit 120 can output the mixed audio data 121 to the audio encoder 122.
[0083] The audio encoder 122 can be used with the above Figure 1 That is, the audio encoder 122 may represent a unit configured to encode the mixed audio data 121 to obtain encoded audio data in the form of a bitstream 131. In some examples, the audio encoder 122 may encode individual audio data in the audio data 119.
[0084] For illustration purposes, referring to an example of a PAN protocol, A number of different types of audio codecs (a word that is a combination of the words "encode" and "decode") are provided, and can be extended to include vendor-specific audio codecs. The Advanced Audio Distribution Profile (A2DP) of MPEG-1 indicates that support for A2DP requires support for the sub-band codecs specified in A2DP. A2DP also supports codecs described in MPEG-1 Part 3 (MP2), MPEG-2 Part 3 (MP3), MPEG-2 Part 7 (Advanced Audio Codec - AAC), MPEG-4 Part 3 (High Efficiency - AAC (HE-AAC)), and Adaptive Transform Acoustic Codec (ATRAC). In addition, as mentioned above, A2DP supports vendor-specific codecs such as aptX TM and various other versions of aptX (e.g., enhanced aptX (E-aptX), aptX live, and aptX high definition (aptX-HD)).
[0085] The audio encoder 122 may operate in accordance with one or more of the audio codecs listed above and one or more of the audio codecs not listed above, but operates to encode the mixed audio data 121 to obtain encoded audio data 131 (which is another way to refer to the bitstream 131). The audio encoder 122 may first call the SAED 124, which may communicate with Figure 1 The SAED 124 may be similar (if not substantially similar) to the SAED 24 shown in the example of FIG. 1 . The SAED 124 may perform the above-described spatial audio compression on the mixed audio data to obtain the ATF audio data 125 (which is similar to the Figure 1 The SAED 124 may output the ATF audio data 25 to the PAED 126.
[0086] PAED 126 can be used with Figure 1 The PAED 126 may be similar (if not substantially similar) to the PAED 26 shown in the example of FIG. The PAED 126 may perform psychoacoustic audio encoding according to any of the aforementioned codecs (including AptX and its variants) to obtain a bitstream 131. The audio encoder 122 may output the encoded audio data 131 to one of the wireless communication units 130 (e.g., the wireless communication unit 130A) managed by the wireless connection manager 128.
[0087] The wireless connection manager 128 may represent a unit configured to allocate bandwidth within certain frequencies of the available spectrum to different wireless communication units in the wireless communication unit 130. For example, The communication protocol operates in the 2.5 GHz spectrum range, which overlaps with the spectrum range used by various WLAN communication protocols. The wireless connection manager 128 can allocate certain portions of the bandwidth to the wireless devices during a given time. The wireless connection manager 128 may be configured to control the bandwidth allocation and other aspects of the wireless communication protocol 129 to achieve a specified quality of service (QoS). In other words, the wireless connection manager 128 may provide an API to adjust the scheme 129 by which the operation of the wireless communication unit 130 is controlled to achieve a specified QoS.
[0088] In other words, the wireless connection manager 128 can manage the coexistence of multiple wireless communication units 130 operating within the same spectrum, such as some WLAN communication protocols and some PAN protocols as described above. The wireless connection manager 128 may include a coexistence scheme 129 (in Figure 2129 ), which indicates when (eg, intervals) each of the wireless communication units 130 can send packets and how many packets to send, the size of the packets sent, and so on.
[0089] The wireless communication units 130 may each represent a wireless communication unit 130 operating according to one or more communication protocols to communicate the bitstream 131 to the sink device 114 via a transmission channel. Figure 2 In the example of FIG. 1 , for the purpose of illustration, it is assumed that the wireless communication unit 130A is based on Assume further that the wireless communication unit 130A operates according to A2DP to establish a PAN link (via a transport channel) to allow the bitstream 131 to be transmitted from the source device 112 to the sink device 114 .
[0090] related For more information about the suite of communication protocols, see the document entitled “Bluetooth Core Specification v 5.0” dated December 6, 2016, which is available at: www.bluetooth.org / en-us / specification / adopted-specifications. For more information about A2DP, see the document entitled “Advanced Audio Distribution Profile Specification” (version 1.3.1) dated July 14, 2015.
[0091] The wireless communication unit 130A may output the bit stream 131 to the sink device 114 via a transmission channel, which is assumed to be a wireless channel in the example of Bluetooth. Figure 2 131 is shown as being sent directly to sink device 114, but source device 112 may output bitstream 131 to an intermediate device located between source device 112 and sink device 114. The intermediate device may store bitstream 131 for later transmission to sink device 114, which may request bitstream 131. The intermediate device may include a file server, a network server, a desktop computer, a laptop computer, a tablet computer, a mobile phone, a smart phone, or any other device capable of storing bitstream 131 for later retrieval by an audio decoder. The intermediate device may be present in a content delivery network capable of streaming bitstream 131 (and possibly in conjunction with transmitting a corresponding video data bitstream) to a subscriber (such as sink device 114) requesting bitstream 131.
[0092] Alternatively, source device 112 may store bitstream 131 to a storage medium such as an optical disc, digital video disc, high-definition video disc, or other storage medium, most of which are capable of being read by a computer and may therefore be referred to as computer-readable storage medium or non-transitory computer-readable storage medium. In this context, transmission channels may refer to those channels through which content stored to these media is sent (and may include retail stores and other store-based delivery mechanisms). In any case, the technology of the present disclosure should not be limited in this regard. Figure 2 .
[0093] like Figure 2 As further shown in the example of FIG. 1 , the sink device 114 includes a wireless connection manager 150 that manages one or more of wireless communication units 152A-152N (“wireless communication units 152”) according to a scheme 151, an audio decoder 132 (including a psychoacoustic audio decoding device (PADD) 134 and a spatial audio decoding device (SADD) 136), and one or more speakers 140A-140N (“speakers 140”, which may be similar to Figure 1 40 as shown in the example of FIG. 1 ). The wireless connection manager 150 may operate in a manner similar to that described above with respect to the wireless connection manager 128, exposing an API to coordinate a scheme 151 by which the operation of the wireless communication unit 152 achieves a specified QoS.
[0094] Wireless communication unit 152 may be similar in operation to wireless communication unit 130, except that wireless communication unit 152 operates inversely to wireless communication unit 130 to receive bit stream 131 via a transmission channel. Assume that one of wireless communication units 152 (e.g., wireless communication unit 152A) receives bit stream 131 according to The wireless communication unit 152A may output the bit stream 131 to the audio decoder 132.
[0095] The audio decoder 132 may operate in a reverse manner to the audio encoder 122. The audio decoder 132 may operate in accordance with one or more of the audio codecs listed above and one or more of the audio codecs not listed above, but operates to decode the encoded audio data 131 to obtain mixed audio data 121'. Again, the primes with respect to the "mixed audio data 121" indicate that there may be some losses due to quantization operations or other lossy operations that occurred during encoding by the audio encoder 122.
[0096] The audio decoder 132 may call the PADD 134 to perform psychoacoustic audio decoding on the bitstream 131 to obtain the ATF audio data 125', which the PADD 134 may output to the SADD 136. The SADD 136 may perform spatial audio decoding to obtain the mixed audio data 121'. Although for the purpose of illustration, Figure 2 The example does not show a renderer (similar to Figure 1 38), but the audio decoder 132 may render the mixed audio data 121' to the speaker feed (using any renderer, such as described above with respect to Figure 1 ) and outputs the speaker feed to one or more speakers 140.
[0097] Each speaker 140 represents a transducer configured to reproduce a sound field fed from the speaker. Figure 2 As shown in the example of , the transducer can be integrated within the sink device 114, or can be communicatively coupled to the sink device 114 (via wired or wireless). Speaker 140 can represent any form of speaker, such as a loudspeaker, an earphone speaker, or a speaker in an earbud. In addition, although described with respect to a transducer, speaker 140 can represent other forms of speakers, such as a "speaker" used in bone conduction headphones, which sends vibrations to the upper jaw, which induces sound in the human auditory system.
[0098] As described above, PAED 126 may perform various aspects of the quantization techniques described above with respect to PAED 26 to quantize spatial components based on the foreground audio signal-dependent bit allocations of the spatial components. PADD 134 may also perform various aspects of the quantization techniques described above with respect to PADD 34 to dequantize the quantized spatial components based on the foreground audio signal-dependent bit allocations of the spatial components. Figure 3A and Figure 3B The example provides more information about PAED 126, while the Figure 4A and Figure 4B The examples in the provide more information about PADD 134.
[0099] Figure 3A is shown in more detail Figure 1 and Figure 2 326A may represent an example of a PAED 26 and 126, where the PAED 326A may be configured to encode audio data for transmission via a PAN (e.g., ) or any other form of wireless communication (such as cellular wireless communication – so-called 3G, 4G and / or 5G wireless communication, WiFiTM However, the techniques of the present disclosure performed by the audio encoder 326A may be used in any context where compressed audio data is required. In some examples, the audio encoder 326A may be configured to transmit audio data in accordance with aptX TM The audio codec encodes the audio data 25, including, for example, enhanced aptX (E-aptX), aptXlive, and aptX high definition. However, the techniques of the present disclosure can be used with any audio codec. As will be explained in more detail below, the audio encoder 326A can be configured to perform various aspects of perceptual audio coding according to various aspects of the techniques described in the present disclosure.
[0100] exist Figure 3A In an example of , the audio encoder 326A can be configured to encode the audio data 25 using a gain-shape vector quantization encoding process. In the gain-shape vector quantization encoding process, the audio encoder 326A is configured to encode both the gain (e.g., energy level) and shape (e.g., residual vector defined by transform coefficients) of a subband of the frequency domain audio data, respectively. Each subband of the frequency domain audio data represents a certain frequency range of a particular frame of the audio data 25. Generally, throughout this disclosure, the term "subband" refers to a frequency range, a frequency band, etc.
[0101] The audio encoder 326A may first invoke a transform codec 328, which represents a unit configured to process the audio data 25. The transform unit 328 may be configured to process the audio data 25 by at least partially applying a transform to frames of the audio data 25 and thereby transforming the audio data 25 from the time domain to the frequency domain to produce frequency domain audio data 329.
[0102] A frame of audio data 25 may be represented by a predetermined number of samples of audio data. In one example, a frame of audio data 25 may be 1024 samples wide. Different frame widths may be selected based on the frequency transform being used and the amount of compression required. Frequency domain audio data 329 may be represented as transform coefficients, wherein the value of each transform coefficient represents the energy of the frequency domain audio data 329 at a particular frequency.
[0103] In one example, the transform unit 328 can be configured to transform the audio data 25 into frequency domain audio data 329 using a modified discrete cosine transform (MDCT). MDCT is an "overlapped" transform based on a type IV discrete cosine transform. MDCT is considered "overlapped" because it can process data from multiple frames. That is, in order to perform the transform using MDCT, the transform unit 328 can include a fifty percent overlapping window into subsequent frames of audio data. The overlapping characteristics of the MDCT may be useful for data compression techniques such as audio coding because it can reduce artifacts encoded at frame boundaries. The transform unit 328 is not necessarily limited to using MDCT but can use other frequency domain transform techniques to transform the audio data 25 into frequency domain audio data 329.
[0104] Although not in Figure 3A As shown in the example of , PAED 326A can then call a subband filter, which can represent a unit configured to separate the frequency domain audio data 329 into subbands. Each subband includes transform coefficients of the frequency domain audio data 329 within a specific frequency range. For example, the subband filter can separate the frequency domain audio data 329 into twenty different subbands. In some examples, the subband filter can be configured to separate the frequency domain audio data 329 into subbands with uniform frequency ranges. In other examples, the subband filter can be configured to separate the frequency domain audio data 329 into subbands with non-uniform frequency ranges.
[0105] For example, the subband filter can be configured to separate the frequency domain audio data 329 into subbands according to a Bark scale. Typically, the subbands of the Bark scale have frequency ranges that are perceptually equal. That is, the subbands of the Bark scale are not equal in terms of frequency range, but are equal in terms of human auditory perception. Generally speaking, subbands at lower frequencies will have fewer transform coefficients because lower frequencies are more easily perceived by the human auditory system.
[0106] As such, the frequency domain audio data 329 in the lower frequency sub-bands of the sub-bands may be less compressed by the PAED 326A than the higher frequency sub-bands. Likewise, the higher frequency sub-bands of the sub-bands may include more transform coefficients because higher frequencies are more difficult for the human auditory system to perceive. Thus, the frequency domain audio 329 in the data in the higher frequency sub-bands of the sub-bands may be more compressed by the PAED 326A than the lower frequency sub-bands.
[0107] The PAED 326A may be configured to process each subband using a subband processing unit. That is, the subband processing unit may be configured to process each subband individually. The subband processing unit may be configured to perform a gain-shape vector quantization process.
[0108] The gain-shape analysis unit 330 may receive subbands as input. For each subband, the gain-shape analysis unit 330 may determine an energy level 331A for each subband. That is, each subband has an associated energy level 331A. The energy level 331A is a scalar value in decibels (dB) that represents the total amount of energy (also referred to as gain) in the transform coefficients of a particular subband in the subband. The gain-shape analysis unit 330 may separate the energy level 331A in one of the subbands from the transform coefficients of the subband to produce a residual vector 331B. The residual vector 331B represents the so-called "shape" of the subband. The shape of the subband may also be referred to as the spectrum of the subband.
[0109] To perform the gain-shape analysis, the PAED 326A may invoke a spectrum perception analysis unit 324, which may represent a unit configured to analyze the frequency domain audio data 329 to identify the perceptual importance of each subband relative to other audio bands (possibly both in time and between subbands in the same time period). The spectrum perception analysis unit 324 may identify which subbands should be subjected to gain-shape analysis, thereby outputting an indication 325 to the gain-shape analysis unit 330 and the static spatial bit redistribution unit 352. The gain-shape analysis unit 330 may then perform the gain-shape analysis discussed above with respect to the subbands based on the indication 325.
[0110] The vector quantizer 332 may represent a unit configured to quantize the residual vector 331B. In one example, the vector quantizer 332 may quantize the residual vector using a quantization process to generate a residual ID 333. Instead of quantizing each sample individually (e.g., scalar quantization), the vector quantizer 332 may be configured to quantize a block of samples included in the residual vector 332 (e.g., a shape vector).
[0111] In some examples, the PAED 326A can dynamically allocate bits for encoding the energy levels 331A and the residual vectors 331B. That is, for each subband, the PAED 326A can invoke a dynamic bit allocation unit 334, which can be configured to determine the number of bits allocated for energy quantization (e.g., by the energy quantizer 336) and the number of bits allocated for vector quantization (e.g., by the vector quantizer 332).
[0112] The dynamic bit allocation unit 334 may allocate bits to each shape vector 331B based on a psychoacoustic model 335A ("PM 335A"), which may attempt to model the human auditory system to identify redundant, imperceptible, masked, and / or other aspects of the sound field that are not perceptible. The dynamic bit allocation unit 334 may then allocate bits based on the PM 335A to reduce or eliminate bits used for sound field components that are not perceptible to the human auditory system or that are redundant within the sound field. The total number of bits allocated for energy quantization may be referred to as energy-specified bits. The allocated bits may then be allocated between a coarse quantization process and a fine quantization process as well as a vector quantization process.
[0113] The dynamic bit allocation unit 334 may obtain the reconstruction energy from the adder 342, which may reconstruct (via addition) the energy 331A' based on the coarse energy 339 and the fine energy 341. The dynamic bit allocation unit 334 may apply the PM 335A to the reconstruction energy 331A' to obtain the bit allocation 337. The dynamic bit allocation unit 334 may output the bit allocation 337 to the shape vector quantizer 332, the dynamic spatial bit redistribution unit 350, and the energy quantizer 336 (although in Figure 3A ).
[0114] The energy quantizer 336 may receive the energy level 331A of the subband and quantize the energy level 336 of the subband into a coarse energy 339 and a fine energy 341. This disclosure will describe the quantization process for one subband, but it should be understood that the energy quantizer 336 may perform energy quantization for one or more subbands (including each subband).
[0115] like Figure 3A As shown in the example of FIG. 3 , the energy quantizer 336 may include a coarse gain quantization (“CGQ”) unit 338, a fine gain quantization (“FGQ”) unit 340, and an adder 342. Although not shown in FIG. Figure 3A As shown in the example of , the energy quantizer 336 may also include a prediction difference (P / D) unit that can predict or otherwise identify the difference between the energy levels 331A for one subband and another subband of the same frame of audio data (which can be referred to as spatial prediction in the frequency domain) or for a same (or possibly different) subband in subbands from different frames (which can be referred to as temporal prediction). The P / D unit can analyze the energy levels 331A in this manner to obtain a predicted energy level (“PEL”) for each subband. The P / D unit can output the predicted energy levels to the coarse quantization unit 338.
[0116] The CGQ unit 338 may represent a unit configured to perform coarse quantization on the predicted energy level to obtain a quantized coarse energy 339. The CGQ unit 338 may output the quantized coarse energy 339 to the bitstream encoder 344 and the adder 342. The CGQ unit 338 may also obtain the difference between the quantized coarse energy 339 and the predicted energy level. The CGQ unit 338 may output the difference to the FGQ unit 340 as an error 343 (which may also be referred to as a "residual 343").
[0117] The FGQ unit 340 may represent a unit configured to perform fine quantization on the error 343. The fine quantization may be considered "fine" relative to the coarse quantization performed by the coarse quantization unit 338. That is, the fine quantization unit 340 may further quantize the error 343 by quantizing according to a step size with a higher resolution than the step size used when performing the coarse quantization. The FGQ unit 340 may obtain a quantized fine energy 341 for each subband as a result of performing the fine quantization on the error 343. The fine quantization unit 340 may output the quantized fine energy 341 to the bitstream encoder 344.
[0118] In general, the energy quantizer 336 may perform a multi-step quantization process. The energy quantizer 336 may first quantize the energy level 338 using a first number of bits for a coarse quantization process to generate a coarse energy 339. The energy quantizer 336 may generate the coarse energy using a predetermined energy level range (e.g., a range defined by a maximum and minimum energy level) for quantization. The coarse energy 339 is close to the value of the energy level 331A.
[0119] Energy quantizer 336 may then determine the difference between coarse energy 339 and the predicted energy level. This difference is sometimes referred to as a quantization error (or residual). Energy quantizer 336 may then quantize the quantization error using a second number of bits in a fine quantization process to produce fine energy 341. The number of bits used for fine quantization bits is determined by subtracting the number of bits used for the coarse quantization process from the total number of bits allocated for the energy. When added together, coarse energy 339 and fine energy 341 represent the total quantized value of energy level 331A.
[0120] The PAED 326A may also be configured to encode the coarse energy 339, the fine energy 341, and the residual ID 333 using a bitstream encoder 344 to create the encoded audio data 31. The bitstream encoder 344 may be configured to further compress the coarse energy 339, the fine energy 341, and the residual ID 333 using one or more of the entropy encoding processes described above.
[0121] Energy quantizer 336 (and / or its components, such as FGQ unit 340) can implement a hierarchical rate control mechanism to provide a greater degree of scalability and enable seamless or substantially seamless real-time streaming. For example, according to aspects of the present disclosure, FGQ unit 340 can implement a hierarchical fine quantization scheme. In some examples, FGQ unit 340 invokes a multiplexer (or "MUX") to implement a selection operation for hierarchical rate control.
[0122] The term "coarse quantization" refers to the combined operation of the above two-step coarse-fine quantization process. The FGQ unit 340 may perform one or more additional iterations of fine quantization on the error 343 received from the CGQ unit 338. The FGQ unit 340 may use a multiplexer to switch and iterate between various (finer) energy levels.
[0123] Hierarchical rate control may refer to a tree-based fine quantization structure or a cascaded fine quantization structure. When viewed as a tree-based structure, the existing two-step quantization operation forms the root node of the tree, and the root node is described as having a resolution depth of one (1). Depending on the availability of bits for further fine quantization, the multiplexer may select (multiple) additional levels of fine granularity quantization. With respect to the tree-based structure representing the multi-level fine quantization technique of the present disclosure, any such subsequent fine quantization level selected by the multiplexer represents a resolution depth of two (2), three (3), etc.
[0124] The FGQ unit 340 may provide improved scalability and control with respect to seamless real-time streaming scenarios in a wireless PAN. For example, the FGQ unit 340 may replicate the hierarchical fine quantization scheme and quantization multiplexing tree at a higher layer level, seeding at the coarse quantization point of the more general decision tree. In addition, the FGQ unit 340 may enable the PAED 326 to achieve seamless or substantially seamless real-time compression and streaming navigation. For example, the FGQ unit 340 may implement a multi-root hierarchical decision structure with respect to multiple levels of fine quantization, thereby enabling the energy quantizer 336 to implement potentially several iterations of fine quantization using the total available bits.
[0125] The FGQ unit 340 can implement the hierarchical rate control process in a variety of ways. The FGQ unit 340 can call a multiplexer on a per-subband basis to independently multiplex the error 343 information belonging to each subband (and thereby select a corresponding tree-based quantization scheme). That is, in these examples, the FGQ unit 340 performs a multiplexing-based hierarchical quantization scheme selection for each corresponding subband, independent of the quantization scheme selection for any other subband in the subband. In these examples, the FGQ unit 340 quantizes each subband according to a target bit rate specified only for the corresponding subband. In these examples, the PAED 326A can signal the details of the specific hierarchical quantization scheme for each subband as part of the encoded audio data 31.
[0126] In other examples, the FGQ unit 340 may call the multiplexer only once and thereby select a single multiplexing-based quantization scheme for the error 343 information belonging to all subbands. That is, in these examples, the FGQ unit 340 quantizes the error 343 information belonging to all subbands according to the same target bit rate, which is selected once and uniformly defined for all subbands. In these examples, the PAED 326A may signal the details of the single hierarchical quantization scheme applied across all subbands as part of the encoded audio data 31.
[0127] like Figure 3A As further shown in the example of , the CGQ unit 338 can output a bit allocation 349 for the coarse energy 339 to the static spatial bit redistribution unit 352. The bit allocation 349 can define the number of bits allocated to the coarse gain 339. The FGQ unit 340 can output a bit allocation 351 for each of the one or more fine energies 341 to the dynamic spatial bit redistribution unit 350. The bit allocation 351 can define the number of bits allocated to each of the one or more fine gains 341.
[0128] The static spatial bit redistribution (“SBR”) unit 352 may represent a unit configured to redistribute bits 339 between different coarse energies of different subbands to achieve a target bit rate (which is typically defined prior to encoding or determined based on various operating conditions, such as communication link bandwidth, communication link signal-to-noise ratio (SNR), battery power, or other conditions) based on the indication of spectrum sensing importance 325 (from the spectrum sensing analysis unit 324) and the bit allocation 349 of the coarse energy 339. The static SBR unit 352 may output a bit redistribution 359 (which may also be referred to as a “static spatial bit distribution 359”) to the spatiotemporal bit distributor 354.
[0129] The dynamic SBR unit 350 may represent a unit configured to redistribute bits among different subbands based on the bit allocation 351 of the dynamic bit allocation 337 and the fine energy 341. The dynamic SBR unit 350 may operate according to a PM 335B similar or substantially similar to the PM 335A to obtain a bit redistribution 361 (which may also be referred to as a "dynamic spatial bit distribution 351"). The dynamic SBR unit 350 may output the bit redistribution 361 to the spatiotemporal bit distributor 354.
[0130] The spatiotemporal bit distributor 354 can then determine the spatiotemporal bit distribution 355 based on the bit distributions 361 and 359. The spatiotemporal bit distribution 355 is both spatial and temporal because the spatial component defines the spatial characteristics of the foreground audio signal over time. The spatiotemporal bit distributor 354 can determine the spatiotemporal bit distribution 355 to achieve the target bit rate. The spatiotemporal bit distributor 354 can output the spatiotemporal bit distribution 355 to the ATF bit allocation unit 356.
[0131] The ATF bit allocation unit 356 may represent a unit configured to determine a spatial component bit allocation 357 based on the spatiotemporal bit distribution 355. The ATF bit allocation unit 356 may determine the spatial component bit allocation 357 based on a target bit rate, allocate some bits to the foreground audio signal and the background audio signal, and allocate the remaining bits to the spatial components. The ATF bit allocation unit 356 may output the spatial component bit allocation 357 to the spatial component quantizer 46, which may operate as described above to quantize the spatial components 45 based on the spatial component bit allocation 357 to obtain quantized spatial components 47.
[0132] In this regard, the PAED 326A may invoke the gain-shape analysis unit 330 to perform a shape and gain analysis on the foreground audio signal (represented by the frequency domain audio data 329) to obtain a shape 331B and a gain 331A representing the foreground audio signal. The PAED 326A may then perform quantization on the gain 331A to obtain a coarse gain 339 and one or more fine residuals 341. The PAED 326A may invoke the dynamic SBR unit 350 and the static SBR unit 352 to determine bit allocations 361 and 359 (respectively), which the spatiotemporal bit distributor 354 may process to obtain a spatiotemporal bit distribution 355. Based on the spatiotemporal bit distribution 355, the ATF bit allocation unit 356 may determine a spatial component bit distribution 357.
[0133] Next reference Figure 3B For example, PAED 326B can represent Figure 1 and Figure 226 and / or 126 shown in the example of FIG. 326B. Thus, PAED 326B may be similar to PAED 326A, except that PAED 326B does not include gain-shape analysis unit 330, shape vector quantizer 332, and hierarchical coarse-fine gain quantization unit 336 (including coarse gain quantization unit 338, fine gain quantization unit 340, and adder 342).
[0134] In contrast, PAED 326B includes a general quantizer 380 that quantizes frequency domain audio data 329 to produce quantized audio data 381 that is output to dynamic bit allocation unit 334 and bitstream encoder 344. Quantizer 380 can perform any form of quantization, whether coarse / fine quantization, hierarchical quantization, multi-step quantization, single-step quantization, dynamic quantization, and / or static quantization. Quantizer 380 also outputs bit allocations to dynamic SBR unit 350, static SBR unit 352, and bitstream encoder 344.
[0135] Figure 4A and Figure 4B is shown in more detail Figure 1 and Figure 2 A block diagram of an example of a psychoacoustic audio decoder. Figure 4A In the example, PADD 434A indicates Figure 1 and Figure 2 Another example of PADD 34 and PADD 134 is shown in the example of . PADD 434A may represent an audio decoder configured to perform psychoacoustic audio decoding according to an AptX decompression algorithm.
[0136] like Figure 4A As further shown in the example of , the PADD 434A includes a bitstream decoder 436, a shape vector dequantizer 438, an energy dequantizer 440 (including a coarse gain dequantizer 442, a fine gain dequantizer 444, an adder 446), a gain-shape synthesis unit 448, and an inverse transform codec 450. The bitstream decoder 436 may represent a unit configured to extract a coarse energy 339, a fine energy 341, and a residual ID 333 from the encoded audio data 31. The bitstream decoder 436 may extract one or more of the coarse energy 339, the fine energy 341, and the residual ID 333 based on the coarse energy bit allocation 349 and the fine energy bit allocation 351. The bitstream decoder 436 may output the coarse energy 339 to the coarse gain dequantizer 442, output the quantized fine energy 341 to the fine gain dequantizer 444, and output the residual ID 333 to the shape vector dequantizer 438.
[0137] The energy dequantizer 440 may be represented as a device configured to Figure 3A A unit that performs inverse quantization in the opposite manner to the quantization performed by the energy quantizer 336 shown. The energy inverse quantizer 336 can perform inverse quantization on the coarse energy 339 and the fine energy 441 to obtain a predicted / difference energy level, and the energy inverse quantizer 336 can perform an inverse prediction or difference calculation to obtain the energy level 331A'. The energy inverse quantizer 440 can output the energy level 331A' to the gain-shape synthesis unit 448.
[0138] If the encoded audio data 31 includes a syntax element set to a value indicating that the fine energy 349 is hierarchically quantized, the energy dequantizer 440 can hierarchically dequantize the fine energy 341. In some examples, the encoded audio data 31 may include a syntax element indicating whether the hierarchically quantized fine energy 349 is formed using the same hierarchical quantization structure across all subbands or using a corresponding hierarchical quantization structure determined separately for each subband. Based on the value of the syntax element, the energy dequantizer 440 can apply the same hierarchical dequantization structure across all subbands, as represented by the fine energy 341, or can update the hierarchical dequantization structure on a per-subband basis when dequantizing the fine energy 341.
[0139] In operation, the bitstream decoder 436 may parse the bit allocation 349 of the coarse energy 339 from the bitstream 31 and provide the bit allocation 349 to the coarse gain dequantizer 442. The coarse gain dequantizer 442 may represent a unit configured to perform dequantization on the coarse gain 339 based on the bit allocation 349 to obtain a dequantized coarse energy 443. The coarse gain dequantizer 442 may output the dequantized coarse energy 443 to the adder 446.
[0140] The bitstream decoder 436 may parse the bit allocation 351 of the fine energy 341 from the bitstream 31 and provide the bit allocation 351 to the fine gain dequantizer 444. The fine gain dequantizer 444 may represent a unit configured to perform dequantization on the fine gain 341 based on the bit allocation 351 to obtain the dequantized fine energy 445. The fine gain dequantizer 444 may output the dequantized fine energy 445 to the adder 446.
[0141] Adder 446 may represent being configured to sum or in other words add the dequantized coarse gain 443 and the dequantized fine gain 445 to obtain energy level 331A', where the prime sign indicates that energy level 331A' may be different from energy level 331A due to the lossy quantization operation. Adder 446 may output energy level 331A' to a gain-shape synthesis unit 448.
[0142] As described above, the bitstream decoder 436 can parse the residual ID 333 to the shape vector dequantizer 438, which can represent a unit configured to perform vector dequantization in a manner opposite to the vector quantization performed by the shape vector quantizer 332. The shape vector dequantizer 438 can perform vector dequantization on the residual ID 333 to obtain the residual vector 331B'. The shape vector dequantizer 438 can output the residual vector 331B' to the gain-shape synthesis unit 448, where again, the prime sign of the residual vector 331B' indicates that the residual vector 331B' may be different from the residual vector 331B due to the lossy quantization operation.
[0143] The shape vector dequantizer 438 may perform vector dequantization based on the dynamically determined bit allocation 337. Figure 4A As further shown in the example of FIG. 4 , PADD 434A includes a dynamic bit allocation unit 452, which can represent a unit configured to apply PM 335A to the dequantized energy level 331A′ to obtain a dynamic bit allocation 337. The dynamic bit allocation unit 452 can be configured as described above with respect to Figure 3A The dynamic bit allocation unit 452 may operate in a similar, if not substantially similar, manner to the dynamic bit allocation unit 334 described in the example of FIG. The dynamic bit allocation unit 452 may output the dynamic bit allocation 337 to the shape vector dequantizer 438 and the dynamic SBR unit 454 .
[0144] The gain-shape synthesis unit 448 may be represented as being configured to be Figure 3A The gain-shape synthesis unit 448 may be a unit that operates in an opposite manner to the gain-shape analysis unit 330 to reconstruct the frequency domain audio data 329'. The gain-shape synthesis unit 448 may reconstruct the frequency domain audio data 329' based on the energy level 331A' and the residual vector 331B', wherein again, the prime sign of the frequency domain audio data 329' may indicate that there is a slight difference between the frequency domain audio data 329' and the frequency domain audio data 329 due to the lossy quantization operation. The gain-shape synthesis unit 448 may output the frequency domain audio data 329' to the inverse transform codec 450 and the spectrum perception analysis unit 456.
[0145] The PADD 434A may include a spectrum sensing analysis unit 456, which may represent a unit configured to perform the same operations as described above with respect to the spectrum sensing analysis unit. Figure 3AThe spectral perception analysis unit 456 may be a unit that operates in a manner similar to (if not substantially similar to) the spectral perception analysis unit 324 described in the example of . The spectral perception analysis unit 456 may perform spectral perception analysis on the frequency domain audio data 329' to determine the spectral perception importance of the encoded foreground audio signal (represented by the frequency domain audio data 329') relative to the sound field represented by the ATF audio data 25'. The spatial perception analysis unit 456 may output an indication 325 of the spectral perception importance to the inverse transform codec 450 and the static SBR unit 458.
[0146] The inverse transform codec 450 may represent a unit configured to operate in an opposite manner to the transform codec 328. As such, the inverse transform codec 450 may apply an inverse transform to the frequency domain audio data 329 to obtain a transmission channel 451, which may represent a time domain audio signal representing a foreground audio signal (and potentially a background audio signal). The inverse transform may transform the frequency domain audio signal 329' from the frequency domain to the time domain, wherein the time domain audio signal represents the transmission channel 451. The inverse transform codec 450 may output the transmission channel 451 to the ATF codec 466.
[0147] like Figure 4A As further shown in the example of , PADD 434A may include a dynamic SBR unit 454 and a static SBR unit 458. Dynamic SBR unit 454 may represent a unit configured to operate in a manner similar to (if not substantially similar to) dynamic SBR unit 350. Dynamic SBR unit 454 may apply PM 335B to dynamic bit allocation 337 and fine bit allocation 351 to obtain dynamic bit redistribution 361. Dynamic SBR unit 454 may output dynamic bit redistribution 361 to spatiotemporal bit distributor 460.
[0148] Static SBR unit 458 may represent a unit configured to operate in a manner similar, if not substantially similar, to static SBR unit 352. Static SBR unit 458 processes indication 325 and coarse bit allocation 349 to obtain static bit redistribution 359. Static SBR unit 458 may output static bit redistribution 359 to spatiotemporal bit distributor 460.
[0149] like Figure 4A As shown in the example of , PADD 434A may further include a spatiotemporal bit distributor 460, an ATF bit allocation unit 462, and a spatial component inverse quantizer 54. The spatiotemporal bit distributor 460 may represent a device configured to perform the same operation as described above with respect to Figure 3AThe spatiotemporal bit distributor 460 may be a unit that operates in a manner similar to (if not substantially similar to) the spatiotemporal bit distributor 354 described in the example of FIG. The spatiotemporal bit distributor 460 may process the dynamic bit redistribution 361 and the static bit redistribution 359 to obtain the spatiotemporal bit distribution 355. The spatiotemporal bit distributor 460 may output the spatiotemporal bit distribution 355 to the ATF bit allocation unit 462.
[0150] The ATF bit allocation unit 462 may be configured to be similar to the above Figure 3A The ATF bit allocation unit 462 may be a unit that operates in a manner similar, if not substantially similar, to the ATF bit allocation unit 356 described in the example of FIG. The ATF bit allocation unit 462 may process the spatiotemporal bit allocation 355 to obtain the spatial component bit allocation 357. The ATF bit allocation unit 462 may output the spatial component bit allocation 357 to the spatial component inverse quantizer 54.
[0151] The spatial component dequantizer 54 may be represented as being configured as described above with respect to Figure 3A The spatial component dequantizer 54 may dequantize the spatial component 47 based on the spatial component bit allocation 357 to obtain the dequantized spatial component 45'. The spatial component dequantizer 54 may output the dequantized spatial component 45' to the ATF codec 466.
[0152] The ATF codec 466 may represent a unit configured to reconstruct the ATF audio data 25' based on the transmission channels 451 and the dequantized spatial components 45'. The ATF codec 466 may multiplex the transmission channels 451 and the dequantized spatial components 45' to formulate the ATF audio data 25' (such as according to the HTF described above). As described above, the ATF codec 466 may output the ATF audio data 25' to the SADD 136.
[0153] Next reference Figure 4B For example, PADD 434B could be similar to Figure 4A , except that PADD 434B does not include shape vector dequantizer 438 and gain-shape synthesis unit 448. In addition, energy dequantizer 440 of PADD 434B does not include coarse / fine quantization and can perform any form of dequantization, whether coarse / fine dequantization, hierarchical dequantization, multi-step dequantization, single-step dequantization, dynamic dequantization, and / or static dequantization. In this way, PADD 434B can be used with Figure 3B The PAED 326B shown in the example of FIG. 1 operates in reverse.
[0154] Figure 5 It is shown Figure 2 A block diagram of exemplary components of a source device is shown in the example of FIG. Figure 5 4, source device 112 includes a processor 412, a graphics processing unit (GPU) 414, a system memory 416, a display processor 418, one or more integrated speakers 440, a display 103, a user interface 420, an antenna 421, and a transceiver module 422. In an example where source device 112 is a mobile device, display processor 418 is a mobile display processor (MDP). In some examples, such as examples where source device 112 is a mobile device, processor 412, GPU 414, and display processor 418 may be formed as an integrated circuit (IC).
[0155] For example, an IC may be considered a processing chip within a chip package and may be a system on chip (SoC). In some examples, two of processor 412, GPU 414, and display processor 418 may be housed together in the same IC, and the other may be housed in a different integrated circuit (i.e., a different chip package), or all three may be housed in different ICs or on the same IC. However, in an example where source device 12 is a mobile device, processor 412, GPU 414, and display processor 418 may all be housed in different integrated circuits.
[0156] Examples of processor 412, GPU 414, and display processor 418 include, but are not limited to, one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Processor 412 may be a central processing unit (CPU) of source device 12. In some examples, GPU 414 may be dedicated hardware that includes integrated and / or discrete logic circuits that provide GPU 414 with massively parallel processing capabilities suitable for graphics processing. In some cases, GPU 414 may also include general-purpose processing capabilities and may be referred to as a general-purpose GPU (GPGPU) when performing general-purpose processing tasks (i.e., non-graphics-related tasks). Display processor 418 may also be specialized integrated circuit hardware that is designed to retrieve image content from system memory 416, combine the image content into image frames, and output the image frames to display 103.
[0157] The processor 412 may execute various types of applications 20. Examples of applications 20 include web browsers, email applications, spreadsheets, video games, other applications that generate visual objects for display, or any of the types of applications listed in more detail above. The system memory 416 may store instructions for executing the applications 20. Executing one of the applications 20 on the processor 412 causes the processor 412 to generate graphics data for image content to be displayed and audio data 21 to be played (possibly via the integrated speakers 105). The processor 412 may transmit the graphics data for the image content to the GPU 414, and for further processing based on instructions or commands sent by the processor 412 to the GPU 414.
[0158] Processor 412 may communicate with GPU 414 according to an application specific processing interface (API). Examples of such APIs include of API, Khronos Group or OpenGL and OpenCL TM However, aspects of the present disclosure are not limited to the APIs of DirectX, OpenGL or OpenCL, and can be extended to other types of APIs. In addition, the techniques described in the present disclosure do not need to work according to the API, and the processor 412 and the GPU 414 can communicate using any technology.
[0159] System memory 416 may be memory for source device 12. System memory 416 may include one or more computer-readable storage media. Examples of system memory 416 include, but are not limited to, random access memory (RAM), electrically erasable programmable read-only memory (EEPROM), flash memory, or other media that may be used to carry or store desired program code in the form of instructions and / or data structures that may be accessed by a computer or processor.
[0160] In some examples, system memory 416 may include instructions that cause processor 412, GPU 414, and / or display processor 418 to perform the functions attributed in this disclosure to processor 412, GPU 414, and / or display processor 418. Thus, system memory 416 may be a computer-readable storage medium having stored thereon instructions that, when executed, cause one or more processors (e.g., processor 412, GPU 414, and / or display processor 418) to perform various functions.
[0161] System memory 416 may include non-transitory storage media. The term "non-transitory" means that the storage medium is embodied in a carrier wave or propagating signal. However, the term "non-transitory" should not be interpreted as meaning that system memory 416 is non-removable or that its contents are static. As an example, system memory 416 may be removed from source device 12 and moved to another device. As another example, a memory (substantially similar to system memory 416) may be inserted into source device 12. In some examples, non-transitory storage media may store data that may change over time (e.g., in RAM).
[0162] The user interface 420 may represent one or more hardware or virtual (meaning a combination of hardware and software) user interfaces through which a user may interact with the source device 12. The user interface 420 may include physical buttons, switches, toggles, lights, or virtual versions thereof. The user interface 420 may also include a physical or virtual keyboard, a touch interface (such as a touch screen), tactile feedback, and the like.
[0163] The processor 412 may include one or more hardware units (including so-called "processing cores") configured to perform all or part of the operations discussed above with respect to one or more of the mixing unit 120, the audio encoder 122, the wireless connection manager 128, and the wireless communication unit 130. The antenna 421 and the transceiver module 422 may represent a unit configured to establish and maintain a wireless connection between the source device 12 and the sink device 114. The antenna 421 and the transceiver module 422 may represent one or more receivers and / or one or more transmitters capable of wireless communication according to one or more wireless communication protocols. That is, the transceiver module 422 may represent a separate transmitter, a separate receiver, a separate transmitter and a separate receiver, or a combined transmitter and receiver. The antenna 421 and the transceiver 422 may be configured to receive encoded audio data that has been encoded according to the techniques of the present disclosure. Similarly, the antenna 421 and the transceiver 422 may be configured to send encoded audio data that has been encoded according to the techniques of the present disclosure. The transceiver module 422 may perform all or part of the operations of one or more of the wireless connection manager 128 and the wireless communication unit 130 .
[0164] Figure 6 It is shown Figure 2 Although sink device 114 may include components similar to those described above with respect to Figure 5 The examples of source device 112 and source device 114 are discussed in greater detail, but in some cases, sink device 14 may include only a subset of the components discussed above with respect to source device 112.
[0165] exist Figure 6 In the example of , sink device 114 includes one or more speakers 802, processor 812, system memory 816, user interface 820, antenna 821, and transceiver module 822. Processor 812 can be similar to or substantially similar to processor 812. In some cases, processor 812 can be different from processor 412 in terms of total processing power or can be tailored for low power consumption. System memory 816 can be similar to or substantially similar to system memory 416. Speaker 140, user interface 820, antenna 821, and transceiver module 822 can be similar to or substantially similar to corresponding speaker 440, user interface 420, and transceiver module 422. Sink device 114 can also optionally include display 800, although display 800 can represent a low-power, low-resolution (perhaps black and white LED) display through which limited information is conveyed, which can be driven directly by processor 812.
[0166] The processor 812 may include one or more hardware units (including so-called "processing cores") configured to perform all or part of the operations discussed above with respect to one or more of the wireless connection manager 150, the wireless communication unit 152, and the audio decoder 132. The antenna 821 and the transceiver module 822 may represent units configured to establish and maintain a wireless connection between the source device 112 and the sink device 114. The antenna 821 and the transceiver module 822 may represent one or more receivers and one or more transmitters capable of wireless communication according to one or more wireless communication protocols. The antenna 821 and the transceiver 822 may be configured to receive encoded audio data that has been encoded according to the techniques of the present disclosure. Similarly, the antenna 821 and the transceiver 822 may be configured to send encoded audio data that has been encoded according to the techniques of the present disclosure. The transceiver module 822 may perform all or part of the operations of one or more of the wireless connection manager 150 and the wireless communication unit 152.
[0167] Figure 7 It is shown Figure 1 Flowchart of exemplary operations of an audio encoder shown in when performing various aspects of the technology described in the present disclosure. The audio encoder 22 may first call the spatial audio encoding device 24 to perform spatial audio encoding on the scene-based audio data 21, and thereby obtain a foreground audio signal and corresponding spatial components (700). However, the spatial audio encoding performed by the spatial audio encoding device 24 omits the above-mentioned quantization of the spatial components because the quantization is again offloaded to the psychoacoustic audio encoding device 26. The spatial audio encoding device 24 may output ATF audio data 25 representing the foreground audio signal and the corresponding spatial components to the psychoacoustic audio encoding device 26.
[0168] The audio encoder 22 calls the psychoacoustic audio encoding device 26 to perform psychoacoustic audio encoding on the foreground audio signal to obtain an encoded foreground audio signal (702). When performing psychoacoustic audio encoding on the foreground audio signal, the psychoacoustic audio encoding device 26 can determine a first bit allocation for the foreground audio signal (704). The psychoacoustic audio encoding device 26 can determine a second bit allocation for the spatial component based on the first bit allocation for the foreground audio signal (706). Because the second bit allocation is determined based on the first bit allocation, the psychoacoustic audio encoding device 26 more fully integrates quantization across the foreground audio signal and the spatial component. The psychoacoustic audio encoding device 26 can call the SCQ 46 to pass the second bit allocation to the SCQ 46.
[0169] The SCQ 46 may apply quantization, such as vector quantization, to achieve a bit allocation for the spatial component that meets or exceeds the second bit allocation. In this manner, the SCO 46 may quantize the spatial component based on the second bit allocation for the spatial component to obtain a quantized spatial component (708). The psychoacoustic audio encoding device 26 may then specify the encoded foreground audio signal and the quantized spatial component in the bitstream 31 (710).
[0170] Figure 8 It is shown Figure 1 32 in performing various aspects of the techniques described in the present disclosure. As described above, the audio decoder 32 can operate inversely to the audio encoder 22. In this way, the audio decoder 32 can obtain an encoded foreground audio signal and corresponding quantized spatial components from the bitstream 31 (720). The audio decoder 32 can call the psychoacoustic audio decoding device 34 to perform psychoacoustic audio decoding on the encoded foreground audio signal to obtain a foreground audio signal (722).
[0171] When psychoacoustic audio encoding is performed on the foreground audio signal, the psychoacoustic audio decoding device 34 can determine a first bit allocation of the encoded foreground audio signal (724). The psychoacoustic audio decoding device 34 can also determine a second bit allocation of the quantized spatial component based on the first bit allocation of the encoded foreground audio signal (726). The psychoacoustic audio decoding device 34 can call a spatial component dequantizer (SCD) 54, which can dequantize the quantized spatial component based on the second bit allocation of the spatial component to obtain the spatial component (728). The psychoacoustic audio decoding device 34 can reconstruct the ATF audio data 25' based on the foreground audio signal and the spatial component. The spatial audio decoding device 36 can then reconstruct the scene-based audio data 21' based on the foreground audio signal and the spatial component (730).
[0172] The foregoing aspects of these techniques may be implemented in accordance with the following clauses.
[0173] Item 1B. A device configured to encode scene-based audio data, the device comprising: a memory configured to store the scene-based audio data; and one or more processors configured to: perform spatial audio encoding on the scene-based audio data to obtain a foreground audio signal and a corresponding spatial component, the spatial component defining a spatial characteristic of the foreground audio signal; perform psychoacoustic audio encoding on the foreground audio signal to obtain an encoded foreground audio signal; when performing psychoacoustic audio encoding on the foreground audio signal, determine a first bit allocation of the foreground audio signal; determine a second bit allocation of the spatial component based on the first bit allocation of the foreground audio signal; quantize the spatial component based on the second bit allocation of the spatial component to obtain a quantized spatial component; and specify the encoded foreground audio signal and the quantized spatial component in a bitstream.
[0174] Clause 2B. The device of clause 1B, wherein the one or more processors are configured to perform psychoacoustic audio encoding according to an AptX compression algorithm on the foreground audio signal to obtain an encoded foreground audio signal.
[0175] Clause 3B. A device according to any combination of clauses 1B and 2B, wherein one or more processors are configured to: perform a shape and gain analysis on a foreground audio signal to obtain a shape and gain representing the foreground audio signal; perform quantization on the gain to obtain a coarse quantization gain and one or more fine quantization residuals; and determine a second bit allocation based on the number of bits allocated to each of the coarse quantization gain and the one or more fine quantization residuals.
[0176] Clause 4B. A device according to any combination of clauses 1B to 3B, wherein the one or more processors are configured to: determine a spectral perceptual importance of the foreground audio signal relative to a sound field represented by the scene-based audio data; and determine a first bit allocation based on the spectral perceptual importance.
[0177] Item 5B. A device according to Item 4B, wherein one or more processors are configured to: determine a static spatial bit distribution based on spectral perception importance and the number of bits allocated to coarse quantization gain; determine a dynamic spatial bit distribution based on the number of bits allocated to each of one or more fine quantization residuals; and determine a second bit allocation based on the static spatial bit distribution and the dynamic spatial bit distribution.
[0178] Clause 6B. The device of any combination of clauses 1B to 5B, wherein the one or more processors are configured to perform a linear reversible transform on the scene-based audio data to obtain a foreground audio signal and a corresponding spatial component.
[0179] Clause 7B. The apparatus of any combination of clauses 1B to 6B, wherein the scene-based audio data comprises ambisonic coefficients corresponding to an order greater than one.
[0180] Clause 8B. The apparatus of any combination of clauses 1B to 6B, wherein the scene-based audio data comprises ambisonic coefficients corresponding to orders greater than zero.
[0181] Clause 9B. The apparatus of any combination of Clauses 1B to 8B, wherein the scene-based audio data comprises audio data defined in a spherical harmonic domain.
[0182] Clause 10B. The apparatus of any combination of clauses 1B to 9B, wherein the foreground audio signal comprises a foreground audio signal defined in a spherical harmonic domain, and wherein the spatial components comprise spatial components defined in the spherical harmonic domain.
[0183] Clause 11B. The apparatus of any combination of clauses 1B to 10B, wherein the scene-based audio data comprises mixed-order surround sound audio data.
[0184] Clause 12B. A method for encoding scene-based audio data, the method comprising: performing spatial audio encoding on the scene-based audio data to obtain a foreground audio signal and a corresponding spatial component, the spatial component defining a spatial characteristic of the foreground audio signal; performing psychoacoustic audio encoding on the foreground audio signal to obtain an encoded foreground audio signal; when performing psychoacoustic audio encoding on the foreground audio signal, determining a first bit allocation of the foreground audio signal; determining a second bit allocation of the spatial component based on the first bit allocation of the foreground audio signal; quantizing the spatial component based on the second bit allocation of the spatial component to obtain a quantized spatial component; and specifying the encoded foreground audio signal and the quantized spatial component in a bitstream.
[0185] Clause 13B. The method of clause 12B, wherein performing psychoacoustic audio encoding comprises performing psychoacoustic audio encoding according to an AptX compression algorithm on the foreground audio signal to obtain an encoded foreground audio signal.
[0186] Clause 14B. A method according to any combination of clauses 12B and 13B, wherein determining a first bit allocation comprises: performing a shape and gain analysis on a foreground audio signal to obtain a shape and a gain representing the foreground audio signal; performing quantization on the gain to obtain a coarse quantization gain and one or more fine quantization residuals; and determining a second bit allocation based on the number of bits allocated to each of the coarse quantization gain and the one or more fine quantization residuals.
[0187] Clause 15B. A method according to any combination of clauses 12B to 14B, wherein determining the first bit allocation comprises: determining a spectral perceptual importance of the foreground audio signal relative to the sound field represented by the scene-based audio data; and determining the first bit allocation based on the spectral perceptual importance.
[0188] Clause 16B. A method according to Clause 15B, wherein determining the second bit allocation includes: determining a static spatial bit distribution based on the spectrum perception importance and the number of bits allocated to the coarse quantization gain; determining a dynamic spatial bit distribution based on the number of bits allocated to each of one or more fine quantization residuals; and determining the second bit allocation based on the static spatial bit distribution and the dynamic spatial bit distribution.
[0189] Clause 17B. The method of any combination of clauses 12B to 16B, wherein performing spatial audio encoding comprises: performing a linear reversible transform on the scene-based audio data to obtain a foreground audio signal and a corresponding spatial component.
[0190] Clause 18B. The method of any combination of clauses 12B to 17B, wherein the scene-based audio data comprises ambisonic coefficients corresponding to an order greater than one.
[0191] Clause 19B. The method of any combination of clauses 12B to 17B, wherein the scene-based audio data includes ambisonic coefficients corresponding to orders greater than zero.
[0192] Clause 20B. The method of any combination of Clauses 12B to 19B, wherein the scene-based audio data comprises audio data defined in a spherical harmonic domain.
[0193] Clause 21B. The method of any combination of clauses 12B to 20B, wherein the foreground audio signal comprises a foreground audio signal defined in a spherical harmonic domain, and wherein the spatial components comprise spatial components defined in the spherical harmonic domain.
[0194] Clause 22B. A method according to any combination of clauses 12B to 21B, wherein the scene-based audio data comprises mixed-order surround sound audio data.
[0195] Item 23B. A device configured to encode scene-based audio data, the device comprising: a component for performing spatial audio encoding on the scene-based audio data to obtain a foreground audio signal and a corresponding spatial component, the spatial component defining a spatial characteristic of the foreground audio signal; a component for performing psychoacoustic audio encoding on the foreground audio signal to obtain an encoded foreground audio signal; a component for determining a first bit allocation of the foreground audio signal when performing psychoacoustic audio encoding on the foreground audio signal; a component for determining a second bit allocation of the spatial component based on the first bit allocation of the foreground audio signal; a component for quantizing the spatial component based on the second bit allocation of the spatial component to obtain a quantized spatial component; and a component for specifying the foreground audio signal and the quantized spatial component in a bitstream.
[0196] Clause 24B. The apparatus of Clause 23B, wherein the means for performing psychoacoustic audio encoding comprises means for performing psychoacoustic audio encoding according to an AptX compression algorithm on the foreground audio signal to obtain an encoded foreground audio signal.
[0197] Clause 25B. A device according to any combination of clauses 23B and 24B, wherein the component for determining a first bit allocation includes: a component for performing a shape and gain analysis on a foreground audio signal to obtain a shape and a gain representing the foreground audio signal; a component for performing quantization on the gain to obtain a coarse quantization gain and one or more fine quantization residuals; and a component for determining a second bit allocation based on the number of bits allocated to each of the coarse quantization gain and the one or more fine quantization residuals.
[0198] Clause 26B. A device according to any combination of clauses 23B to 25B, wherein the component for determining the first bit allocation includes: a component for determining the spectral perceptual importance of the foreground audio signal relative to the sound field represented by the scene-based audio data; and a component for determining the first bit allocation based on the spectral perceptual importance.
[0199] Clause 27B. An apparatus according to clause 26B, wherein the component for determining the second bit allocation includes: a component for determining a static spatial bit distribution based on spectral perception importance and the number of bits allocated to coarse quantization gain; a component for determining a dynamic spatial bit distribution based on the number of bits allocated to each of one or more fine quantization residuals; and a component for determining the second bit allocation based on the static spatial bit distribution and the dynamic spatial bit distribution.
[0200] Clause 28B. The apparatus of any combination of clauses 23B to 27B, wherein the means for performing spatial audio encoding comprises means for performing a linear reversible transform on the scene-based audio data to obtain a foreground audio signal and a corresponding spatial component.
[0201] Clause 29B. The apparatus of any combination of clauses 23B to 28B, wherein the scene-based audio data comprises ambisonic coefficients corresponding to an order greater than one.
[0202] Clause 30B. The apparatus of any combination of clauses 23B to 29B, wherein the scene-based audio data comprises ambisonic coefficients corresponding to orders greater than zero.
[0203] Clause 31B. The apparatus of any combination of Clauses 23B to 30B, wherein the scene-based audio data comprises audio data defined in a spherical harmonic domain.
[0204] Clause 32B. The apparatus of any combination of Clauses 23B to 31B, wherein the foreground audio signal comprises a foreground audio signal defined in a spherical harmonic domain, and wherein the spatial components comprise spatial components defined in the spherical harmonic domain.
[0205] Clause 33B. A device according to any combination of clauses 23B to 32B, wherein the scene-based audio data includes mixed-order surround sound audio data.
[0206] Item 34B. A non-temporary computer-readable storage medium having instructions stored thereon, which instructions, when executed, cause one or more processors to: perform spatial audio encoding on scene-based audio data to obtain a foreground audio signal and a corresponding spatial component, which defines spatial characteristics of the foreground audio signal; perform psychoacoustic audio encoding on the foreground audio signal to obtain an encoded foreground audio signal; when performing psychoacoustic audio encoding on the foreground audio signal, determine a first bit allocation for the foreground audio signal; determine a second bit allocation for the spatial component based on the first bit allocation for the foreground audio signal; quantize the spatial component based on the second bit allocation for the spatial component to obtain a quantized spatial component; and specify the foreground audio signal and the quantized spatial component in a bitstream.
[0207] Item 1C. A device configured to decode a bitstream representing encoded scene-based audio data, the device comprising: a memory configured to store the bitstream, the bitstream comprising an encoded foreground audio signal and a corresponding quantized spatial component, the spatial component defining a spatial characteristic of the foreground audio signal; and one or more processors configured to: perform psychoacoustic audio decoding on the encoded foreground audio signal to obtain the foreground audio signal; when performing psychoacoustic audio decoding on the encoded foreground audio signal, determine a first bit allocation of the encoded foreground audio signal; determine a second bit allocation of the quantized spatial component based on the first bit allocation of the encoded foreground audio signal; dequantize the quantized spatial component based on the second bit allocation of the quantized spatial component to obtain the spatial component; and reconstruct the scene-based audio data based on the foreground audio signal and the spatial component.
[0208] Clause 2C. The device of Clause 1C, wherein the one or more processors are configured to perform psychoacoustic audio decoding according to an AptX decompression algorithm on the encoded foreground audio signal to obtain the foreground audio signal.
[0209] Clause 3C. A device according to any combination of clauses 1C and 2C, wherein one or more processors are configured to: obtain from a bitstream a number of bits allocated to each of a coarse quantization gain and one or more fine quantization residuals, the coarse quantization gain and the one or more fine quantization residuals representing a gain of a foreground audio signal; and determine a second bit allocation based on the number of bits allocated to each of the coarse quantization gain and the one or more fine quantization residuals.
[0210] Clause 4C. A device according to any combination of clauses 1C to 3C, wherein one or more processors are configured to: determine a spectral perceptual importance of the encoded foreground audio signal relative to a sound field represented by the scene-based audio data; and determine a first bit allocation based on the spectral perceptual importance.
[0211] Item 5C. A device as described in Item 4C, wherein one or more processors are configured to: determine a static spatial bit distribution based on spectral perception importance and the number of bits allocated to coarse quantization gain; determine a dynamic spatial bit distribution based on the number of bits allocated to each of one or more fine quantization residuals; and determine a second bit allocation based on the static spatial bit distribution and the dynamic spatial bit distribution.
[0212] Clause 6C. The apparatus of any combination of clauses 1C to 5C, wherein the scene-based audio data comprises ambisonic coefficients corresponding to an order greater than one.
[0213] Clause 7C. The apparatus of any combination of Clauses 1C to 5C, wherein the scene-based audio data comprises audio data defined in a spherical harmonic domain.
[0214] Clause 8C. The apparatus of any combination of Clauses 1C to 7C, wherein the foreground audio signal comprises a foreground audio signal defined in a spherical harmonic domain, and wherein the spatial components comprise spatial components defined in the spherical harmonic domain.
[0215] Clause 9C. A device according to any combination of clauses 1C to 8C, wherein the one or more processors are further configured to: render the scene-based audio data to one or more speaker feeds; and reproduce the sound field represented by the scene-based audio data based on the speaker feeds.
[0216] Clause 10C. A device according to any combination of clauses 1C to 8C, wherein one or more processors are further configured to: render scene-based audio data to one or more speaker feeds, and wherein the device includes one or more speakers that are configured to reproduce a sound field represented by the scene-based audio data based on the speaker feeds.
[0217] Clause 11C. A device as defined in any combination of clauses 1C to 10C, wherein the scene-based audio data comprises mixed-order surround sound audio data.
[0218] Clause 12C. A method for decoding a bitstream representing scene-based audio data, the method comprising: obtaining an encoded foreground audio signal and a corresponding quantized spatial component from the bitstream, the quantized spatial component defining a spatial characteristic of the encoded foreground audio signal; performing psychoacoustic audio decoding on the encoded foreground audio signal to obtain the foreground audio signal; when performing psychoacoustic audio decoding on the encoded foreground audio signal, determining a first bit allocation of the encoded foreground audio signal; determining a second bit allocation of the quantized spatial component based on the first bit allocation of the encoded foreground audio signal; dequantizing the quantized spatial component based on the second bit allocation of the quantized spatial component to obtain the spatial component; and reconstructing the scene-based audio data based on the foreground audio signal and the spatial component.
[0219] Clause 13C. The method of Clause 12C, wherein performing psychoacoustic audio decoding comprises performing psychoacoustic audio decoding according to an AptX decompression algorithm on the encoded foreground audio signal to obtain the foreground audio signal.
[0220] Clause 14C. A method according to any combination of clauses 12C and 13C, wherein determining the second bit allocation comprises: obtaining from a bitstream a number of bits allocated to each of a coarse quantization gain and one or more fine quantization residuals, the coarse quantization gain and the one or more fine quantization residuals representing a gain of a foreground audio signal; and determining the second bit allocation based on the number of bits allocated to each of the coarse quantization gain and the one or more fine quantization residuals.
[0221] Clause 15C. A method according to any combination of clauses 12C to 14C, wherein determining the first bit allocation comprises: determining a spectral perceptual importance of the encoded foreground audio signal relative to the sound field represented by the scene-based audio data; and determining the first bit allocation based on the spectral perceptual importance.
[0222] Clause 16C. A method according to Clause 15C, wherein determining the second bit allocation includes: determining a static spatial bit distribution based on the spectrum perception importance and the number of bits allocated to the coarse quantization gain; determining a dynamic spatial bit distribution based on the number of bits allocated to each of one or more fine quantization residuals; and determining the second bit allocation based on the static spatial bit distribution and the dynamic spatial bit distribution.
[0223] Clause 17C. The method of any combination of clauses 12C to 16C, wherein the scene-based audio data comprises ambisonic coefficients corresponding to an order greater than one.
[0224] Clause 18C. The method of any combination of Clauses 12C to 16C, wherein the scene-based audio data comprises audio data defined in a spherical harmonic domain.
[0225] Clause 19C. The method of any combination of clauses 12C to 18C, wherein the foreground audio signal comprises a foreground audio signal defined in a spherical harmonic domain, and wherein the spatial components comprise spatial components defined in the spherical harmonic domain.
[0226] Clause 20C. A method according to any combination of clauses 12C to 19C, further comprising: rendering the scene-based audio data to one or more speaker feeds; and reproducing the sound field represented by the scene-based audio data based on the speaker feeds.
[0227] Clause 21C. A method according to any combination of clauses 12C to 20C, wherein the scene-based audio data comprises mixed-order surround sound audio data.
[0228] Item 22C. A device configured to decode a bitstream representing encoded scene-based audio data, the device comprising: a component for obtaining an encoded foreground audio signal and a corresponding quantized spatial component from the bitstream, the quantized spatial component defining a spatial characteristic of the encoded foreground audio signal; a component for performing psychoacoustic audio decoding on the encoded foreground audio signal to obtain the foreground audio signal; a component for determining a first bit allocation of the encoded foreground audio signal when performing psychoacoustic audio decoding on the encoded foreground audio signal; a component for determining a second bit allocation of the quantized spatial component based on the first bit allocation of the encoded foreground audio signal; a component for dequantizing the quantized spatial component based on the second bit allocation of the quantized spatial component to obtain the spatial component; and a component for reconstructing scene-based audio data based on the foreground audio signal and the spatial component.
[0229] Clause 23C. The apparatus of Clause 22C, wherein the means for performing psychoacoustic audio decoding comprises means for performing psychoacoustic audio decoding according to an AptX decompression algorithm on the encoded foreground audio signal to obtain the foreground audio signal.
[0230] Clause 24C. A device according to any combination of clauses 22C and 23C, wherein the component for determining the second bit allocation includes: a component for obtaining from a bitstream a number of bits allocated to each of a coarse quantization gain and one or more fine quantization residuals, the coarse quantization gain and the one or more fine quantization residuals representing a gain of a foreground audio signal; and a component for determining the second bit allocation based on the number of bits allocated to each of the coarse quantization gain and the one or more fine quantization residuals.
[0231] Clause 25C. An apparatus according to any combination of clauses 22C to 24C, wherein the means for determining the first bit allocation comprises: means for determining the spectral perceptual importance of the encoded foreground audio signal relative to the sound field represented by the scene-based audio data; and means for determining the first bit allocation based on the spectral perceptual importance.
[0232] Clause 26C. An apparatus according to Clause 25C, wherein the component for determining the second bit allocation includes: a component for determining a static spatial bit distribution based on spectral perception importance and the number of bits allocated to coarse quantization gain; and a component for determining a dynamic spatial bit distribution based on the number of bits allocated to each of one or more fine quantization residuals; and determining the second bit allocation based on the static spatial bit distribution and the dynamic spatial bit distribution.
[0233] Clause 27C. The apparatus of any combination of Clauses 22C to 26C, wherein the scene-based audio data comprises ambisonic coefficients corresponding to an order greater than one.
[0234] Clause 28C. The apparatus of any combination of Clauses 22C to 26C, wherein the scene-based audio data comprises audio data defined in a spherical harmonic domain.
[0235] Clause 29C. The apparatus of any combination of Clauses 22C to 28C, wherein the foreground audio signal comprises a foreground audio signal defined in a spherical harmonic domain, and wherein the spatial components comprise spatial components defined in the spherical harmonic domain.
[0236] Clause 30C. A device according to any combination of clauses 22C to 29C, further comprising: components for rendering scene-based audio data to one or more speaker feeds; and components for reproducing a sound field represented by the scene-based audio data based on the speaker feeds.
[0237] Clause 31C. The apparatus of any combination of clauses 22C to 30C, wherein the scene-based audio data comprises mixed-order surround sound audio data.
[0238] Item 32C. A non-temporary computer-readable storage medium storing instructions that, when executed, cause one or more processors to: obtain an encoded foreground audio signal and a corresponding quantized spatial component from a bit stream representing scene-based audio data, the quantized spatial component defining a spatial characteristic of the encoded foreground audio signal; perform psychoacoustic audio decoding on the encoded foreground audio signal to obtain the foreground audio signal; when performing psychoacoustic audio decoding on the encoded foreground audio signal, determine a first bit allocation of the encoded foreground audio signal; determine a second bit allocation of the quantized spatial component based on the first bit allocation of the encoded foreground audio signal; dequantize the quantized spatial component based on the second bit allocation of the quantized spatial component to obtain the spatial component; and reconstruct the scene-based audio data based on the foreground audio signal and the spatial component.
[0239] In some contexts, such as broadcast contexts, the audio encoding device can be divided into a spatial audio encoder and a psychoacoustic audio encoder 26, where the spatial audio encoder performs some form of intermediate compression for the surround sound representation, including gain control, and the psychoacoustic audio encoder (which is also called the "perceptual audio encoder 26") performs perceptual audio compression to reduce data redundancy between transmission channels for gain normalization.
[0240] Furthermore, the above techniques may be performed for any number of different contexts and audio ecosystems, and should not be limited to any of the above contexts or audio ecosystems. Several exemplary contexts are described below, although the techniques should be limited to the exemplary contexts. An exemplary audio ecosystem may include audio content, a movie studio, a music studio, a game audio studio, channel-based audio content, an encoding engine, game audio stems, a game audio encoding / rendering engine, and a transport system.
[0241] Movie studios, music studios, and game audio studios can receive audio content. In some examples, audio content can represent the output of the collection. Movie studios can output channel-based audio content (e.g., 2.0, 5.1, and 7.1), such as by using a digital audio workstation (DAW). Music studios can output channel-based audio content (e.g., 2.0 and 5.1), such as by using a DAW. In either case, the encoding engine can receive and encode channel-based audio content based on one or more codecs (e.g., AAC, AC3, Dolby True HD, Dolby Digital Plus, and DTS Body Audio) for transmission system output. Game audio studios can output one or more game audio stems, such as by using a DAW. Game audio encoding / rendering engines can encode and / or render audio stems into channel-based audio content for transmission system output. Another exemplary context in which these techniques can be performed includes an audio ecosystem, which can include broadcast recording audio objects, professional audio systems, capture on consumer devices, surround sound audio formats, on-device rendering, consumer audio, televisions and accessories, and car audio systems.
[0242] Broadcast recorded audio objects, professional audio systems, and capture on consumer devices can all use surround sound audio formats to encode their output. In this way, audio content can be encoded into a single representation using surround sound audio formats that can be played using on-device rendering, consumer audio, televisions and accessories, and car audio systems. In other words, a single representation of audio content can be played in a universal audio playback system (i.e., as opposed to requiring a specific configuration such as 5.1, 7.1, etc.), such as audio playback system 16.
[0243] Other examples of contexts in which these techniques may be performed include an audio ecosystem that may include acquisition elements and playback elements. Acquisition elements may include wired and / or wireless acquisition devices (e.g., intrinsic (Eigen) microphones), on-device surround sound capture, and mobile devices (e.g., smartphones and tablets). In some examples, the wired and / or wireless acquisition devices may be coupled to the mobile device via (multiple) wired and / or wireless communication channels.
[0244] According to one or more techniques of the present disclosure, a mobile device can be used to obtain a sound field. For example, a mobile device can obtain a sound field via a wired and / or wireless acquisition device and / or surround sound capture on a device (e.g., multiple microphones integrated into a mobile device). The mobile device can then encode the obtained sound field into surround sound coefficients for one or more playback elements. For example, a user of a mobile device can record (obtain its sound field) a live event (e.g., a meeting, conference, play, concert, etc.) and encode the recording into surround sound coefficients.
[0245] The mobile device can also play the surround sound encoded sound field using one or more playback elements. For example, the mobile device can decode the surround sound encoded sound field and output a signal to one or more playback elements, which causes one or more of the playback elements to recreate the sound field. As an example, the mobile device can output the signal to one or more speakers (e.g., speaker arrays, sound bars, etc.) using wireless and / or wireless communication channels. As another example, the mobile device can output the signal to one or more docking stations and / or one or more docking speakers (e.g., smart cars and / or home audio systems) using a docking solution. As another example, the mobile device can output the signal to a set of headphones using headphone rendering, for example, to create realistic binaural sound.
[0246] In some examples, a particular mobile device can both acquire a 3D sound field and later play the same 3D sound field. In some examples, a mobile device can acquire a 3D sound field, encode the 3D sound field into an HOA, and send the encoded 3D sound field to one or more other devices (e.g., other mobile devices and / or other non-mobile devices) for playback.
[0247] Another context in which these techniques may be performed includes an audio ecosystem that may include audio content, a game studio, encoded audio content, a rendering engine, and a transmission system. In some examples, a game studio may include one or more DAWs that may support editing of surround sound signals. For example, one or more DAWs may include surround sound plug-ins and / or tools that may be configured to operate (e.g., work together) with one or more game audio systems. In some examples, a game studio may output a new stem format that supports HOA. In any case, the game studio may output the encoded audio content to a rendering engine that may render the sound field for playback by a transmission system.
[0248] These techniques may also be performed for an exemplary audio acquisition device. For example, these techniques may be performed with respect to an intrinsic microphone, which may include multiple microphones that are collectively configured to record a 3D sound field. In some examples, multiple microphones of the intrinsic microphone may be located on the surface of a substantially spherical ball with a radius of approximately 4 cm. In some examples, the audio encoding device 20 may be integrated into the intrinsic microphone so as to output a bitstream 21 directly from the microphone.
[0249] Another exemplary audio acquisition context may include a production truck that may be configured to receive signals from one or more microphones, such as one or more intrinsic microphones. The production truck may also include an audio encoder, such as Figure 1 A spatial audio encoding device 24.
[0250] In some cases, the mobile device may also include multiple microphones that are collectively configured to record a 3D sound field. In other words, the multiple microphones may have X, Y, Z diversity. In some examples, the mobile device may include a microphone that can be rotated to provide X, Y, Z diversity for one or more other microphones of the mobile device. The mobile device may also include an audio encoder, such as Figure 1 Audio encoder 22.
[0251] The enhanced video capture device can be further configured to record a 3D sound field. In some examples, the enhanced video capture device can be attached to a helmet of a user engaging in an activity. For example, the enhanced video capture device can be attached to a helmet of a user whitewater rafting. In this way, the enhanced video capture device can capture a 3D sound field representing the action around the user (e.g., water crashing behind the user, another raftsman talking in front of the user, etc.).
[0252] These techniques can also be performed for an accessory-enhanced mobile device, which can be configured to record a 3D sound field. In some examples, the mobile device can be similar to the mobile device discussed above, but with one or more accessories added. For example, an intrinsic microphone can be attached to the above-mentioned mobile device to form an accessory-enhanced mobile device. In this way, the accessory-enhanced mobile device can capture a higher quality version of the 3D sound field than using only the sound capture component integrated with the accessory-enhanced mobile device.
[0253] An exemplary audio playback device that can perform various aspects of the techniques described in this disclosure is discussed further below. According to one or more techniques of this disclosure, the speakers and / or soundbar can be arranged in any arbitrary configuration while still playing a 3D sound field. In addition, in some examples, the headphone playback device can be coupled to the decoder 32 (which refers to the decoder 32) via a wired or wireless connection. Figure 1According to one or more techniques of the present disclosure, a single generic representation of a sound field can be used to render the sound field on any combination of speaker, soundbar, and headphone playback devices.
[0254] Many different exemplary audio playback environments may also be suitable for performing various aspects of the techniques described in this disclosure. For example, a 5.1 speaker playback environment, a 2.0 (e.g., stereo) speaker playback environment, a 9.1 speaker playback environment with full-height front speakers, a 22.2 speaker playback environment, a 16.0 speaker playback environment, a car speaker playback environment, and a mobile device with an earbud playback environment may be suitable environments for performing various aspects of the techniques described in this disclosure.
[0255] According to one or more techniques of the present disclosure, a single universal representation of a sound field can be used to render the sound field on any of the aforementioned playback environments. In addition, the techniques of the present disclosure enable the renderer to render the sound field from the universal representation for playback in playback environments other than the aforementioned environments. For example, if design considerations do not permit proper placement of the speakers according to a 7.1 speaker playback environment (e.g., if placement of a right surround speaker is not possible), the techniques of the present disclosure enable the renderer to compensate with the other 6 speakers, thereby enabling playback in a 6.1 speaker playback environment.
[0256] In addition, the user may be wearing headphones to watch a sports game. According to one or more techniques of the present disclosure, a 3D sound field of a sports game can be obtained (for example, one or more intrinsic microphones can be placed in and / or around a baseball field), surround sound coefficients corresponding to the 3D sound field can be obtained and sent to a decoder, the decoder can reconstruct the 3D sound field based on the surround sound coefficients and output the reconstructed 3D sound field to a renderer, the renderer can obtain an indication of the type of playback environment (for example, headphones) and render the reconstructed 3D sound field into a signal that causes the headphones to output a 3D sound field representation of the sports game.
[0257] In each of the various examples described above, it should be understood that the audio encoding device 22 may perform a method or otherwise include components for performing each step of the method that the audio encoding device 22 is configured to perform. In some instances, the components may include one or more processors. In some instances, one or more processors may represent a dedicated processor configured by instructions stored in a non-transitory computer-readable storage medium. In other words, various aspects of the technology in each set of encoding examples may provide a non-transitory computer-readable storage medium having instructions stored thereon, which, when executed, cause one or more processors to perform the method that the audio encoding device 20 has been configured to perform.
[0258] In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored as one or more instructions or codes on or sent through a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media, which corresponds to tangible media such as data storage media. Data storage media may be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, codes, and / or data structures for implementing the techniques described in the present disclosure. A computer program product may include a computer-readable medium.
[0259] By way of example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory, or any other medium that can be used to store the desired program code in the form of instructions or data structures and can be accessed by a computer. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals or other temporary media, but rather involve non-temporary, tangible storage media. As used herein, disks and optical disks include compact disks (CDs), laser optical disks, optical optical disks, digital versatile disks (DVDs), floppy disks, and Blu-ray disks, where disks typically reproduce data magnetically, while optical disks reproduce data optically using lasers. The above combinations should also be included within the scope of computer-readable media.
[0260] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Therefore, the term "processor" as used herein may refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. In addition, in some aspects, the functionality described herein may be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into a combined codec. In addition, these techniques may be fully implemented in one or more circuits or logic elements.
[0261] The techniques of the present disclosure may be implemented in a variety of devices or apparatuses, including wireless handsets, integrated circuits (ICs), or a set of ICs (e.g., chipsets). Various components, modules, or units are described in this disclosure to emphasize functional aspects of devices configured to perform the disclosed techniques, but do not necessarily need to be implemented by different hardware units. Instead, as described above, the various units may be combined in a codec hardware unit or provided by a collection of interoperable hardware units including one or more processors as described above in combination with appropriate software and / or firmware.
[0262] Furthermore, as used herein, "A and / or B" means "A or B", or "A and B".
[0263] Various aspects of the technology have been described. These and other aspects of the technology are within the scope of the following claims.
Claims
1. A device for encoding scene-based audio data, the device comprising: a memory configured to store scene-based audio data; as well as One or more processors configured to: performing spatial audio coding on the scene-based audio data to obtain a foreground audio signal and a corresponding spatial component, the spatial component defining a spatial characteristic of the foreground audio signal; performing psychoacoustic audio coding on the foreground audio signal to obtain an encoded foreground audio signal; Performing shape and gain analysis on the foreground audio signal to obtain a shape and gain representing the foreground audio signal; determining a first bit allocation of the foreground audio signal when performing psychoacoustic audio coding on the foreground audio signal; Performing quantization on the gain to obtain a coarse quantization gain and one or more fine quantization residuals; determining a second bit allocation for the spatial component based on the first bit allocation for the foreground audio signal, including determining the second bit allocation based on a number of bits allocated to each of the coarse quantization gain and one or more fine quantization residuals; quantizing the spatial component based on the second bit allocation of the spatial component to obtain a quantized spatial component; as well as The encoded foreground audio signal and the quantized spatial components are specified in a bitstream.
2. The device according to claim 1, wherein: The one or more processors are configured to: determining a spectral perceptual importance of the foreground audio signal relative to a sound field represented by the scene-based audio data; and Based on the spectrum perception importance, the first bit allocation is determined.
3. The device according to claim 2, wherein: The one or more processors are configured to: determining a static spatial bit distribution based on the spectrum perceptual importance and the number of bits allocated to the coarse quantization gain; determining a dynamic spatial bit distribution based on a number of bits allocated to each of the one or more refined quantized residuals; as well as The second bit allocation is determined based on the static spatial bit distribution and the dynamic spatial bit distribution.
4. The device according to claim 1, wherein: The one or more processors are configured to perform a linear reversible transform on the scene-based audio data to obtain the foreground audio signal and the corresponding spatial component.
5. The device according to claim 1, wherein: The scene-based audio data includes ambisonic coefficients corresponding to an order greater than one.
6. The device according to claim 1, wherein: The scene-based audio data includes ambisonic coefficients corresponding to orders greater than zero.
7. The device according to claim 1, wherein: The scene-based audio data includes audio data defined in a spherical harmonic domain.
8. The device according to claim 1, in, The foreground audio signal comprises a foreground audio signal defined in a spherical harmonic domain, and The spatial components include spatial components defined in the spherical harmonic domain.
9. The device according to claim 1, wherein: The scene-based audio data includes mixed-order surround sound audio data.
10. The device of claim 1, further comprising a microphone configured to capture the scene-based audio data.
11. A method for encoding scene-based audio data, the method comprising: performing spatial audio coding on the scene-based audio data to obtain a foreground audio signal and a corresponding spatial component, the spatial component defining a spatial characteristic of the foreground audio signal; performing psychoacoustic audio coding on the foreground audio signal to obtain an encoded foreground audio signal; Performing shape and gain analysis on the foreground audio signal to obtain a shape and gain representing the foreground audio signal; determining a first bit allocation of the foreground audio signal when performing psychoacoustic audio coding on the foreground audio signal; Performing quantization on the gain to obtain a coarse quantization gain and one or more fine quantization residuals; determining a second bit allocation for the spatial component based on the first bit allocation for the foreground audio signal, including determining the second bit allocation based on a number of bits allocated to each of the coarse quantization gain and one or more fine quantization residuals; quantizing the spatial component based on the second bit allocation of the spatial component to obtain a quantized spatial component; as well as The encoded foreground audio signal and the quantized spatial components are specified in a bitstream.
12. An apparatus for decoding a bitstream representing encoded scene-based audio data, the apparatus comprising: a memory configured to store the bitstream, the bitstream comprising an encoded foreground audio signal and a corresponding quantized spatial component, the spatial component defining a spatial characteristic of the foreground audio signal; as well as One or more processors configured to: performing psychoacoustic audio decoding on the encoded foreground audio signal to obtain a foreground audio signal; determining a first bit allocation of the encoded foreground audio signal when performing the psychoacoustic audio decoding on the encoded foreground audio signal; obtaining from the bitstream a number of bits allocated to each of a coarse quantization gain and one or more fine quantization residuals, the coarse quantization gain and the one or more fine quantization residuals representing a gain of the foreground audio signal; determining a second bit allocation for the quantized spatial component based on the first bit allocation for the encoded foreground audio signal, comprising determining the second bit allocation based on a number of bits allocated to each of the coarse quantization gain and one or more fine quantization residuals; Dequantizing the quantized spatial component based on the second bit allocation of the quantized spatial component to obtain a spatial component; as well as The scene based audio data is reconstructed based on the foreground audio signal and the spatial component.
13. The device according to claim 12, wherein: The one or more processors are configured to: determining a spectral perceptual importance of the encoded foreground audio signal relative to a sound field represented by the scene-based audio data; and Based on the spectrum perception importance, the first bit allocation is determined.
14. The device according to claim 13, wherein: The one or more processors are configured to: determining a static spatial bit distribution based on the spectrum perceptual importance and the number of bits allocated to the coarse quantization gain; determining a dynamic spatial bit distribution based on a number of bits allocated to each of the one or more refined quantized residuals; as well as The second bit allocation is determined based on the static spatial bit distribution and the dynamic spatial bit distribution.
15. The apparatus according to claim 12, wherein: The scene-based audio data includes ambisonic coefficients corresponding to an order greater than one.
16. The apparatus according to claim 12, wherein: The scene-based audio data includes audio data defined in a spherical harmonic domain.
17. The device according to claim 12, in, The foreground audio signal comprises a foreground audio signal defined in a spherical harmonic domain, and The spatial components include spatial components defined in the spherical harmonic domain.
18. The apparatus according to claim 12, wherein: The one or more processors are further configured to: rendering the scene-based audio data to one or more speaker feeds; as well as Based on the speaker feeds, a sound field represented by the scene based audio data is reproduced.
19. The device according to claim 12, in, The one or more processors are further configured to render the scene-based audio data to one or more speaker feeds, and Therein, the device comprises one or more speakers configured to reproduce a sound field represented by the scene-based audio data based on the speaker feeds.
20. The apparatus of claim 12, wherein: The scene-based audio data includes mixed-order surround sound audio data.
21. The apparatus of claim 12, wherein: The one or more processors are further configured to: rendering the scene-based audio data to one or more speaker feeds; as well as The speaker feeds are output to one or more speakers to reproduce a sound field represented by the scene based audio data based on the speaker feeds.
22. The device according to claim 12, in, The one or more processors are further configured to render the scene-based audio data to one or more speaker feeds, and Therein, the device comprises one or more speakers configured to reproduce a sound field represented by the scene-based audio data based on the speaker feeds.
23. A method for decoding a bitstream representing scene-based audio data, the method comprising: Obtaining from the bitstream an encoded foreground audio signal and corresponding quantized spatial components, the quantized spatial components defining spatial characteristics of the encoded foreground audio signal; performing psychoacoustic audio decoding on the encoded foreground audio signal to obtain a foreground audio signal; determining a first bit allocation of the encoded foreground audio signal when performing psychoacoustic audio decoding on the encoded foreground audio signal; obtaining from the bitstream a number of bits allocated to each of a coarse quantization gain and one or more fine quantization residuals, the coarse quantization gain and the one or more fine quantization residuals representing a gain of the foreground audio signal; determining a second bit allocation for the quantized spatial component based on the first bit allocation for the encoded foreground audio signal, comprising determining the second bit allocation based on a number of bits allocated to each of the coarse quantization gain and one or more fine quantization residuals; Dequantizing the quantized spatial component based on the second bit allocation of the quantized spatial component to obtain a spatial component; as well as The scene based audio data is reconstructed based on the foreground audio signal and the spatial component.
24. A non-transitory computer-readable storage medium having stored thereon instructions that, when executed, cause one or more processors to perform the method of any one of claims 11 or 23.
25. A device for encoding scene-based audio data, the device comprising: means configured to perform spatial audio coding on the scene-based audio data to obtain a foreground audio signal and corresponding spatial components, the spatial components defining spatial characteristics of the foreground audio signal; means configured to perform psychoacoustic audio coding on the foreground audio signal to obtain an encoded foreground audio signal; A component configured to perform a shape and gain analysis on the foreground audio signal to obtain a shape and gain representation of the foreground audio signal; means configured to determine a first bit allocation of the foreground audio signal when performing psychoacoustic audio coding on the foreground audio signal; A component configured to perform quantization on the gain to obtain a coarse quantization gain and one or more fine quantization residuals; means configured to determine a second bit allocation for the spatial component based on the first bit allocation for the foreground audio signal, the second bit allocation being determined based on a number of bits allocated to each of the coarse quantization gain and one or more fine quantization residuals; means configured to quantize the spatial component based on the second bit allocation of the spatial component to obtain a quantized spatial component; as well as Means configured to specify the encoded foreground audio signal and the quantized spatial component in a bitstream.
26. An apparatus for decoding a bitstream representing encoded scene-based audio data, the apparatus comprising: means configured to perform psychoacoustic audio decoding on an encoded foreground audio signal to obtain a foreground audio signal, wherein the bitstream comprises the encoded foreground audio signal and corresponding quantized spatial components defining spatial characteristics of the foreground audio signal; means configured to determine a first bit allocation of the encoded foreground audio signal when performing the psychoacoustic audio decoding on the encoded foreground audio signal; means configured to obtain from the bitstream a number of bits allocated to each of a coarse quantization gain and one or more fine quantization residuals, the coarse quantization gain and the one or more fine quantization residuals representing a gain of the foreground audio signal; means configured to determine a second bit allocation for the quantized spatial components based on the first bit allocation for the encoded foreground audio signal, the second bit allocation being determined based on a number of bits allocated to each of the coarse quantization gain and one or more fine quantization residuals; means configured to, based on the second bit allocation of the quantized spatial component, dequantize the quantized spatial component to obtain a spatial component; and Means configured to reconstruct the scene-based audio data based on the foreground audio signal and the spatial component.
Citation Information
Patent Citations
Quantizing spatial components based on bit allocations determined for psychoacoustic audio coding
US12142285B2
Mixed-order ambisonics (MOA) audio data for computer-mediated reality systems
US20190007781A1
Quantizing spatial components based on bit allocations determined for psychoacoustic audio coding
US20200402522A1
Audio encoder, audio decoder, methods for encoding and decoding an audio signal, audio stream and computer program
CN102089808A
Low bitrate audio encoding / decoding scheme with common preprocessing
CN102124517A
