Method and system for encoding audio data
Through matching tracking encoding algorithms and metadata rendering technology, the audio signal is converted into high-order stereo high-fidelity reverb representation, which solves the problem of bandwidth limitation in traditional audio transmission and realizes high-efficiency and low bit rate audio data transmission.
Patent Information
- Application Number
- CN202410706187.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2024-05-13
- Filing Date
- 2024-06-03
- Publication Date
- 2024-12-03
AI Technical Summary
The transmission of traditional audio content requires a large bandwidth, especially multi-channel audio, object-based audio and stereo high-fidelity reverberation formats, making it difficult to maintain audio quality under limited bandwidth while reducing audio data size. .
The audio content is encoded and decoded using a matching tracking encoding algorithm, and the audio content is converted into a high-order stereo high-fidelity reverb representation, and the audio driver signal is rendered based on metadata, reducing the bit rate while maintaining audio quality.
Effectively transmit high-quality audio content under limited bandwidth, improves the transmission efficiency and quality of audio data, and supports encoding and decoding of multiple audio formats.
Smart Images

Figure CN119068885A_ABST
Abstract
Description
[0001] Related patent applications
[0002] This application claims the benefit of priority to U.S. Provisional Application No. 63 / 506,069, filed on June 3, 2023, which is incorporated herein by reference. Technical Field
[0003] One aspect of the disclosure relates to an audio processing system that encodes different types of audio data into higher order stereo ambisonics (HOA) data for bit rate reduction. Other aspects are also described. Background Art
[0004] A high-order stereo audiophile (HOA) signal may include a three-dimensional (3D) representation of a sound field. Specifically, the sound field may be represented by a sum of weighted spherical harmonic basis functions of increasing order 0, 1, 2, ... As the set of basis functions is extended to include higher order elements (second order and higher), the representation of the sound field becomes more detailed, e.g., with higher acoustic resolution. The weights applied to the basis functions are called spherical harmonic coefficients. Summary of the invention
[0005] According to one aspect of the present disclosure, a decoder-side method includes: receiving a bitstream including an encoded representation of an input audio signal and metadata associated with the input audio signal; generating a decoded representation of the input audio signal by decoding the encoded representation using an algorithm based on matching pursuit (MP) coding; generating a set of audio drive signals by rendering the input audio signal based on the metadata; and driving a set of speakers using the set of audio drive signals.
[0006] In one aspect, the decoded representation comprises a higher order stereo ambisonics (HOA) representation of the input audio signal, wherein the method further comprises applying a transformation matrix to the HOA representation to reconstruct the input audio signal. In another aspect, the input audio signal comprises a set of full-range audio channels in a surround sound format, wherein the method further comprises receiving at least one limited bandwidth audio channel associated with the surround sound format, wherein generating the set of audio drive signals comprises assigning each of the channels to a particular speaker in the set of speakers based on the metadata.
[0007] In one aspect, the input audio signal includes a set of one or more audio objects, and the metadata includes position information associated with the set of one or more audio objects, wherein the set of audio drive signals is generated by spatially rendering the set of one or more audio objects according to the position information. In another aspect, the method further includes: determining a number of the set of one or more audio objects; and determining a transformation matrix based on the number. In another aspect, the method further includes receiving an output speaker layout of the set of speakers, wherein the set of one or more audio objects is spatially rendered according to the output speaker layout.
[0008] In another aspect, a code stream is received from an encoder-side device, wherein the conversion matrix is an inverse matrix of a matrix used by the encoder-side device to generate an encoded representation of an input audio signal.
[0009] In one aspect, the decoded representation of the input audio signal comprises a mixed signal, wherein the method further comprises separating the mixed signal into: a set of surround sound channels in a surround sound format, one or more audio objects comprising one or more audio signals, and HOA data comprising a plurality of HOA signals. On the other hand, generating the set of audio drive signals comprises: rendering the set of surround sound channels, one or more audio signals, and the set of HOA signals according to the metadata and the output speaker layout of the set of speakers; and mixing the rendering into the set of audio drive signals.
[0010] According to another aspect of the present disclosure, an encoder-side method includes: receiving an input audio signal of a piece of audio content and metadata related to the input audio signal; encoding the input audio signal using an algorithm based on MP coding; and transmitting the encoded input audio signal and metadata to an audio playback device.
[0011] In one aspect, the input audio signal includes a set of surround sound audio channels including a sound source, wherein the set of surround sound audio channels includes a first set of one or more full-range audio channels and a second set of one or more limited bandwidth audio channels, wherein the method further includes converting the first set into an HOA representation of the sound source, wherein the encoding includes encoding the HOA representation into a bitstream using an algorithm based on MP encoding for transmission to an audio playback device. On the other hand, the method further includes encoding the second set into the bitstream separately from the encoded HOA representation. On the other hand, the metadata includes surround sound speaker layout information for the set of surround sound audio channels, wherein the first set is converted into the HOA representation based on the surround sound speaker layout information.
[0012] In one aspect, receiving an input audio signal includes receiving a set of one or more audio objects, each audio object having at least one audio signal, wherein the method further includes generating an HOA representation of the set of one or more audio objects, wherein encoding includes encoding the HOA representation into a bitstream using an algorithm based on MP encoding for transmission to an audio playback device. In another aspect, the method further includes determining a number of the set of one or more audio objects; and determining a transformation matrix based on the number, wherein the HOA representation is generated by applying the transformation matrix to the set of one or more audio objects.
[0013] In one aspect, the input audio signal includes: a set of surround sound audio channels of the segment of audio content, an HOA representation of the segment of audio content, and a set of one or more audio objects of the segment of audio content, wherein the method also includes generating a mixed audio signal, the mixed audio signal including the set of surround sound audio channels, the HOA representation, and the set of one or more audio objects, wherein encoding includes encoding the mixed audio signal into a bitstream using an algorithm based on MP encoding for transmission to an audio playback device.
[0014] The above summary does not include an exhaustive list of all aspects of the present disclosure. It is contemplated that the present disclosure includes all systems and methods that can be practiced by all suitable combinations of the various aspects summarized above and disclosed in the detailed description below and specifically pointed out in the claims. Such combinations may have specific advantages not specifically set forth in the above summary of the invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Aspects are shown by way of example and not limitation in the illustrations of the accompanying drawings, where similar reference numerals indicate similar elements. It should be noted that reference to "one" or "an" aspect in the present disclosure is not necessarily the same aspect, and it means at least one. In addition, for the sake of brevity and to reduce the total number of drawings, a certain drawing may be used to illustrate features of more than one aspect, and for a certain aspect, not all elements in the drawing may be required.
[0016] Figure 1 A block diagram of a system for encoding audio content for rendering through one or more speakers is shown according to one aspect.
[0017] Figure 2 A block diagram of a system for encoding and decoding multi-channel audio including converting surround sound audio channels into a higher order stereo Ambisonics (HOA) format is shown according to one aspect.
[0018] Figure 3A block diagram of a system for encoding and decoding multi-channel audio using a transition matrix codebook for converting the audio into a HOA format is shown according to one aspect.
[0019] Figure 4 A block diagram of a system for encoding and decoding multi-channel audio using an identity matrix based on the number of surround sound audio channels of the channel audio is shown according to one aspect.
[0020] Figure 5 A block diagram of a system for encoding and decoding multi-channel audio according to one aspect is shown.
[0021] Figure 6 A block diagram of a system for encoding and decoding audio objects using a transition matrix codebook for assigning audio objects to speaker locations for conversion to an HOA format is shown according to one aspect.
[0022] Figure 7 A block diagram of a system for encoding and decoding audio objects using a transition matrix codebook for assigning audio objects to speaker locations for conversion to an HOA format is shown according to another aspect.
[0023] Figure 8 A block diagram of a system for encoding and decoding audio objects using a transition matrix codebook according to another aspect is shown.
[0024] Fig. 9 A block diagram of a system for encoding and decoding audio objects using an identity matrix according to one aspect is shown.
[0025] Fig.10 A block diagram of a system for encoding and decoding audio objects according to one aspect is shown.
[0026] Fig.11 A block diagram of a system for encoding and decoding mixed audio according to one aspect is shown.
[0027] Fig.12 An example of system hardware is illustrated. DETAILED DESCRIPTION
[0028] The various aspects of the present disclosure will now be explained with reference to the accompanying drawings. As long as the shapes, relative positions and other aspects of the components described in a certain aspect are not clearly defined, the scope of the present disclosure is not limited to the components shown here, and the components shown are only for illustrative purposes. In addition, although many details are set forth, it should be understood that some embodiments can be implemented without these details. In other cases, well-known circuits, structures and technologies are not shown in detail to avoid blurring the understanding of the description. In addition, unless the meaning is clearly contrary, all ranges shown herein are considered to include the end values of each range.
[0029] Traditionally, audio content has been produced, distributed, and consumed using a two-channel stereo format. Other audio formats, such as multi-channel audio, object-based audio, and / or stereo high-fidelity reverberation, are intended to provide a more immersive listener experience. However, the delivery of immersive audio content may be associated with greater bandwidth requirements, such as requiring greater data rates for streaming and downloading than content in a two-channel stereo format. If bandwidth is limited, techniques are needed that can reduce the size of the audio data while maintaining the best possible audio quality. Therefore, there is a need to deliver richer and more immersive audio content using limited bandwidth.
[0030] In order to overcome these defects, the present disclosure provides an audio processing system for encoding and decoding audio content using a matching pursuit (MP) coding algorithm, which can allow the bit rate to be reduced based on the change of the total bandwidth of the system while maintaining the high quality of the audio content. For example, an encoder side device can receive an input audio signal of a section of audio content and metadata related to the input audio signal. For example, the input audio signal may include multi-channel audio content in a surround sound format such as 5.1, and / or may include one or more audio objects. The system can encode any one (or both) of these types of audio content using MP. For example, the system can convert the audio content into a high-order stereo high-fidelity reverberation (HOA) representation, wherein the sound source of the input audio signal can be converted into a three-dimensional (3D) sound field, and the HOA representation can be encoded. The system can transmit the encoded input audio signal together with the received metadata associated with the content as a coded stream to an audio playback device or a decoder side device for decoding and rendering for playback. Therefore, the present disclosure effectively encodes the audio content and transmits it to the audio playback device together with the associated metadata, so that the decoder side device can use the metadata to render the audio content.
[0031] As referred to herein, "audio content" (or audio data) may be (and include) any type of audio, such as a musical composition, a podcast, audio for a virtual reality (VR) environment, a soundtrack for a movie, and the like. In one aspect, the audio content may be part of a segment of audio content, which may be an audio program or an audio file comprising one or more audio signals, the one or more audio signals comprising at least a portion of the audio content. In some aspects, the audio program may be any type of audio content format. In one aspect, the audio program may include audio content for spatial rendering in one or more 3D audio formats as one or more data files, such as having one or more audio channels. For example, the audio program may include a mono audio channel or may be a multi-audio channel format (e.g., two stereo channels, six surround source channels (in a 5.1 surround format), etc.). In another aspect, the audio program may include: one or more audio objects, each audio object having at least one audio signal; and metadata, which may include positional data in 3D sound (audio signals for spatially rendering objects). In another aspect, the audio program may be represented in a spherical audio format (such as an HOA audio format).
[0032] Figure 1A block diagram of an audio processing system (or system) 10 for encoding audio content for rendering through one or more speakers 17 according to one aspect is shown. The system 10 includes an encoder-side device 11 and a decoder-side device 12. In one aspect, each of the devices can be any electronic device that can include electronic components (e.g., one or more processors, memory, etc.) and is capable of performing (e.g., by using one or more electronic components) one or more computing operations for encoding / decoding audio content, transmitting / receiving one or more encoded code streams, and (spatially) rendering audio content (e.g., through one or more speakers). For example, each of the source devices can be a desktop computer, a laptop computer, a digital media player, etc. In one aspect, the device can be a portable electronic device (e.g., handheld), such as a tablet computer, a smart phone, etc. In another aspect, the source device can be a wearable device (e.g., a device designed to be worn on (e.g., attached to) a user's clothing and / or body, such as a smart watch). In another aspect, one of the devices can be a (e.g., remote) electronic server. For example, the encoder side device 11 can be an independent electronic server, a computer (e.g., a desktop computer) or a server computer cluster configured to store, encode, stream (transmit) and / or receive digital audio content (e.g., as one or more audio signals in any audio format). In this case, the encoder side device 11 can transmit the encoded audio content to the decoder side device 12, which can be an audio playback device, such as a smart phone capable of decoding the audio content to play back through one or more speakers.
[0033] As shown, the decoder side device 12 includes one or more speakers 17 for outputting or playing back audio content. In one aspect, one or more speakers can be, for example, electric drivers that can be specially designed for sound output in a specific frequency band, such as woofers, tweeters, or midrange drivers. In one aspect, one or more speakers can be "full-range" (or "full-frequency") electric drivers, each of which reproduces as much of the audible frequency range as possible. In one aspect, the speaker can be integrated in the decoder side device (e.g., the housing of the decoder side device). On the other hand, the speaker can be part of a separate electronic device that can be communicatively coupled with the decoder side device. In some aspects, the speaker can be a loudspeaker or can be part of a headset (e.g., with two speakers). In another embodiment, the device 12 can be part of a playback system with a speaker (e.g., a loudspeaker). On the other hand, the speaker can be part of an electronic device that can be configured to render audio of a virtual (augmented or mixed) reality environment.
[0034] Examples may include a stand-alone speaker, a smart speaker, a home theater system, or an infotainment system integrated into a vehicle. In another aspect, the decoder device may be a head mounted device having a speaker 17 arranged to output sound to the surrounding environment. For example, where the device is a pair of smart glasses, the device may include an "external" speaker arranged to project sound into the surrounding environment (e.g., in a direction away from at least a portion of the wearer, such as an ear or ear canal), in contrast to an "internal" speaker of a pair of headphones that is arranged to project sound into (or towards) the user's ear canal when worn.
[0035] The encoder side device 11 may include several operation blocks, such as a transform 20 (which may be optional), an audio encoder 21, and a metadata encoder 22. The decoder side device 12 may include several blocks, such as an audio decoder 23, an inverse transform 24 (which may be optional), a metadata decoder 25, a (spatial audio) renderer 16, and an optional output channel layout 18. In one aspect, each of these blocks represents one or more operations that can be performed by each corresponding device. Specifically, the operation can be performed by one or more processors (or controllers) integrated into (or coupled to) each device. For example, one or more processors of the encoder side device can be configured to perform the operations of blocks 20-22. In one aspect, the blocks can be merged (or combined). For example, the encoder side device may include an encoder that can be configured to encode audio data and metadata. On the other hand, either device can perform the operations of another device. For example, as described herein, when receiving encoded data, the encoder side device 11 can perform one or more decoder side operations.
[0036] The encoder side device 11 may be configured to receive one or more input audio signals 14 of a section of audio content, N signals (or channels) (or associated therewith), and metadata 15 associated with the received input audio signal. In one aspect, the input audio signal may be any type of audio content in any type of audio format. For example, the input audio signal may include one or more surround sound audio channels (in surround sound formats, such as 5.1, 7.1.4, etc.). In this case, the input signal may include one or more full-range audio channels and one or more limited bandwidth audio channels. For example, in the case of 5.1 surround sound audio content, the input signal may include five full-range audio channels (left front, right front, center channel, and two surround channels) and a low frequency effect (LFE) channel. On the other hand, the input audio signal 14 may include one or more audio objects, each of which includes (or is associated with) one or more audio signals, which include the sound of one or more sound sources and metadata associated with one or more sound sources. Specifically, the input audio signal 14 may include one or more audio signals of each audio object. Additionally, the input audio signal 14 may include an angular / parametric representation of a virtual sound source (or sound field), such as an HOA representation of a sound space including the sound source (e.g., a virtual position located within the space). Specifically, the input audio signal may include HOA data (or signals).
[0037] The received metadata 15 may be associated with the received input audio signal 14 and may include information related to the signal. For example, in the case of multi-channel audio content, the metadata may include input audio channel layout information related to the content, whereby the layout information may indicate the speaker positions of the audio channels and may indicate the number (and type) of channels. For example, for 5.1 surround sound audio, the metadata may include surround sound speaker layout information indicating that the input audio signal includes (and / or indicates which channels of the input audio signal are) five full-range audio channels and one LFE channel and / or which channels will be located at specific surround sound positions (e.g., indicating that one channel is for the left surround speaker). In the case of an audio object, the metadata may include position (or spatialization) information (e.g., position and / or orientation) of one or more sound sources associated with the audio object relative to a reference point (e.g., a listener position). On the other hand, the metadata may include any information (e.g., necessary) for rendering the audio content of the input audio signal. As another example, the metadata may include other information, such as one or more gains, one or more equalization (EQ) operations, may indicate the number of channels (signals) associated with the audio content, etc.
[0038] In one aspect, the metadata 15 may be received as part of (or included in) at least one of the input audio signals 14. In another aspect, the metadata may be received separately from the input audio signal. For example, the input audio signal may be retrieved from a memory of the encoder-side device 11, while at least a portion of the metadata 15 may be retrieved from a remote electronic device (e.g., an electronic server).
[0039] The transform 20 may be configured to receive N input audio signals 14 and apply one or more operations to transform or convert the N signals into L signals. For example, the transform may apply one or more conversion matrices to one or more received signals 14 to convert the signals into L signals, which may have more, less, or the same number of signals as the received signals. Specifically, the transform 20 may apply a conversion matrix (e.g., which may be stored in a memory of an encoder-side device and / or may be generated by the device) to convert the received input audio signal from one audio format to another audio format, such as converting the input audio signal into an HOA representation of the input audio signal (having one or more HOA signals as HOA data). For example, in the case of an audio object, the transform 20 may apply the conversion matrix to one or more audio objects (e.g., an audio signal of an audio object) to generate a matrix (e.g., placing the audio object in a channel domain), and then convert the matrix into an HOA representation of the audio object. As another example, when the input audio signal includes several surround sound audio channels including one or more sound sources, the transform 20 may convert at least some of the surround sound channels into an HOA representation. Specifically, the transform 20 may apply a conversion matrix to the full-range audio channels of the surround sound channels to convert the channels into the HOA format while encoding and transmitting the LFE channel separately. More information about the conversion matrix is described herein.
[0040] The audio encoder 21 can be configured to receive (e.g., transformed) input audio signals and generate encoded audio signals having L audio signals (or channels) by performing one or more encoding operations. Specifically, the audio encoder can be configured to encode audio data (e.g., having one or more audio signals) into a coded code stream 13 for transmission to one or more decoder side devices 12. Specifically, the audio data can be encoded into a code stream, which can be transmitted (e.g., via a computer network) to one or more devices. On the other hand, the code stream can be transmitted wirelessly. For example, the encoder side device can be configured to establish a wireless connection with the decoder side device via a wireless communication protocol (e.g., Bluetooth protocol or any other wireless communication protocol). During the established wireless connection, the device 11 can exchange (e.g., transmit and receive) data packets (e.g., Internet Protocol (IP) packets) with the device 12, which may include encoded audio digital data (e.g., any audio format).
[0041] In one aspect, the coded data can be transmitted in "real time", which means that when the data is encoded, it can be transmitted later (e.g., with minimal time). In addition, when the audio content is streamed to the decoder side device for playback, the audio content can be encoded and transmitted. When the audio content is live broadcast, this may be the case. On the other hand, the coded data can be stored in the memory of the encoder side device and transmitted as a code stream 13 at a later time.
[0042] In one aspect, the audio encoder 21 may encode the input audio signal 14 using an algorithm based on matching pursuit (MP) coding. MP is a sparse approximation algorithm that computes a (e.g., nonlinear) approximation of a signal by constructing a series of sparse approximations or significant components to the signal. In one aspect, the algorithm may be principal component analysis, PCA, or any linear transform in which one or more spatial descriptors ("SD") are generated from the input signal. Specifically, the input signal may be an input ("H") matrix represented by an HOA, where the SD describes spatial aspects, significant audio components, such as the direction of arrival and diffusion associated with the H matrix. PCA or a linear transform may be performed directly on a zero mean covariance matrix. In one aspect, the zero mean covariance matrix may be calculated for the result of subtracting a column-wise mean vector from the input H matrix. In one aspect, the column-wise mean vector minus the H matrix may be referred to herein as an H~ matrix. The SD and H' matrices may then be used to perform a significant component (SC) extraction process (e.g., whereby PCA may be applied to the H' matrix), which produces "P" significant audio components, which may then be quantized and transmitted as a (e.g., encoded bitstream) bitstream to one or more decoder-side devices. In one aspect, the column-wise mean vectors may be transmitted (via the bitstream) to the decoder side to generate (reconstruct) the recovered (synthesized) H' matrix at the decoder side through a decoding process (e.g., by adding the mean vector to the product of the recovered P significant audio components and the recovered spatial descriptor).
[0043] On the other hand, the algorithm can be modified so that the column-by-column mean vector does not need to be transmitted to the decoder side device, which can advantageously reduce the required bandwidth. Specifically, when extracting significant components, the significant component extraction can be modified at the encoder side device to directly use the input audio signal (e.g., H matrix) instead of using the H~ matrix with the mean subtracted column by column. Using this method, the synthesis (performed on the decoding side) calculates an accurate H' matrix, although the column-by-column mean vector cannot be accessed. On the other hand, the audio encoder can use any type of PCA encoding algorithm to encode and (decode) audio data. On the other hand, the encoder side device can use any type of audio codec to encode (and decode) audio data for transmission.
[0044] The metadata encoder 22 may be configured to receive metadata 15, which may be extracted from the input audio signal 14 or may be received separately from the signal 14, and may be configured to encode the metadata 15 into the (audio) code stream 13. For example, the encoder may encode the metadata into any type of encoding format, such as extensible markup language (XML). As described herein, the encoder-side device 11 may encode one or more input audio signals 14 and metadata 15 into the (same) code stream 13. In another aspect, the device 11 may encode in separate code streams, for example, encoding the encoded audio signal into a code stream 13 that may be transmitted to the decoder-side device 12, and encoding the encoded metadata into a different code stream for transmission to the device 12. In another aspect, the encoded metadata may be transmitted at a different time than the transmission of the encoded audio content.
[0045] As described above, one or more input audio signals 14 may include audio content in a specific format, such as a surround sound format, an audio object format, or an HOA format. On the other hand, the input audio signal 14 may include one or more audio formats. For example, the encoder side device 11 may include different types of audio content in different formats, such as a movie soundtrack in a surround sound 5.1 format, a VR environment audio content in an HOA format, and a musical work in an audio format with one or more audio objects. As another example, the input audio signal may include a segment of audio content (at least in part) in different audio formats. For example, the input audio signal 14 may include: several surround sound audio channels of a segment of audio content (e.g., a movie soundtrack); an HOA representation of a sound field, the HOA representation of the sound field including at least one sound source of the segment of audio content as several HOA signals; and one or more audio objects, the one or more audio objects including at least one audio signal containing a sound including a sound source.
[0046] The encoder side device 11 can be configured to generate one or more mixed audio signals, which include all (or at least some) audio data in the audio data of the input audio signal 14. Specifically, the mixed signal may include surround sound audio channels, HOA signals, and audio signals of audio objects. The audio encoder 21 can encode the mixed signal into a bitstream 13 using an algorithm based on MP encoding as described herein for transmission to the decoder side device 12. In addition, the encoded and transmitted metadata 15 may include information about the mixed audio content, such as speaker layout information with a surround sound format, position information of audio objects, and any other information related to the mixed audio data. Therefore, the encoder side device is able to efficiently encode and transmit multiple types of audio content for playback by another (or one or more) playback device.
[0047] Turning to the decoder side device 12, the device 12 may be configured to receive the codestream 13 and render audio data from the codestream to one or more speakers 17. Specifically, based on the received codestream 13, the N channels of the input audio signal 14 and metadata 15 may be decoded, wherein an optional inverse transform 24 may be applied to the L decoded channels to convert the channels into N channels that may be provided to the renderer 16. In one aspect, the decoded data may be provided to the renderer, which then renders the audio data based on the metadata and / or (optional) output channel layout 18, which may indicate the layout of the speakers 17, e.g., whether the speakers are in a surround sound 7.1.4 layout or a headphone layout, as described herein.
[0048] In one aspect, the decoder side device 12 can receive the bitstream 13 in real time, which means that the decoder side device 12 can receive the bitstream 13 when it is received (retrieved), encoded and / or transmitted by the encoder side device 11, and takes into account the minimum amount of time generated by the transmission of data over one or more networks. Specifically, the decoder side device 12 can receive the bitstream, which may include an encoded representation of the input audio signal 14 and encoded metadata 15 that may be associated with the input audio signal.
[0049] The audio decoder 23 may be configured to generate a decoded representation of the input audio signal by decoding the encoded representation using an algorithm based on MP coding, which has been used by the audio encoder 21 to encode the representation. Specifically, the encoded representation may include significant audio components and spatial descriptors generated by the audio encoder 21, wherein the audio decoder may use these features to synthesize the restored (or reconstructed) input audio signal 14. For example, the decoded representation may be an HOA representation of a sound field associated with (or including) the input audio signal 14. Specifically, the audio decoder 23 may generate H' as a reconstruction of the input H matrix encoded by the audio encoder 21. In addition, the decoder may generate the received matrix using a column-by-column mean vector (e.g., by adding the mean vector to the product of the restored significant audio component and the received spatial descriptor). The metadata decoder 25 may be configured to receive encoded metadata (e.g., from the bitstream 13 or from a different bitstream) and generate (or reconstruct) metadata 15 by decoding the encoded data.
[0050] The inverse transform 24 may be an optional block that may be applied to the decoded audio data if the encoder-side device has applied the optional transform 20. Specifically, the transform 24 may apply a conversion matrix to a decoded representation from the audio decoder 23 to reconstruct the input audio signal 14 for rendering by the renderer 16. For example, the transform 20 of the encoder-side device may have applied a conversion matrix to transform an audio object of the input audio signal 14 into an HOA format. In this case, the inverse transform 24 may transform the decoded representation, which may be in an HOA format, back into an audio object by applying the inverse matrix of the matrix originally applied by the transform 20. In one aspect, where the bitstream 13 includes an audio object, the metadata reconstructed by the decoder-side device may include position information associated with the audio object, which may allow the renderer 16 to spatially render the audio signal of the audio object.
[0051] As another example, when the decoded representation is an HOA format constructed by the transform 20 from several full-range audio channels of a surround sound format, the inverse transform 24 can reconstruct the original full-range audio channels. As described herein, the LFE channel may not be encoded with the full-range channels. In this case, the decoder-side device 12 can receive at least one LFE channel associated with the surround sound format, wherein the decoder-side device (e.g., the renderer 16) can be configured to receive the full-range audio channels and the LFE channels, and can be configured to generate an audio drive signal by assigning each of the channels to a specific speaker 17 based on metadata 15, which metadata may include an input speaker layout associated with the surround sound format.
[0052] As described herein, the transform 20 and the inverse transform 24 may be optional. In this case, the audio encoder 21 may receive N input audio signals 14, generate an encoded bitstream 13 having the signals 14, for transmission by the encoder-side device 11 to the decoder-side device 12. The audio decoder 23 may decode the data to generate a reconstruction of the N signals 14 for the renderer. In one aspect, this may be the case when the input audio signal includes one or more HOA signals in an HOA format (such as an A format, a B format, etc.). In this case, the audio encoder 21 may encode the N HOA input audio signals 14 into a bitstream 13, wherein the audio decoder may decode the encoded bitstream to reconstruct the N HOA input audio signals 14, and then the renderer 16 may render the HOA input audio signals according to the layout 18.
[0053] The renderer 16 may be configured to receive the reconstructed N input audio signals 14 and render these signals as one or more drive signals for driving one or more speakers 17. Specifically, the renderer may be configured to generate the drive signals by rendering a decoded representation of the input audio signals (e.g., reconstruction of the signals) based on the decoded metadata 15 and / or based on an (optional) output channel layout 18 indicating the layout of the speakers 17. Specifically, the layout may include a table (data structure) indicating the speaker positions (e.g., relative to the listener position) of the speakers 17 in the listening environment. On the other hand, the speaker positions may indicate the positions of the multi-channel layout. For example, the layout 18 may indicate which speaker is the left front speaker of the surround sound layout and which speaker is the right front speaker. In one aspect, the layout 18 may be received from a local memory or may be retrieved from a remote storage device.
[0054] In some aspects, the renderer 16 may perform spatial audio rendering operations to render one or more audio signals spatially. For example, the renderer may apply a spatial filter, such as a head-related transfer function (HRTF), which may be personalized for a user of the system 10 to account for the user's anthropometric measurements. In another aspect, the spatial filter may be a default (or generic) filter. Thus, the renderer is configured to generate spatial audio signals (e.g., binaural audio signals) that, when output through the speakers 17 (e.g., speakers of a headset), produce 3D sound (e.g., giving the user a sense that sound is emanating from a particular location within an acoustic space).
[0055] As described herein, the encoder side device 11 can generate an encoded mixed audio signal, which may include a mixture of one or more audio formats. For example, a code stream may include an encoded mixed signal, which includes several surround sound signals in a surround sound format, one or more audio objects including one or more audio signals, and HOA data including several HOA signals, wherein each different format may include similar or different audio content (e.g., a surround sound signal with a movie soundtrack, and the HOA data may include a musical work). In this case, the decoder side device may be configured to separate these corresponding signals, wherein the renderer 16 may render the separated signals spatially according to the metadata 15 and / or the output channel layout 18 to generate a drive signal. Therefore, the decoder side device can play back several different types of audio content through a speaker. On the other hand, the decoder side device 12 may store some audio formats in a memory (for later playback), and may play back only one of the audio formats, such as a surround sound format, through a speaker 17.
[0056] In one aspect, the system 10 can be configured to perform the audio encoding operations described herein in real time, such that the encoder-side device 11 can perform audio processing operations on the input audio signal 14 and metadata to encode the data and transmit the data via the codestream 13 to the decoder-side device 12, which can then decode the information for rendering through the speaker 17, wherein these operations can be performed in a minimal amount of time (e.g., taking into account the processing time for the operations performed by the two devices and the transmission time for transmitting the data in the codestream 13).
[0057] Figures 2 to 5 Includes several aspects of encoding and decoding multi-channel audio. Specifically, Figure 2 A block diagram of a system 10 for encoding and decoding multi-channel audio according to one aspect is shown, the system including converting surround sound audio channels into a high-order stereo high-fidelity reverberation (HOA) format. Specifically, the figure shows multi-channel audio encoding in the HOA domain, whereby one or more channels of the multi-channel audio are encoded in the HOA domain for transmission to a decoder side device 12. The encoder side device 11 may have several operating blocks, such as a separator 32, an LFE encoder 35, an audio encoder 21, a channel converter 34, a speaker position extractor 33, and a metadata encoder 22. As shown, the encoder side device 11 may receive audio content in a surround sound format, which may include metadata based on the format. Specifically, the audio content may include multi-channel audio 31 and metadata 15 associated with the audio 31. The encoder device 11 receives audio 31 as one or more input audio signals, wherein the audio 31 may include one or more, K surround sound channels. For example, when the audio 31 is in a 7.1.4 format, the audio 31 may include eleven channels, or when the audio 31 is in a 5.1 format, the audio may include six channels. In this case, the audio 31 may include one or more full-range channels (e.g., channels to be played back by surround sound speakers of a surround sound system), and / or may include one or more LFE or limited bandwidth channels that may be arranged to be played back by loudspeakers such as a subwoofer. Turning to the metadata 15, the information may indicate a channel layout for the multi-channel content, such as indicating whether the audio content is 5.1 surround sound audio or 7.1.4 surround sound audio. On the other hand, the layout may indicate which channel is associated with a particular speaker, such as indicating whether the channel is associated with a front center speaker.
[0058] The separator 32 may be configured to receive the multi-channel audio 31 and may be configured to separate (or separate) different types of channels from each other. Specifically, the separator may receive metadata 15, which may indicate which channels are full-range and which channels are limited bandwidth. Specifically, using the metadata 15, the separator 32 may be configured to separate and output the K channels of the audio 31 as Q LFE (or limited bandwidth) channels 51 and R non-LFE (or full-range) channels 52. For example, when the multi-channel audio 31 is 5.1 surround sound audio, Q is one channel and R is five channels.
[0059] The speaker position extractor 33 can be configured to receive the metadata 15 and extract the position of the speaker 53 associated with each of the channels of the audio 31 (e.g., channel layout). In one aspect, the speaker position can be the position of the speaker relative to the listener's position. In one aspect, the extracted position may include an azimuth and / or an elevation. On the other hand, the extracted position may include a distance from the listener's position to the speaker position. In one aspect, the extractor can determine the information based on the metadata. Specifically, the extractor 33 can determine the surround sound format of the audio 31, and the speaker position can be derived based on the format. For example, when it is determined that the audio 31 is in a 5.1 surround sound format, the extractor can perform a table lookup on a data structure indicating one or more speaker positions for each full-range channel based on the format of the audio. On the other hand, the metadata may include the speaker position. In this case, the extractor can retrieve the information from the metadata.
[0060] The channel converter 34 may be configured to receive one or more non-LFE channels 52 and convert the channels into HOA representations 58 based on the metadata 15 (e.g., surround sound speaker layout information). Specifically, the converter may receive the speaker positions 53 and may generate a channel (CH) to HOA conversion spherical harmonic function (SPH) matrix based on the speaker positions 53 of the non-LFE channels 52 or, more specifically, directions associated with the non-LFE channels. Specifically, the SPH matrix, when applied to one or more non-LFE channels 52, produces a corresponding HOA signal as a spherical harmonic function signal (or function) of the HOA 58. Thus, the HOA 58 may be the result of the converter 34 applying the conversion matrix to the channel 52.
[0061] As described herein, the audio encoder 21 may be configured to receive the HOA 58 and encode the HOA representation into the bitstream 13 using an algorithm based on MP encoding together with the metadata encoded by the metadata encoder 22. The LFE encoder 35 may be configured to receive one or more, Q LFE channels 51 extracted from the audio 31 and may encode the channels into the bitstream 13 using any codec. In some aspects, the encoder may use any suitable audio codec, such as Advanced Audio Coding (AAC), MPEG Audio Layer II, MPEG Audio Layer III, or Free Lossless Audio Codec (FLAC). Thus, the LFE channel 51 may be encoded into the bitstream 13 separately from the HOA representation 58 representing the non-LFE channel 52. In one aspect, the encoded HOA representation, LFE channel, and metadata may be transmitted to the decoder side device 12 as the bitstream 13. In another aspect, each may be transmitted in a separate bitstream and / or may be transmitted at different times from each other.
[0062] Thus, as described above, at the encoder side device, based on the metadata of the multi-channel audio, or more specifically based on the channel layout, the LFE channel 51 can be encoded by the LFE encoding tool (e.g., where the encoder 35 uses an encoding algorithm to encode a channel of limited bandwidth). Based on the channel layout, the non-LFE channel 52 can be converted into a converted audio, such as HOA 58, by multiplying the CH to HOA conversion matrix generated by the SPH for the non-LFE speaker position 53. The converted audio can be encoded by the audio encoder 21 (e.g., using an algorithm based on MP encoding). The metadata 15 can be encoded by the metadata encoder 22, and each encoded element can be transmitted to the decoder side device 12 via the code stream 13.
[0063] The decoder side device 12 includes an LFE decoder 36, an audio decoder 23, a metadata decoder 25, a speaker position extractor 33, a channel converter 37 and a merger 39. The decoder side device receives an encoded code stream 13, which includes an encoded HOA signal, one or more encoded LFE channels and encoded associated metadata. The metadata decoder 25 decodes the encoded metadata and generates metadata 15 (e.g., reconstruction of the metadata). As described herein, the audio decoder 23 generates a decoded HOA signal (e.g., HOA 58) using an algorithm based on MP encoding. In addition, the LFE decoder 36 is configured to generate (or reconstruct) an LFE channel 51 by decoding the encoded LFE channel within the code stream 13.
[0064] As described herein, the speaker position extractor 33 receives the metadata 15 and generates speaker positions 53 for the non-LFE channels 52. The channel converter 37 receives the positions (layout) 53 and generates an HOA to channel conversion SPH matrix based on the positions 53. In one aspect, the HOA to CH conversion matrix may be a pseudo-inverse of a CH to HOA conversion matrix generated by the channel converter 34 at the encoder-side device 11 to convert the non-LFE channels 52 to HOA 58. The converter 37 receives the HOA signal 58 from the decoder 23 and converts (e.g., converts back) the HOA signal to one or more non-LFE channels 52 using the HOA to CH conversion matrix generated based on the metadata 15.
[0065] The merger 39 may be configured to receive one or more reconstructed LFE channels 51 and one or more reconstructed non-LFE channels 52, and may be configured to merge the channels into several surround sound channels in a surround sound format based on the metadata 15. Specifically, the merger reconstructs the multi-channel audio 31, which includes K channels and may be in an original surround sound format, such as a 5.1 surround sound format. The reconstructed audio 31 may then be sent to the renderer 16, which may then use the channels to drive the speakers 17 according to its format. Specifically, based on the layout of the speakers 17, the renderer may spatially render the multi-channel audio 31 to the speakers 17. Thus, the figure provides an encoding operation that allows multi-channel audio to be encoded and transmitted to a decoder-side device, which may then reconstruct the original multi-channel audio for playback.
[0066] Therefore, as described herein, based on the received bitstream 13, the decoder-side device 12 can decode the LFE channel, the HOA signal 58, and the metadata 15. The decoder-side device can reconstruct the non-LFE channel 52 by multiplying the HOA signal 58 by the HOA to CH conversion matrix, which can be generated by the pseudo-inverse matrix of the SPH of the speaker position 53, wherein the matrix is the pseudo-inverse matrix relative to the encoder-side matrix, which the encoder side uses to initially convert the non-LFE channel into the HOA signal 58. Based on the metadata 15, or more specifically the channel layout of the multi-channel audio 31, the decoded LFE channel 51 and the non-LFE channel 52 can be placed in the correct channel position (e.g., within the surround sound format).
[0067] Figure 3 A block diagram of a system 10 for encoding and decoding multi-channel audio 31 using a conversion matrix codebook for converting the audio into a HOA format is shown in accordance with one aspect. Figure 2, except that the encoder side device 11 in this figure includes a conversion matrix codebook 46 and the decoder side device 12 includes a conversion matrix codebook 47. As described herein, the encoder side device 11 includes audio content (e.g., audio 31) in a surround sound format, such as 5.1 surround sound including metadata 15 based on the format (e.g., indicating a speaker layout in the 5.1 format), wherein the content includes a number (K) of surround sound audio channels. The separator 32 separates the K channels into a number of full-range or non-LFE channels 52 and at least one LFE channel 51 based on the metadata 15. In one aspect, a "full-range" audio channel can be an audio channel that includes spectral content along a wider frequency range than the LFE channel 51. On the other hand, when used to drive a full-range driver speaker, the full-range channel can be configured to provide audio content along the entire audible frequency range.
[0068] The conversion matrix codebook 46 may be a data structure (e.g., a table) including CH to HOA conversion matrices, which may be previously generated (e.g., in a controlled environment (e.g., a laboratory)). In one aspect, the codebook may include a SPH matrix based on the speaker position, which, when applied to multi-channel audio content, may convert the content into a stereo high-fidelity reverberation of a certain order in the HOA format, as described herein. In one aspect, the codebook 46 may associate different matrices with different speaker positions. Thus, the encoder-side device may be configured to determine and retrieve the conversion matrix associated with the speaker position 53 (e.g., a surround sound speaker layout) from the codebook 46, for example, by performing a table lookup using the position 53 extracted by the extractor 33. The channel converter 34 receives the channels 52 and converts them into HOA representations 58 using the retrieved conversion matrix (e.g., by applying the matrix to the channels 52), and encodes the HOA representation into the bitstream 13 using an algorithm based on MP encoding for transmission to the decoder-side device 12.
[0069] In one aspect, based on the channel layout indicated by the metadata 15, the LFE encoder 35 encodes one or more LFE channels 51 (separately) into the bitstream 13. Additionally, based on the channel layout, the non-LFE channels 52 are converted to HOA 58 by multiplying by a CH to HOA conversion matrix stored in the codebook 46, wherein the HOA 58 is encoded into the bitstream 13 by the audio encoder 21 along with the metadata.
[0070] Turning to the decoder side device 12, the device 12 receives one or more HOA signals, one or more LFE channels, and metadata as an encoded code stream 13, and extracts data from the code stream 13, wherein the HOA signal is extracted by decoding the HOA signal using an algorithm based on MP coding. Similar to the encoder side device 11, the decoder device 12 can use the code book 47 to retrieve the conversion matrix to convert the decoded HOA 58 back to (e.g., reconstruct) a non-LFE channel. Specifically, the code book 47 can be a data structure including HOA to CH conversion matrices, wherein each matrix can be associated with (e.g., a different) speaker position. Therefore, the code book 47 may include an inverse matrix of the matrix of the code book 46 of the encoder side device 11. In one aspect, both code books may include two sets of matrices (e.g., CH to HOA matrices and corresponding HOA to CH matrices) so that either device can perform encoding operations and decoding operations. Therefore, the decoder side device 12 retrieves the HOA to CH conversion matrix from the code book 47 stored in the memory of the device 12 based on the metadata 15. Specifically, the device 12 may perform a table lookup in the codebook 47 using the speaker positions 53 extracted from the metadata. The channel converter 37 of the device 12 generates (reconstructs) the non-LFE channels 52 by applying the conversion matrix from the codebook 47 to the HOA 58 extracted from the bitstream 13. The merger 39 then generates the multi-channel audio 31 by merging the one or more extracted LFE channels 51 and the one or more non-LFE channels 52.
[0071] Therefore, based on the received code stream 13, one or more LFE channels 51, HOA 58 and metadata 15 are decoded. The non-LFE channel 52 is reconstructed by multiplying the HOA 58 by the HOA to CH conversion matrix stored in the code book 47, and based on the metadata, the one or more decoded LFE channels and the one or more non-LFE channels are placed in the correct channel positions.
[0072] Figure 4 A block diagram of a system 10 for encoding and decoding multi-channel audio using a transformation matrix based on an identity matrix based on the number of surround sound audio channels of the channel audio is shown in accordance with one aspect. As shown in the figure, the figure is similar to Figure 2, except that in this figure the encoder-side device 11 includes a non-LFE number calculator 56 (which can be configured to perform at least some of the operations of the extractor 33) and a generator 48 based on the identity matrix, and the decoder-side device 12 includes a non-LFE number calculator 56, a generator 48 based on the identity matrix, and a transposed matrix 38. As described herein, the encoder-side device 11 receives audio content in a surround sound format (e.g., a 5.1 surround sound format) including metadata 15 and a number of multi-channel audio channels 31, wherein the encoder device separates the LFE channel from other (e.g., full-range) audio channels based on the metadata 15. The non-LFE number calculator 56 can be configured to receive the metadata and determine the number 57 of non-LFE channels received with the audio 31. For example, the calculator can determine the number based on the channel layout of the surround sound format. For example, for a 5.1 surround sound format, the calculator can determine that there are five non-LFE audio channels.
[0073] The identity matrix-based generator 48 may be configured to generate a transformation matrix based on the number of LFE channels. Specifically, the generator 48 receives the number 57 of non-LFE channels included in the multi-channel audio 31 and determines an HOA order associated with an equal or greater number of HOA channels. For example, a 1st order stereo high-fidelity reverb has four audio channels, while a 2nd order stereo high-fidelity reverb has nine audio channels. In the event that the number of channels 57 indicates that the multi-channel audio 31 has five channels (e.g., a 5.1 surround sound format), the generator 48 selects a 2nd order stereo high-fidelity reverb with nine channels and generates an identity matrix based on the selection. Specifically, the generator 48 may be configured to generate an identity matrix having an order based on the number of non-LFE channels, in which case the identity matrix would be 5 The generator 48 generates a transformation matrix by adding additional columns to the identity matrix such that the total number of columns is equal to the number of stereo high-fidelity reverb channels. In one aspect, the additional columns will have zero values such that each of the elements is 0. An exemplary transformation matrix generated by the generator 48 for five non-LFE channels may be:
[0074]
[0075] The channel converter 34 may be configured to receive the non-LFE channels 52 and multiply the generated conversion matrix (e.g., having at least five diagonal elements of "1") by the non-LFE channels 52 to produce a matrix in which the non-LFE channels 52 are assigned to at least one column of the conversion matrix. For example, in the case of five non-LFE channels 52, each of the first five columns may be assigned a non-LFE channel. In addition, the converter 34 may assign four additional channels to the unassigned four columns of the conversion matrix, so that the resulting matrix assigns a total of nine channels, five non-LFE channels, and four additional channels. In one aspect, the additional channels do not include multi-channel audio content. As described herein, the resulting matrix may allow an audio encoder to perform an algorithm based on MP encoding to encode audio data. In one aspect, the non-LFE channels can be converted to HOA representations of the non-LFE channels (and additional channels) by a conversion matrix (e.g., an H matrix), in this case, a 2nd order stereo high-fidelity reverberation representation, where five of the HOA signals can be assigned to the non-LFE audio channels and four of the remaining HOA signals can be assigned to additional channels that do not include audio content, as described herein. In one aspect, the additional channels may not include any data. Specifically, the channel converter applies the conversion matrix to the channel 52 and produces converted audio 98 (e.g., which can be an HOA signal). As described herein, the converted audio 98 can be encoded into the bitstream 13 by the audio encoder using an algorithm based on MP encoding.
[0076] In one aspect, based on metadata 15 (e.g., the channel layout of the multi-channel audio 31), the LFE channel is encoded by an LFE encoder 35 (e.g., using an LFE encoding algorithm). Based on the metadata, the non-LFE channels can be converted into converted audio by multiplying a conversion matrix based on a unit matrix by the non-LFE channels. The number R of non-LFE channels and the number of converted audio channels (e.g., the number of L, N, and L-N converted audio channels) can be filled with non-LFE channels and zeros, respectively. The converted audio can be encoded by an algorithm based on MP encoding. In one aspect, the encoded LFE channel, the converted audio, and the metadata can be transmitted to a decoder-side device via one or more code streams 13.
[0077] The decoder-side device 12 receives an encoded bitstream 13, which may include an LFE channel 51 of a segment of surround sound audio content, metadata 15, and converted audio 98 (or audio content), the converted audio may include a number of channels generated by applying a conversion matrix based on an identity matrix to a non-LFE surround sound channel (e.g., channel 52) of the segment of surround sound audio content, the identity matrix being based on metadata, wherein the channels of the encoded audio content in the bitstream may include a non-LFE surround sound channel and at least one additional channel that does not include audio data. Specifically, as described herein, the number of channels in the encoded audio content in the bitstream may be equal to the number of channels of a stereo high-fidelity reverberation of a given order, the number of channels of the stereo high-fidelity reverberation of the given order being at least equal to the number of channels of the segment of surround sound audio content. In the case where the number of channels of the stereo high-fidelity reverberation of a given order is greater, additional channels may be added, wherein the additional channels may not include audio data of the surround sound content.
[0078] The decoder-side device 12 may be configured to use the generator 48 to determine a conversion matrix that is used by the channel converter 34 to generate the converted audio at the encoder-side device (e.g., a 5×9 matrix including a 5th-order identity matrix when the segment of surround sound audio content is in a 5.1 surround sound format). The transpose matrix 38 receives the conversion matrix and generates a transpose matrix of the identity matrix. The channel converter 37 receives the decoded converted audio 98 from the audio decoder 23 and reconstructs the non-LFE surround sound channels and at least one additional channel by applying the transpose matrix of the conversion matrix to the converted audio 98. The decoder-side device 12 may combine the LFE channel 51 and the non-LFE channel 52 to reconstruct the multi-channel audio 31 (which may include metadata 15) and may be used (sent to the renderer 16) to drive the speaker 17. In one aspect, the channel converter 37 may truncate (or discard) the additional channel.
[0079] In one aspect, one or more LFE channels, converted audio (which may include surround sound audio and additional content), and metadata are decoded based on the received bitstream 13. Based on the decoded metadata 15, the number of non-LFE channels is calculated 57. The non-LFE channels 52 are reconstructed by multiplying the decoded converted audio 98 by the transposed matrix of the transformation matrix based on the identity matrix (and by removing the last additional channel of the converted audio that does not include audio data for the multi-channel audio). Based on the metadata, the renderer 16 can place the decoded LFE channels and the reconstructed non-LFE channels to the correct channel speaker positions.
[0080] Figure 5A block diagram of a system 10 for encoding and decoding multi-channel audio according to one aspect is shown. Specifically, the figure shows non-LFE channels 52 of multi-channel audio 31. Specifically, the audio encoder 21 can receive one or more non-LFE channels 52 and directly encode the channels using an algorithm based on MP encoding. In this case, the encoder-side device 11, or more specifically, the LFE encoder 35 and the metadata encoder 22 can encode one or more LFE channels 51 and metadata 15 together with the encoded non-LFE channels 52 into the bitstream 13. In one aspect, the encoder-side device 11 can apply the non-LFE channels 52 to an (optional) identity matrix 59 (the identity matrix, for example, has an order based on the number of non-LFE channels) so that the resulting matrix can be encoded by the audio encoder, as described herein. Therefore, based on the metadata, the LFE channel can be encoded by the LFE encoder, the non-LFE channel can be encoded by an algorithm based on MP encoding, and the metadata can be encoded by the encoder 22, where the encoded data is transmitted via the bitstream 13.
[0081] The decoder side device 12 may be configured to receive an encoded bitstream including a non-LFE (surround sound) channel, one or more LFE channels, and / or metadata 15, wherein the content may be extracted (e.g., by decoders 36, 23, and / or 25) by executing one or more decoding algorithms, including algorithms using MP-based encoding. Based on the metadata 15, the decoded channels are combined to produce (original) multi-channel audio 31 (in surround sound format). Thus, as described herein, the LFE channel and the non-LFE channel may be placed in the correct channel position.
[0082] As described herein, the system 10 may separate the LFE channel from the non-LFE channel and encode both into the bitstream 13. In one aspect, the LFE channel may be an optional channel. Specifically, the multi-channel audio 31 may not include the LFE channel. In one aspect, the multi-channel audio may be several (e.g., full-range) channels of audio content (e.g., may or may not be a surround sound format). In this case, the multi-channel audio may be passed to the channel converter 34 for conversion and encoding. In this case, the bitstream 13 may include HOA representations and metadata of the non-LFE channels without encoding the LFE channels. Therefore, the decoder-side device 12 may convert the HOA representation back to non-LFE channels and use these channels (without the LFE channel) to render the audio content.
[0083] As described above, system 10 can be configured to convert multi-channel audio into different audio formats, and transmit the encoded version of the multi-channel audio to a decoder side device, which can decode and reconstruct the original multi-channel audio for rendering. This has several advantages. For example, along with the transmitted audio, associated metadata can also be transmitted, both of which allow audio processing operations to be performed on the original audio (e.g., the audio received by the encoder side device 11). This provides flexibility on the decoder side, wherein audio signal processing operations can be performed on the decoder side device. This is in contrast to receiving the audio that has been processed by the encoder side device, thereby not allowing the decoder side device to undo some of the processing that has been performed. In addition, by using an algorithm based on MP encoding to encode audio data, the encoded audio data can be effectively transmitted by the code stream 13 even when the bit rate is reduced, while maintaining high audio quality.
[0084] Figures 6 to 10 Involves performing encoding operations on one or more audio objects, where an audio object may be associated with one or more pieces of audio content.
[0085] Figure 6 A block diagram of a system 10 for encoding and decoding audio objects using a conversion matrix codebook for assigning audio objects to speaker positions for conversion into an HOA format according to one aspect is shown. The encoder side device 11 may include several operating blocks, such as an object to HOA converter 61, an audio encoder 21, an object to HOA conversion matrix codebook 62 based on speaker positions, an object quantity calculator 63, and a metadata encoder 22. The encoder side device 11 may be configured to receive a set of one or more audio objects 60, wherein each object may include one or more audio signals, the one or more audio signals including the sound of a sound source and metadata 15 (e.g., position data) that may indicate the position of the corresponding sound source (e.g., the audio object of the sound source) relative to a reference point (e.g., a listener position) in an acoustic environment. The object quantity calculator 63 may be configured to receive the metadata 15 and determine the number 64 of the audio objects of the object 60 based on the associated metadata. In one aspect, when each of the audio objects includes one audio signal, the number 64 may be the received audio signal. In one aspect, when one or more objects include one or more audio signals, the number 64 may be more than the number 60 of audio objects. In one aspect, the object number calculator 63 may determine the number of objects 64 based on the position data (eg, based on the number of azimuth and elevation pairs).
[0086] The object-to-HOA conversion matrix codebook 62 based on speaker positions can be a data structure storing an object-to-HOA conversion matrix that, when applied to one or more audio objects, can convert the object into a stereo high-fidelity reverberation of a certain order, as described herein. In one aspect, the codebook can include an SPH matrix that, when applied to multi-channel audio, can convert the audio into a stereo high-fidelity reverberation based on the speaker positions of the multi-channel audio, as described herein. Thus, the codebook can include different conversion (SPH) matrices based on the number of speaker channels (or more specifically, the speaker positions associated with the channels, such as for different surround sound formats, such as 5.1, 7.1, 7.1.4, etc.). In one aspect, the codebook can specify the speaker positions associated with the conversion matrix based on MPEG Coding Independent Code Point (CICP) speaker layout index values for different speaker layouts.
[0087] The encoder side device 11 may be configured to determine (or select) a conversion matrix from the codebook 62 based on the number of audio objects 64, so that 1) each audio signal of the audio object 60 may be mapped to a surround sound speaker position of the speaker position of the selected matrix, wherein the number of speaker positions of the conversion matrix may be greater than or equal to the number of audio signals of the audio object. For example, if there are five audio object signals, the encoder side device may select a conversion matrix having speaker positions associated with non-LFE audio channels of a 5.1 surround sound audio format, wherein each of the five surround sound positions may be associated with one of the five audio object signals.
[0088] However, if the number of audio objects is different from the number of speaker positions of the conversion matrix, the device can select a conversion matrix in which the number of speaker positions can be greater than the number of audio object speakers, but the number of redundant speaker positions can be minimized. For example, when six audio object signals are received, the encoder side device 11 can use the number of audio objects 64 to select a conversion matrix with speaker positions for a 7.1 surround sound format, which is different from the 7.1.4 surround sound format because the 7.1 surround sound format has seven non-LFE speaker positions and 7.1.5 has eleven non-LFE speaker positions. Specifically, in order to make this selection, the encoder side device can use the number of audio objects 64 to perform a table lookup in the codebook 62 to select a conversion matrix, as described herein.
[0089] The object to HOA converter 61 receives the audio signal X of the audio object 60 and generates an HOA representation 58 of the object (having Y HOA signals 58) by applying the transformation matrix from the codebook 62 to the object 60. Specifically, the matrix maps the object 60 to the speaker position (e.g., thereby placing the object in the multi-channel domain), and then converts the object to the HOA domain according to the spherical harmonic function of the matrix. Therefore, the transformation matrix is used to 1) map each audio signal of the audio object to the surround sound speaker position, and 2) convert the object into the HOA representation according to the speaker position. The audio encoder 21 receives the HOA representation 58 and encodes it into the codestream 13 using an algorithm based on MP encoding, which may include associated encoding metadata (e.g., by the encoder 22).
[0090] In one aspect, based on a set of different non-LFE speaker positions, an object-to-HOA conversion matrix using SPH can be generated and stored in a codebook 62, where the speaker positions can be specified with CICP indices. Therefore, the codebook 62 may include different matrices for different speaker positions (layouts). Based on the number of audio objects 64X, an object-to-HOA conversion matrix with one or more non-LFE speaker positions can be selected from the codebook 62. For example, the matrix can be selected so that the number of non-LFE speaker positions is already greater than or equal to the number of objects 64, while also minimizing the difference between the number of non-LFE positions and the number of objects (e.g., in order to reduce the required bandwidth). By multiplying the selected conversion matrix by the audio object signal 60, the object audio signal can be converted into a stereo high-fidelity reverberation (e.g., HOA 58). The HOA 58 can be encoded along with the metadata 15 and transmitted to the decoder side device via the bitstream 13.
[0091] The decoder side device 12 may include an audio decoder 23, an object extractor 66, a renderer 16, a HOA to object conversion matrix codebook based on speaker position 65, an object quantity calculator 63, a metadata decoder 25, and an output channel layout 18. As described herein, the decoder side device 11 receives an encoded bitstream 13 from an encoder side device, which may include a converted audio (e.g., HOA) representation of audio content and associated metadata. The metadata decoder 25 generates decoded metadata 15 from the bitstream, and the object quantity calculator 63 is configured to determine the number 64 of audio objects represented by the HOA representation received within the bitstream 13.
[0092] The HOA to object conversion matrix codebook 65 based on speaker positions can be a data structure storing a conversion matrix for converting HOA data into one or more audio objects. Specifically, the codebook may include a pseudo-inverse SPH matrix of the matrix stored by the codebook 62 at the encoder side device 11. In one aspect, each of the codebooks 62 and 65 can store two types of matrices. Therefore, the codebook 65 includes different conversion SPH matrices based on the number of speaker channels (or more specifically, the speaker positions associated with the channels). The decoder side device 12 can select a conversion matrix from the codebook 65 based on the number of audio objects 64 and based on the aforementioned criteria for selecting a conversion matrix from the codebook 62. For example, the selected pseudo-inverse conversion matrix can be associated with a speaker layout including a number of speaker positions, the number of speaker positions being greater than or equal to the number of objects 64, where the difference between the numbers is minimized.
[0093] The object extractor 66 receives the HOA representation 58 and the selected transformation matrix from the codebook 65, and generates one or more audio objects 60 from the HOA 58 using the transformation matrix (e.g., multiplying it by the HOA representation 58). Thus, the transformation matrix can convert the HOA representation into several surround sound audio channels, and reconstruct the audio objects from the surround sound audio channels.
[0094] In one aspect, each audio object signal can be assigned to a specific speaker position. In this case, the audio object can be reconstructed by extracting the audio object's respective signal from the converted surround sound audio channels. In one aspect, any redundant surround sound audio channels (e.g., based on fewer audio objects than surround sound speaker positions and not including audio object data) can be discarded.
[0095] The renderer 16 receives the audio object 60, the metadata 15, and the output channel layout 18 indicating the layout of the speaker 17 (e.g., indicating whether the speaker is a headset or a layout of one or more loudspeakers), and can use the audio object and the metadata to spatially render the loudspeaker or headphone drive signal (e.g., based on the layout 18). In one aspect, the renderer can render the audio object in real time, meaning that the object can be rendered with a minimal amount of time (e.g., considering processing time) in between when it is received via the bitstream 13. In another aspect, the audio object can be stored in a memory of the decoder-side device 12 for later rendering.
[0096] In another aspect, at least some of the processes described herein may be performed in real time. For example, when the audio decoder 23, the object extractor 66 decodes the HOA 58, the pseudo-inverse conversion process may be performed in real time. In this case, the extractor 66 may receive the number of audio objects 64 and extract the objects accordingly. In this case, the codebook 65 may be optional.
[0097] In one aspect, the pseudo-inverses of all code vectors in the object to HOA matrix codebook 62 can be stored in a speaker position based HOA to object transformation matrix codebook 65. Based on the received code stream, the HOA 58 and the metadata 15 are decoded. Based on the number of converted surround sound channels of the HOA 58 and the number of audio objects indicated by the metadata 15, a transformation matrix is selected from the codebook 65, where the number of surround sound speaker positions associated with the selected matrix is greater than or equal to the number of audio objects and is also minimized. The audio object 60 can be reconstructed from the HOA 58 by multiplying the HOA 58 with the selected transformation matrix. Based on the audio object signal and the metadata, the object rendering performed by the renderer 16 produces one or more output signals for the speaker 17.
[0098] Figure 7 A block diagram of a system 10 for encoding and decoding audio objects using a conversion matrix codebook according to another aspect is shown, the conversion matrix codebook being used to assign audio objects to speaker positions for conversion into an HOA format. Specifically, the figure relates to encoding audio objects by mapping objects to speaker positions, which are arranged as T-shaped design points along a sphere (or a projection grid). Specifically, an encoder-side device includes an object-to-HOA conversion matrix codebook 68 based on a T-shaped design, which is used to convert one or more audio objects into converted (e.g., HOA) data for encoding and transmission as a code stream 13.
[0099] In one aspect, if the integral of any polynomial of at most t degree on the spherical grid is equal to the average of the polynomial over a set of n points, then the set of n points can be referred to as a spherical T-design. In some aspects, these points can be associated with speaker positions, where these positions can be located along a spherical projection grid (or the surface of a sphere). In one aspect, the spherical projection grid can be a sphere around a reference point (e.g., a listening position). In one aspect, several transformation matrices of the codebook 68 can be generated, where each matrix can take into account a (e.g., different) number of speaker positions around the sphere. For example, several spherical projection grids can be generated, where each spherical projection grid can have a different number of speaker positions on the surface of the spherical projection grid according to a spherical T-design (e.g., for a given T-design index). For each spherical grid, a transformation matrix can be generated and stored in the codebook 68, where the matrix can convert one or more audio objects into HOA using spherical harmonics of the corresponding number of speaker positions. Specifically, the transformation matrix can be used to transform the audio objects using spherical harmonics of the positions of the T-design speaker positions mapped to the matrix.
[0100] In one aspect, the encoder side device 11 can use the codebook 68 to convert the audio object 60 into the HOA 58. Specifically, the encoder side device can select (retrieve) a transformation matrix from the codebook 68 based on the number of audio objects 64. In one aspect, the encoder side device can select a transformation matrix in which the number of associated speaker positions of the T-shaped design is greater than or equal to the number of audio objects 64, and the difference between the number of positions and the number of objects 64 is minimized (e.g., as close to zero as possible or zero). As described herein, using the selected transformation matrix, the object to HOA converter 61 can apply the transformation matrix to the audio object 60 to generate the HOA representation 58 of the audio object.
[0101] As described herein, the encoder side device 11 can generate transformation matrices for the codebook 68 (or they can be previously generated). In one aspect, for a given T-shaped design index (e.g., the number of points), the speaker positions can be specified (e.g., at the positions of the points of a specific T-shaped design size, such as 50 speaker positions corresponding to 50 points). For a given T-shaped design index, an object-to-HOA transformation matrix for the SPH using a set of speaker positions can be generated and stored in the codebook 68. In one aspect, several transformation matrices can be generated for different T-shaped design indices.
[0102] Based on the number of audio objects 64, a T-shaped design index can be selected, where the number of speaker positions of the index can be greater than or equal to the number of objects 64, and the difference between the two numbers can be minimal (e.g., zero or as close to zero as possible or zero). The audio object signal can be converted to transformed audio (e.g., HOA 58) by multiplying the selected object to HOA conversion matrix. As described herein, the HOA 58 and metadata 15 of the audio object 60 can be encoded and transmitted via the bitstream 13.
[0103] The decoder side device 12 includes an HOA to object conversion matrix codebook 69 based on a T-shaped design, which may include a conversion matrix configured to convert the converted audio (e.g., HOA 58) back to one or more audio objects. Specifically, the codebook 69 may include an inverse matrix or a transposed matrix in the codebook 68 of the encoder side device 11. In one aspect, each of the codebooks 68 and 69 may include one or two sets of conversion matrices. In this case, the decoder side device 12 receives a code stream having encoded HOA data of one or more audio objects and encoded audio object metadata. The decoder side device decodes the HOA data and selects (e.g., inverse) a conversion matrix from the conversion matrix codebook 69 based on the number of audio objects 64 according to the metadata. The object extractor 66 generates an audio object 60 reconstructed from the HOA 58 by applying the selected matrix to the decoded HOA data.
[0104] In one aspect, the conversion matrix can reconstruct the audio object by converting the HOA data into the number of (e.g., surround sound) speaker channels according to the speaker positions (T-shaped design index based on the number of audio objects). According to the speaker channels and based on the metadata, the extractor can reconstruct the audio object audio signal.
[0105] Figure 8 A block diagram of a system 10 for encoding and decoding an audio object using a conversion matrix codebook according to another aspect is shown. Figure 7 Similar features, except that the encoder and decoder do not have a T-design based transition matrix codebook 68 and 69, respectively, the figure includes a Fliege-Maier-Set transition matrix codebook. As described herein, the T-design can be an arrangement of points (or speaker locations) on the surface of a sphere. The Fliege-Maier-Set can be another uniform arrangement (or distribution) of speaker locations along a spherical grid. In one aspect, the speaker locations of the Fliege-Maier-Set can be equidistant around the spherical grid and around a reference point (e.g., the center of the sphere is the listening position).
[0106] In one aspect, the codebooks 70 and 71 of the figure can be Figure 7The codebook of is similarly generated. For example, for a given Fliege-Maier-Set index indicating multiple points, the speaker position can be specified. Specifically, a spherical projection grid can be generated, which has multiple speaker positions on the surface of the grid according to the Fliege-Maier set. For the index, a transformation matrix can be generated and stored in the codebook 70, which is used to convert one or more audio objects into HOA using spherical harmonics of the speaker positions of the spherical grid. Based on this, multiple transformation matrices can be generated for different indices (for example, considering different numbers of speaker positions). In one aspect, the codebook 71 of the decoder side device 12 (and / or the codebook 70 of the encoder side device 11) may include a transformation matrix and an inverse matrix of one or more matrices, so that the audio object of the converted audio received by the decoder device can be reconstructed at the decoder side device 12, as described herein.
[0107] As described above, the object number calculator 63 may be configured to determine the number 64 of audio objects from the metadata 15 associated with the received audio object. In another aspect, the calculator 63 may determine the number based on the audio object 60. For example, the calculator may detect how many audio signals are included in the object 60 and generate the number 64 of objects.
[0108] Fig. 9 A block diagram of a system 10 for encoding and decoding audio objects using a transformation matrix based on an identity matrix according to one aspect is shown, wherein the identity matrix may be based on the number of audio objects to be transmitted to a decoder-side device via an encoded bitstream. Specifically, the encoder-side device 11 includes a generator 48 based on an identity matrix, which may be configured to generate a transformation matrix based on the identity matrix, the identity matrix having an order based on the number of audio objects to be encoded and transmitted via a bitstream 13. Specifically, the generator 48 receives the number 64 of audio objects calculated by a calculator 63 based on metadata 15 associated with an audio object 60, and generates a transformation matrix based on the number. Specifically, the generator 48 determines an identity matrix having an order based on (e.g., equal to) the number 64 of objects.
[0109] In one aspect, the generator determines whether the order of the identity matrix (or the number of audio objects) is equal to or greater than the order of the stereo high-fidelity reverberation. Specifically, the generator can be configured to determine whether the number of audio object signals to be encoded can be equal to (or greater than / less than) the number of HOA channels of the order of the stereo high-fidelity reverberation. In one aspect, if the order of the identity matrix is equal to the number of channels associated with the order of the stereo high-fidelity reverberation, the generator can set the identity matrix to a conversion matrix. For example, if there are nine audio objects, the generator 48 can set the conversion matrix to a ninth-order identity matrix because a second-order stereo high-fidelity reverberation includes a ninth-order HOA signal.
[0110] However, if the number of objects 64 may be greater than (or less than) the number of channels of the order of the stereo high-fidelity reverberation, the generator may generate the transformation matrix to have a size based on the number of channels of the next order of stereo high-fidelity reverberation, which may have more channels than the number of objects. Returning to the previous example, if the number of objects is five objects, the generator may generate a fifth-order identity matrix (e.g., one main diagonal element of each audio object is "1"), and generate the transformation matrix based on the generated identity matrix. Specifically, the generated transformation matrix may have a size that takes into account a 2nd-order stereo high-fidelity reverberation. For example, the transformation matrix may include nine columns, each column for each channel of the 2nd order, whereby the transformation matrix may include a generated fifth-order identity matrix having five columns and four additional columns, such that the size of the transformation matrix is a 5×9 matrix.
[0111] The object to HOA converter 61 receives an audio object (signal) 60 having X signals and converts the signals into an HOA representation 58 based on a unit matrix, which is based on the associated metadata. For example, the order of the unit matrix may be equal to X. The HOA 58 includes a number of HOA signals Y, which include the audio object signal 60 and one or more additional audio signals, so that the total number of signals is equal to the order Y of the stereo high-fidelity reverberation of the HOA 58. For example, the number of additional signals may be equal to YX. The additional signal may not include audio data associated with the audio object 60. As described herein, the audio encoder 21 can be configured to encode the HOA 58 into a bitstream 13 using an algorithm based on MP encoding, which may include encoded metadata 15 encoded by the encoder 22.
[0112] In one aspect, the object audio signal can be converted to converted audio (e.g., HOA 58) by multiplying the audio signal by a conversion matrix based on the identity matrix. If the number of objects is X and the number of converted audio channels is Y, then the X and YX channels of the converted audio are filled with input channels based on the audio object signal and zeros, respectively. The converted audio can be encoded into the bitstream 13 along with the associated metadata using an algorithm based on MP encoding.
[0113] The decoder side 12 includes a generator 48 based on a unit matrix and is configured to generate a conversion matrix used by the encoder side device 11 to convert the audio object 60 into converted audio based on the number of audio objects 64. Specifically, the decoder side device receives a coded bitstream 13 including several HOA channels and metadata 15 associated with several audio objects. Based on the number of audio objects, the generator 48 determines a conversion matrix based on the unit matrix, which includes an order equal to the number of audio objects. The transpose matrix 38 transposes the determined conversion matrix, and the object extractor 66 can use the transpose matrix to convert the decoded HOA 58 (decoded by the audio decoder 23) into the audio object 60. Specifically, the extractor 66 converts the HOA channel Y of the HOA 58 into an audio object signal X, and into one or more additional signals, where the number of signals generated based on the conversion is equal to Y. In this case, when there are additional signals, the extractor can discard these signals and send the audio object signal to the renderer 16, where the renderer can receive the metadata 15 and the output channel layout 18 to generate a driving signal by rendering the audio signal based on the metadata (and layout). In one aspect, the decoder-side device may use the drive signal to drive a loudspeaker or a headphone, which may be coupled to the decoder-side device 12 .
[0114] In one aspect, based on the received bitstream, the HOA 58 and the metadata 15 are decoded. Based on the number of audio objects (and based on the number of HOA channels), a pseudo-inverse matrix of a conversion matrix based on the identity matrix used by the encoder side device 11 is calculated, and the pseudo-inverse matrix is used to reconstruct the audio object by multiplying the HOA 58 by the conversion pseudo-inverse matrix.
[0115] Fig.10A block diagram of a system 10 for encoding and decoding audio objects according to one aspect is shown. Specifically, the figure shows that received audio objects (e.g., audio signals) 60 and associated metadata are encoded into a bitstream 13 for transmission to a decoder-side device 12. For example, the audio encoder can encode the audio signal of the audio object using an algorithm based on MP encoding, and the metadata encoder 22 can use metadata encoding tools. In one aspect, the encoder-side device can (e.g., optionally) apply an identity matrix 74, which can have an order equal to the number of audio objects by multiplying the audio object 60 by the identity matrix 74.
[0116] On the decoder side device 12, the encoded audio object and the associated metadata are received through the encoded code stream 13 and decoded. The renderer 16 receives the decoded audio object and the associated metadata, and determines the speaker layout of the speaker 17 based on the output channel (speaker) layout 18. For example, the layout may indicate whether the speaker 17 belongs to a headphone or a loudspeaker of a surround sound system. The renderer generates driving signals by rendering the decoder audio object based on the speaker layout and metadata, and uses these signals to drive the speakers.
[0117] As described above, system 10 can be configured to encode and transmit several types of audio data, such as multi-channel audio and audio objects. In one aspect, system 10 can be configured to encode and transmit multiple (different or similar) types of audio as mixed audio. Fig.11 A block diagram of such a system according to one aspect is shown. Specifically, the figure shows that the encoder side device 11 includes a content merger 92, an audio encoder 21, and a metadata encoder 22. The encoder side device 11 can receive multiple types of audio. Specifically, it receives (e.g., from an audio source that can be a remote external source, such as an electronic server, and / or can be from a local memory of the device 11) multi-channel audio 31, HOA data 91 (with one or more HOA signals) and / or a set of one or more audio objects 60. In one aspect, different types of audio can be different audio formats of the same audio content (e.g., the soundtrack of a movie). For example, each of the audio formats can include the same sound source of a piece of audio content. On the other hand, different types of audio can correspond to different parts of the same audio content (e.g., with different sound sources), or at least some types of audio can be different segments of audio content. For example, the multi-channel audio 31 can be a surround sound audio content (5.1 surround sound format) of the soundtrack of a movie, and the HOA data 91 can include a virtual sound space of a virtual environment.
[0118] As shown, the encoder side device 11 is receiving one of each type of audio content. On the other hand, the device may include more or less types and / or more or less similar types of audio content. For example, the encoder side device 11 can receive multi-channel audio and HOA without receiving audio objects. As another example, the device can receive multiple types of multi-channel audio. For example, the multi-channel audio 31 may include the first segment of audio content and the second segment of audio content in the same (or different) surround sound format. In addition, the encoder side device can receive metadata 15 associated with at least some of the audio content in each segment of audio content being received. In one aspect, metadata 15 may include channel layout information of the signal mixed into the mixed audio. On the other hand, metadata may include information related to a specific type of audio (such as position data with audio objects) and other information (such as the type of audio processing operations that can be performed by one or more renderers of the decoder side device, such as gain, etc.).
[0119] The content merger 92 can be configured to receive audio content and merge the content into the mixed audio 93. For example, the merger can mix the audio signal of the received audio content into one or more mixed audio signals. In one aspect, the content merger 92 can first convert the received audio content into the same audio format (e.g., converting the HOA data 91 into a surround sound format), and then the audio signals of the same format can be mixed to produce mixed audio. In some aspects, the merger can use a mixing matrix to mix audio. In one aspect, the mixed audio can be a HOA representation of the mixed audio signal (e.g., based on the conversion matrix of the application, as described herein).
[0120] The audio encoder 21 receives the mixed audio 93 and encodes the mixed audio into a code stream 13 using an algorithm based on MP encoding, while the metadata encoder 22 may encode metadata 15 into the code stream.
[0121] In one aspect, multi-channel audio, audio objects, and HOA signals are combined into one or more mixed audio signals. In one aspect, the resulting mixed audio can be a combination of any heterogeneous input audio types, such as channel-based audio and HOA, etc. As described herein, the encoder-side device can receive multi-channel audio for combining and encoding. On the other hand, the device 11 can combine and encode any type of audio. For example, the device can receive one or more audio signals, such as a speech signal, which may not be part of a multi-channel audio format.
[0122] The decoder side device 12 includes an audio decoder 23, a metadata decoder 25, a separator 32, an output channel layout 18 (which may be stored in a memory of the decoder side device 12), a channel renderer 94, an HOA renderer 95, an object renderer 96, a mixer 97, and a speaker 17. In one aspect, although the three renderers are shown as separate blocks, they may be part of one renderer, such as the renderer 16 (which may also include the mixer 97).
[0123] The decoder side device 12 receives the encoded code stream 13 and metadata with mixed audio, and the decoders 23 and 25 decode the mixed audio 93 and metadata 15, respectively. The separator 32 can be configured to receive the mixed audio 93 and separate (or extract) one or more different types of audio content. For example, in this case, the separator 32 can extract the audio object 60, one or more HOA signals 91, and the multi-channel audio 31 from the mixed audio 93. In one aspect, the separator can separate the audio elements based on the received metadata 15. For example, metadata (e.g., channel layout information) can indicate which signals of the mixed audio 93 belong to a specific type of audio content, or it can indicate which parts of the mixed audio 93 can be associated with the type of audio content.
[0124] The decoder-side device 12 generates a first set of one or more drive signals by rendering the multi-channel audio 31 based on the metadata 15 and / or the output channel layout 18 (which may indicate the layout and number of the speakers 17), 2) generates a second set of one or more drive signals by rendering the HOA 91, and / or 3) generates a third set of one or more drive signals by rendering the audio object 60. Specifically, the channel renderer 94 can receive the multi-channel audio 31 and render the multi-channel audio signal to the speaker 17 based on the layout 18, the HOA renderer 95 can render the HOA 91 signal according to the layout of the speaker, and the object renderer 96 can use the metadata 15 (which may indicate the position data of the sound source within the sound field of the audio object) and the layout 18 to render the audio object 60 (so as to appropriately place the audio object within the sound field of the speaker). In one aspect, each of the renderers can generate a drive signal for each speaker 17.
[0125] The mixer 97 receives the rendered audio signals from each (or at least one) renderer and generates a set of mixed one or more drive signals by mixing the rendered (corresponding) drive signals together. The decoder-side device 12 drives the speaker 17 using the mixed drive signals. In one aspect, one or more spatial filters (e.g., HRTFs) may be applied to the mixed signals to generate spatially rendered audio, such as binaural audio that can be used to drive the speakers of a headphone.
[0126] In one aspect, based on the received bitstream 13, the mixed audio signal and metadata (which may include input audio channel layout information) are decoded (for example, the mixed audio signal may be determined using an algorithm based on MP encoding). Based on the decoded metadata, the mixed audio signal may be divided into original input types, such as channels, objects, and HOA signals. Based on the decoded metadata, the decoder-side device 12 may render the original input types (which are reconstructed by the decoder-side device), such as channels, objects, and HOA signals based on the output signal.
[0127] As described herein, the encoder side device 11 and the decoder side device 12 can perform one or more operations of one or more operation blocks. Specifically, these devices perform operations through one or more electronic components (such as one or more processors) of the corresponding devices, and these electronic components perform the operations of the operation blocks described herein.
[0128] In one aspect, the operations described herein that may be performed by the encoder side device 11 and / or the decoder side device 12 may be implemented in software (e.g., as instructions stored in a memory and executed by one or more processors of one or both devices) and / or may be implemented by the hardware logic structure of the devices described herein.
[0129] In another aspect, at least some of the operations performed by the system 10 as described herein may be performed by the encoder-side device 11 and / or the decoder-side device 12. For example, the encoder-side device may include two or more speakers and may be configured to perform decoder-side operations to render audio content received via one or more code streams. In another aspect, at least some of the operations may be performed by a remote server communicatively coupled to either device, for example, via a network (e.g., the Internet).
[0130] Fig.12 A block diagram of audio processing system hardware that can be used with any of the aspects described herein (e.g., encoder-side device 11 and / or decoder-side (or audio playback) device 12) is shown. The audio processing system can represent a general-purpose computer system or a special-purpose computer system. Note that although Fig.12 Various components of an audio processing system are shown that may be incorporated into one or more of the devices described herein, but this is merely one example of a particular implementation and is merely to illustrate the types of components that may be present in the system. Fig.12 It is not intended to represent any particular architecture or manner of interconnecting components, as such details are not germane to the various aspects described herein. It should also be understood that other components having a greater Fig.12 Other types of audio processing systems may include fewer or more components than those shown. Therefore, the processes described herein are not limited to those described herein. Fig.12Used together with hardware and software.
[0131] As shown Fig.12 As shown, an audio processing system (or system) 200 (e.g., a laptop, desktop computer, mobile phone, smart phone, tablet computer, smart speaker, head mounted display (HMD), headset (headphone), or infotainment system for an automobile or other vehicle) includes one or more buses 208 for interconnecting various components of the system. As is known in the art, one or more processors 207 are coupled to the bus 208. The one or more processors may be a microprocessor or a special purpose processor, a system on a chip (SOC), a central processing unit, a graphics processing unit, a processor created by an application specific integrated circuit (ASIC), or a combination thereof. The memory 206 may include a read-only memory (ROM), a volatile memory, and a non-volatile memory, or a combination thereof, coupled to the bus using techniques known in the art. A camera 201, one or more microphones 202, one or more speakers 203, and one or more displays 204 may be coupled to the bus.
[0132] Memory 206 may be connected to the bus and may include DRAM, a hard drive or flash memory, or a magnetic optical drive or magnetic storage, or an optical drive or other type of memory system that maintains data even after the system is powered off. In one aspect, processor 207 retrieves computer program instructions stored in a machine-readable storage medium (memory) and executes these instructions to perform the operations described herein.
[0133] Although not shown, audio hardware may be coupled to one or more buses 208 to receive audio signals to be processed and output (or played back) by speaker 203. Audio hardware may include digital-to-analog converters and / or analog-to-digital converters. Audio hardware may also include audio amplifiers and filters. Audio hardware may also be connected to microphone 202 (e.g., microphone array) to receive audio signals (whether analog or digital), digitize them when necessary, and transmit the signals to bus 208.
[0134] The network interface 205 can communicate with one or more remote devices and networks. For example, the interface can communicate via known technologies such as Wi-Fi, 3G, 4G, 5G, Bluetooth, ZigBee, or other equivalent technologies. The interface may include a wired or wireless transmitter and receiver that can communicate (e.g., receive and transmit data) with networked devices such as a server (e.g., the cloud) and / or other devices such as a remote speaker and a remote microphone.
[0135] It should be understood that aspects disclosed herein may utilize memory remote from the system, such as a network storage device coupled to the audio processing system via a network interface such as a modem or Ethernet interface. As is well known in the art, the bus 208 may be connected to each other via various bridges, controllers, and / or adapters. In one aspect, one or more network devices may be coupled to the bus 208. One or more network devices may be wired network devices (e.g., Ethernet) or wireless network devices (e.g., WI-FI, Bluetooth). In some aspects, it is described that various aspects (e.g., determining, estimating, analyzing, modeling, etc.) may be performed by a networked server in communication with a capture device.
[0136] Various aspects described herein may be embodied at least in part in software. That is, the techniques may be implemented in an audio processing system in response to its processor executing a sequence of instructions contained in a storage medium, such as a non-transitory machine-readable storage medium (e.g., DRAM or flash memory). In various aspects, hardwired circuitry may be used in conjunction with software instructions to implement the techniques described herein. Thus, the techniques are not limited to any specific combination of hardware circuitry and software, nor to any particular source of instructions executed by the audio processing system.
[0137] In this specification, certain terms are used to describe the features of various aspects. For example, in some cases, the terms "converter", "renderer", "layout", "separator", "controller", "component", "unit", "module", "logic", "extractor", "converter", "model", "combiner", "codebook", "filter", "generator", "calculator", "processor", "mixer", "matrix", "transpose matrix", "encoder" and "decoder" represent hardware and / or software configured to perform one or more processes or functions. For example, examples of "hardware" include, but are not limited to, integrated circuits such as processors (e.g., digital signal processors, microprocessors, application specific integrated circuits, microcontrollers, etc.). Therefore, as understood by those skilled in the art, different combinations of hardware and / or software can be implemented to perform the processes or functions described by the above terms. Of course, the hardware can be alternatively implemented as a finite state machine or even a combinational logic component. Examples of "software" include executable code in the form of applications, applets, routines, and even a series of instructions. As described above, the software can be stored in any type of machine-readable medium.
[0138] According to another aspect of the present disclosure, a method for encoding audio data performed by an encoder-side device includes: receiving an audio content in a surround sound format including metadata based on a surround sound format, the audio content including a group of surround sound channels, the group of surround sound channels including a group of full-range audio channels and a low-frequency effect (LFE) channel; dividing the group of surround sound channels into the group of full-range audio channels and the LFE channel; using a spherical harmonic function (SPH) matrix based on metadata to convert the full-range audio channels into a higher-order stereo high-fidelity reverberation (HOA) representation; encoding the HOA representation using an algorithm based on matching pursuit (MP) coding; and transmitting the encoded HOA representation, the LFE channel, and the metadata as at least one bitstream to a decoder-side device. In one aspect, the method also includes encoding the LFE channel separately from the group of full-range audio channels, the encoded LFE channel being included in the at least one bitstream.
[0139] According to another aspect of the present disclosure, a method for decoding audio data performed by a decoder-side device includes: receiving an encoded bitstream, the encoded bitstream including a higher-order stereo high-fidelity reverberation (HOA) signal, a low-frequency effect (LFE) channel, and associated metadata; generating a decoded HOA signal using an algorithm based on matching pursuit (MP) coding; converting the decoded HOA signal into a set of full-range audio channels using a spherical harmonic function (SPH) matrix based on metadata; merging the LFE channel and the set of full-range audio channels into a set of surround sound channels in a surround sound format based on the metadata; and driving a set of speakers using the surround sound channels according to the surround sound format.
[0140] According to another aspect of the present disclosure, a method for encoding audio data performed by an encoder-side device includes: receiving audio content in a surround sound format including metadata based on a surround sound format, wherein the audio content includes a group of surround sound audio channels; dividing the group of surround sound audio channels into a group of full-range audio channels and at least one low-frequency effect (LFE) channel based on the metadata; retrieving a transformation matrix from a codebook stored in a memory of the encoder-side device based on the metadata; using the transformation matrix to convert the group of full-range audio channels into a higher-order stereo high-fidelity reverberation (HOA) representation; and encoding the HOA representation into a bitstream using an algorithm based on matching pursuit (MP) coding for transmission to a decoder-side device.
[0141] According to another aspect of the present disclosure, a method for decoding audio data performed by a decoder-side device includes: receiving a set of higher-order stereo high-fidelity reverberation (HOA) signals, a low-frequency effect (LFE) channel, and metadata as an encoded bitstream; extracting the set of HOA signals, the LFE channel, and metadata from the encoded bitstream, wherein the set of HOA signals is extracted by using an algorithm based on matching pursuit (MP) coding; retrieving an HOA-to-channel conversion matrix from a codebook stored in a memory of the decoder-side device based on the metadata; generating a set of full-range audio channels by applying the HOA-to-channel conversion matrix to the extracted set of HOA signals; and generating multi-channel audio by merging the set of full-range audio channels with the LFE channel.
[0142] According to another aspect of the present disclosure, a method for encoding audio data performed by an encoder-side device includes: receiving audio content in a surround sound format including metadata based on a surround sound format; separating a group of audio channels and a low-frequency effect (LFE) channel from the audio content based on the metadata; converting the group of audio channels into converted audio based on a unit matrix, the unit matrix being based on the metadata, wherein the converted audio includes a group of channels, the group of channels including the group of audio channels and one or more additional channels, so that the total number of channels is equal to the order of a stereo high-fidelity reverberation, wherein the one or more additional channels do not include audio content; and encoding the converted audio into a bitstream using an algorithm based on matching pursuit (MP) coding.
[0143] According to another aspect of the present disclosure, a method for decoding audio data performed by a decoder-side device includes: receiving audio content of a low frequency effect (LFE) channel of a segment of surround sound audio content and metadata associated with the audio content and the LFE channel as an encoded bitstream, wherein the audio content includes a group of channels generated by applying a transformation matrix based on an identity matrix to a group of non-LFE surround sound channels of the segment of surround sound audio content, the identity matrix being based on the metadata, wherein the group of channels includes the group of non-LFE surround sound channels and at least one additional channel that does not include audio data; generating a transposed matrix of the transformation matrix; reconstructing the group of non-LFE surround sound channels and the at least one additional channel by applying the transposed matrix of the transformation matrix to the audio content; and driving a group of speakers using the group of non-LFE surround sound channels and the LFE channel.
[0144] According to another aspect of the present disclosure, a method for encoding audio data performed by an encoder-side device includes: receiving audio content in a surround sound format including metadata based on a surround sound format; separating a group of surround sound audio channels and a low frequency effect (LFE) channel from the audio content based on the metadata; encoding the group of surround sound audio channels using an algorithm based on matching pursuit (MP) coding; encoding the LFE channel and the metadata; and transmitting the group of encoded surround sound audio channels, the LFE channel, and the metadata as a code stream to a decoder-side device.
[0145] According to another aspect of the present disclosure, a method for decoding audio data performed by a decoder-side device includes: receiving an encoded bitstream, the encoded bitstream including a group of surround sound channels, a low frequency effect (LFE) channel and metadata; extracting the channels and metadata by executing one or more decoding algorithms including an algorithm based on matching pursuit (MP) encoding; and generating a surround sound format including the group of surround sound channels and the LFE channel based on the metadata.
[0146] According to another aspect of the present disclosure, a method for encoding audio data performed by an encoder-side device includes: receiving a group of one or more audio objects, each audio object having an audio signal and associated metadata; determining the number of audio objects in the group based on the associated metadata; determining a transformation matrix from a transformation matrix codebook based on the number of audio objects in the group, wherein the transformation matrix is used to 1) map each audio signal of the group to one of a group of surround sound speaker positions, and 2) convert a group of one or more audio objects into a higher-order stereo high-fidelity reverberation (HOA) representation according to the group of surround sound speaker positions, wherein the number of speaker positions of the transformation matrix is greater than or equal to the number of audio signals of the group; generating an HOA representation of the group by applying the transformation matrix to the group of one or more audio objects; and encoding the HOA representation into a bitstream including the associated metadata using an algorithm based on matching pursuit (MP) coding.
[0147] According to another aspect of the present disclosure, a method for decoding audio data performed by a decoder-side device includes: receiving an encoded bitstream, the encoded bitstream including 1) a higher-order stereo high-fidelity reverberation (HOA) representation of audio content and 2) metadata associated with the audio content; determining a number of audio objects of the audio content based on the metadata; determining a transformation matrix from a transformation matrix codebook based on the number of audio objects, wherein the transformation matrix is used to 1) convert the HOA representation into the group of surround sound audio channels, and 2) reconstruct one or more audio objects from the group of surround sound audio channels; generating one or more audio objects from the HOA representation using the transformation matrix; and spatially rendering a loudspeaker or headphone drive signal using the one or more audio objects and the metadata.
[0148] According to another aspect of the present disclosure, a method for encoding audio data performed by an encoder-side device includes: generating a set of spherical projection grids, each spherical projection grid having a different number of speaker positions on a surface of the spherical projection grid according to a spherical T-shaped design; generating a transformation matrix for each spherical projection grid in the set of spherical projection grids and storing the transformation matrix in a codebook, the transformation matrix being used to convert one or more audio objects into a high-order stereo high-fidelity reverberation (HOA) using spherical harmonics of a corresponding number of speaker positions; receiving a set of one or more audio objects, each audio object having an audio signal and associated metadata; determining the number of audio objects in the group based on the associated metadata; retrieving a specific transformation matrix associated with the number of speaker positions based on the number of audio objects and from the codebook, the number of speaker positions being greater than or equal to the number of audio objects in the group; generating an HOA representation of the group by applying the set of one or more audio signals to the specific transformation matrix; and encoding the HOA representation into a codestream including the associated metadata using an algorithm based on matching pursuit (MP) coding.
[0149] According to another aspect of the present disclosure, a method for decoding audio data performed by a decoder-side device includes: receiving a bitstream comprising encoded higher-order stereo high-fidelity reverberation (HOA) data of a group of one or more audio objects and encoded audio object metadata; decoding the encoded HOA data using an algorithm based on matching pursuit (MP) coding; determining the number of group audio objects using the audio object metadata from the bitstream; selecting from a transformation matrix codebook comprising a group of transformation matrices, each transformation matrix being used to convert the HOA data into a different number of audio objects; generating a group of one or more audio objects reconstructed from the decoded HOA data by applying the selected transformation matrix to the decoded HOA data; and spatially rendering a loudspeaker or headphone drive signal using the one or more audio objects and the metadata.
[0150] According to another aspect of the present disclosure, a method for encoding audio data performed by an encoder-side device includes: generating a set of spherical projection grids, each spherical projection grid having a different number of speaker positions on the surface of the spherical projection grid according to a Fliege-Maier set; generating a transformation matrix for each spherical projection grid in the set of spherical projection grids and storing the transformation matrix in a codebook, the transformation matrix being used to convert one or more audio objects into a higher-order stereo high-fidelity reverberation (HOA) using spherical harmonics of a corresponding number of speaker positions; receiving a set of one or more audio objects, each audio object having an audio signal and associated metadata; determining the number of audio objects in the group based on the associated metadata; retrieving a specific transformation matrix associated with the number of speaker positions based on the number of audio objects and from the codebook, the number of speaker positions being greater than or equal to the number of audio objects in the group; generating an HOA representation of the group by applying each audio signal of the group to a specific transformation matrix; and encoding the HOA representation into a codestream including the associated metadata using an algorithm based on matching pursuit (MP) coding.
[0151] According to another aspect of the present disclosure, a method for encoding audio data performed by an encoder-side device includes: receiving a group of audio object signals and associated metadata of a group of audio objects; determining the number of audio objects in the group based on the associated metadata; converting the group of audio object signals into an HOA representation based on a unit matrix based on the associated metadata, wherein the HOA representation includes a group of HOA signals, the group of HOA signals including the group of audio object signals and one or more additional audio signals, so that the total number of audio signals is equal to the number of HOA signals of the order of stereo high-fidelity reverberation represented by the HOA, wherein the one or more additional audio signals do not include audio data of the group of audio objects; and encoding the HOA representation into a bitstream including the associated metadata using an algorithm based on matching pursuit (MP) coding.
[0152] According to another aspect of the present disclosure, a method for decoding audio data performed by a decoder-side device includes: receiving an encoded bitstream, the encoded bitstream including 1) a group of high-order stereo high-fidelity reverberation (HOA) channels and 2) metadata associated with a group of audio objects; determining the number of audio objects in the group based on the metadata; determining a transformation matrix based on a unit matrix based on the number of audio objects, the unit matrix including an order equal to the number of audio objects; converting the group of HOA channels into a group of audio signals of the group of audio objects and one or more additional audio signals, wherein the number of audio signals based on the transformation is equal to the number of HOA channels in the group; generating a group of driving signals by rendering the group of audio signals based on the metadata; and using the group of driving signals to drive a group of amplifiers or a group of speakers of headphones.
[0153] In one aspect, the size of the transformation matrix is based on the number of HOA channels in the group.
[0154] According to another aspect of the present disclosure, a method for encoding audio data performed by an encoder-side device includes: receiving a group of one or more audio objects, each audio object including an audio signal and associated metadata; encoding each audio signal of the group of one or more audio objects using an algorithm based on matching pursuit (MP) coding; and transmitting the encoded audio signal together with the associated metadata as a code stream to a decoder-side device.
[0155] According to another aspect of the present disclosure, a method for decoding audio data performed by a decoder-side device includes: receiving an encoded code stream, the encoded code stream including a set of audio signals and associated metadata of a set of audio objects; decoding the set of audio signals using an algorithm based on matching pursuit (MP) coding; determining a speaker layout of a set of speakers of the decoder-side device; and generating a set of driving signals by rendering the set of audio signals based on the speaker layout and the metadata; and driving the set of speakers using the set of driving signals.
[0156] According to another aspect of the present disclosure, a method for encoding audio data performed by an encoder-side device includes: receiving: a set of audio signals in a surround sound format; a set of one or more audio objects, each audio object including an audio signal; a high-order stereo high-fidelity reverberation (HOA) representation of the sound field, the HOA representation including a set of HOA signals; and metadata associated with at least some of the received signals; merging the received signals into mixed audio; encoding the mixed audio using an algorithm based on matching pursuit (MP) coding; and transmitting the encoded mixed audio in a bitstream including the metadata to a decoder-side device.
[0157] According to another aspect of the present disclosure, a method for decoding audio data performed by a decoder-side device includes: receiving a bitstream including an encoded audio mix and metadata associated with the encoded audio mix; generating a decoded audio mix by decoding the encoded audio mix using an algorithm based on matching pursuit (MP) coding; extracting one or more surround sound audio channels, one or more audio objects, and one or more HOA signals from the decoded audio mix; based on the metadata and an output channel layout of one or more speakers, 1) generating a first group of one or more drive signals by rendering one or more surround sound audio channels, 2) generating a second group of one or more drive signals by rendering one or more audio objects, and 3) generating a third group of one or more drive signals by rendering one or more HOA signals; generating a group of mixed one or more drive signals by mixing the first group, the second group, and the third group together; and driving one or more speakers of the decoder-side device using the group of mixed one or more drive signals.
[0158] In one aspect, the encoded input audio signal and metadata can be transmitted from the encoder side to the decoder side in the same audio code stream or in different audio code streams (e.g., at different times). In another aspect, the metadata includes speaker layout information associated with the surround sound audio channels and position information related to a set of one or more audio objects. In another aspect, converting a first group of non-LFE channels to an HOA representation includes determining a transformation matrix based on the surround sound speaker layout information of the non-LFE channels; and generating the HOA representation by applying the transformation matrix to the first group. In another aspect, the identity matrix is based on the number of non-LFE channels, wherein the conversion includes applying the identity matrix to the non-LFE channels.
[0159] Certain portions of the foregoing detailed description have been presented in terms of algorithms and symbolic representations of operations on data bits in a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the art of audio processing to most effectively convey the substance of their work to others skilled in the art. An algorithm is here and generally referred to as a self-consistent sequence of operations leading to a desired result. These operations are those requiring physical manipulation of physical quantities. However, it should be kept in mind that all of these and similar terms are associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless otherwise specifically stated, it will be apparent from the foregoing discussion that it will be understood that throughout this specification, discussions using terms such as those given in the claims below refer to actions and processes of an audio processing system or similar electronic device that manipulates data represented as physical (electronic) quantities in the system's registers and memories and converts them into other data similarly represented as physical quantities in the system's memories or registers or other such information storage, transmission or display devices.
[0160] The processes and blocks described herein are not limited to the specific examples described, and are not limited to the specific order used as examples in this article. On the contrary, any processing block can be reordered, combined or removed, executed in parallel or serially as needed to achieve the above results. The processing blocks associated with the implementation of the audio processing system can be executed by one or more programmable processors executing one or more computer programs stored on a non-transient computer-readable storage medium to perform the functions of the system. All or part of the audio processing system can be implemented as a dedicated logic circuit (e.g., FPGA (field programmable gate array) and / or ASIC (application-specific integrated circuit)). All or part of the audio system can be implemented using an electronic hardware circuit including at least one of an electronic device such as a processor, a memory, a programmable logic device, or a logic gate. In addition, the process can be implemented in any combination of hardware devices and software components.
[0161] As previously described, one aspect of the present disclosure may be a non-transitory machine-readable medium (such as a microelectronic memory) having stored thereon instructions that program one or more data processing components (generally referred to herein as "processors") to perform encoding (e.g., encoding and / or decoding) of audio data (content), digital signal processing operations, audio (spatial) rendering operations, network operations, and audio signal processing operations, as described herein. In other aspects, some of these operations may be performed by specific hardware components that include hardwired logic. Alternatively, those operations may be performed by any combination of programmed data processing components and fixed hardwired circuit components.
[0162] To assist the Patent Office and any reader of any patent issuing in this application in interpreting the appended claims, Applicants wish to note that they do not intend for any of the appended claims or claim elements to invoke 35 U.S.C. § 112(f) unless the words “means for” or “step for” are expressly used in a particular claim.
[0163] It is understood that the use of personally identifiable information should be subject to privacy policies and practices that are generally recognized to meet or exceed industry or government requirements for maintaining user privacy. Specifically, personally identifiable information data should be managed and processed to minimize the risk of unintentional or unauthorized access or use, and the nature of the authorized use should be clearly stated to users.
[0164] Although certain aspects have been described and shown in the drawings, it is to be understood that such aspects are merely illustrative of the broad disclosure and not limiting, and the disclosure is not limited to the specific constructions and arrangements shown and described, since various other modifications may occur to one of ordinary skill in the art. Therefore, the description is to be regarded as illustrative and not limiting.
[0165] In some aspects, the present disclosure may include language such as “at least one of [element A] and [element B]”. The language may refer to one or more of these elements. For example, “at least one of A and B” may refer to “A”, “B”, or “A and B”. Specifically, “at least one of A and B” may refer to “at least one of A and at least one of B” or “at least either A or B”. In some aspects, the present disclosure may include language such as “[element A], [element B], and / or [element C]”. The language may refer to any one of these elements or any combination thereof. For example, “A, B, and / or C” may refer to “A”, “B”, “C”, “A and B”, “A and C”, “B and C”, or “A, B, and C”.
Claims
1. A decoder side method, the method comprising: receiving a bitstream comprising an encoded representation of an input audio signal and metadata associated with the input audio signal; generating a decoded representation of the input audio signal by decoding the encoded representation using an algorithm based on matching pursuit (MP) coding; generating a plurality of audio drive signals by rendering the input audio signal based on the metadata; as well as A plurality of speakers are driven using the plurality of audio drive signals.
2. The method according to claim 1, wherein: The decoded representation comprises a higher order ambisonics (HOA) representation of the input audio signal, wherein the method further comprises applying a transformation matrix to the HOA representation to reconstruct the input audio signal.
3. The method according to claim 2, wherein: The input audio signal comprises a plurality of full-range audio channels in a surround sound format, wherein the method further comprises receiving at least one bandwidth-limited audio channel associated with the surround sound format, wherein generating the plurality of audio drive signals comprises assigning each of the channels to a particular speaker of the plurality of speakers based on the metadata.
4. The method according to claim 2, wherein: The input audio signal includes a set of one or more audio objects, and the metadata includes position information related to the set of one or more audio objects, wherein the multiple audio driving signals are generated by spatially rendering the set of one or more audio objects according to the position information.
5. The method according to claim 4, further comprising: determining a quantity of the set of one or more audio objects; as well as The transformation matrix is determined based on the quantity. 6 . The method of claim 4 , further comprising receiving an output speaker layout of the plurality of speakers, wherein the set of one or more audio objects are spatially rendered according to the output speaker layout.
7. The method according to claim 2, wherein: The code stream is received from an encoder-side device, wherein the conversion matrix is an inverse matrix of a matrix used by the encoder-side device to generate the encoded representation of the input audio signal.
8. The method according to claim 1, wherein: The decoded representation of the input audio signal comprises a mixed signal, wherein the method further comprises separating the mixed signal into: Multiple surround sound channels in surround sound format, one or more audio objects comprising one or more audio signals, and The HOA data includes multiple HOA signals.
9. The method according to claim 8, wherein: Generating the plurality of audio driving signals comprises: rendering the plurality of surround sound channels, the one or more audio signals, and the plurality of HOA signals according to the metadata and an output speaker layout of the plurality of speakers; and The rendering is mixed into the plurality of audio drive signals.
10. A decoder side device, comprising: at least one processor; as well as a memory having stored therein instructions which, when executed by the at least one processor, cause the decoder-side device to: receiving a bitstream comprising an encoded representation of an input audio signal and encoded metadata associated with the input audio signal; generating a decoded representation of the input audio signal by decoding the encoded representation using an algorithm based on matching pursuit (MP) coding; generating a plurality of audio drive signals based on the decoded representation of the input audio signal and the metadata; as well as A plurality of speakers are driven using the plurality of audio drive signals.
11. The decoder-side device according to claim 10, wherein: The decoded representation comprises a higher order stereo ambidextrous (HOA) representation of the input audio signal, wherein the memory has further instructions to apply a transformation matrix to the HOA representation to reconstruct the input audio signal, wherein the plurality of audio drive signals are generated by rendering the input audio signal based on the metadata.
12. The decoder-side device according to claim 11, wherein: The input audio signal comprises a plurality of full-range audio channels in a surround sound format, wherein the memory has further instructions for receiving at least one bandwidth-limited audio channel associated with the surround sound format, wherein the instructions for generating the plurality of audio drive signals comprise instructions for assigning each of the channels to a particular speaker of the plurality of speakers based on the metadata.
13. The decoder-side device according to claim 11, wherein: The input audio signal includes a group of one or more audio objects, and the metadata includes position information related to the group of one or more audio objects, wherein the multiple audio driving signals are generated by spatially rendering the group of one or more audio objects according to the position information and the layout of the multiple speakers.
14. The decoder-side device according to claim 11, wherein: The code stream is received from an encoder-side device, wherein the conversion matrix is an inverse matrix of a matrix used by the encoder-side device to generate the encoded representation of the input audio signal.
15. The decoder-side device according to claim 10, wherein: The decoded representation of the input audio signal comprises a mixed signal, wherein the memory has further instructions to separate the mixed signal into: Multiple surround sound channels in surround sound format, one or more audio objects comprising one or more audio signals, and The HOA data includes multiple HOA signals.
16. The decoder-side device according to claim 15, wherein: The instructions for generating the plurality of audio drive signals include instructions for: rendering the plurality of surround sound channels, the one or more audio signals, and the plurality of HOA signals according to the metadata and an output speaker layout of the plurality of speakers; as well as The rendering is mixed into the plurality of audio drive signals.
17. An encoder-side method, the method comprising: Receiving an input audio signal of a segment of audio content and metadata associated with the input audio signal; encoding the input audio signal using an algorithm based on matching pursuit (MP) coding; as well as The encoded input audio signal and the metadata are transmitted to an audio playback device.
18. The method according to claim 17, wherein: The input audio signal includes a plurality of surround sound audio channels including a sound source, wherein the plurality of surround sound audio channels include a first group of one or more full-range audio channels and a second group of one or more limited-bandwidth audio channels, wherein the method further includes converting the first group into a high-order stereo high-fidelity reverberation (HOA) representation of the sound source, wherein encoding includes encoding the HOA representation into a bitstream using the MP-based encoding algorithm for transmission to the audio playback device.
19. The method of claim 18, further comprising encoding the second group into the codestream separately from the encoded HOA representation.
20. The method according to claim 18, wherein: The metadata includes surround sound speaker layout information of the plurality of surround sound audio channels, wherein the first group is converted into an HOA representation according to the surround sound speaker layout information.
21. The method according to claim 17, The receiving of the input audio signal comprises receiving a group of one or more audio objects, each audio object having at least one audio signal, wherein the method further comprises generating a higher-order stereo ambidextrous (HOA) representation of the set of one or more audio objects, The encoding includes encoding the HOA representation into a bitstream using the MP-based encoding algorithm for transmission to the audio playback device.
22. The method according to claim 21, further comprising: determining a quantity of the set of one or more audio objects; as well as Determine a transformation matrix based on the quantities, Wherein the HOA representation is generated by applying the transformation matrix to the set of one or more audio objects.
23. The method according to claim 17, The input audio signal comprises: a plurality of surround sound audio channels of the segment of audio content, a high-order stereo ambidextrous (HOA) representation of the segment of audio content, and a set of one or more audio objects of the segment of audio content, The method further includes generating a mixed audio signal, the mixed audio signal including the multiple surround sound audio channels, the HOA representation and the set of one or more audio objects, wherein encoding includes encoding the mixed audio signal into a bitstream using the MP encoding-based algorithm for transmission to the audio playback device.
24. A method for encoding audio data performed by an encoder-side device, the method comprising: receiving audio content in a surround sound format including metadata based on the surround sound format, wherein the audio content includes a plurality of surround sound channels including a plurality of full-range audio channels and a low frequency effects (LFE) channel; dividing the plurality of surround sound channels into the plurality of full-range audio channels and the LFE channel; converting the full-range audio channels into a higher-order stereo ambidextrous (HOA) representation using a spherical harmonics (SPH) matrix based on the metadata; encoding the HOA representation using an algorithm based on matching pursuit (MP) coding; as well as The encoded HOA representation, the LFE channel and the metadata are transmitted as at least one code stream to a decoder-side device.
25. The method of claim 24, further comprising encoding the LFE channel separately from the plurality of full-range audio channels, wherein the encoded LFE channel is included in the at least one bitstream.
26. A method for decoding audio data performed by a decoder-side device, the method comprising: receiving an encoded bitstream, the encoded bitstream comprising a higher-order stereo high-fidelity reverberation (HOA) signal, a low-frequency effects (LFE) channel, and associated metadata; The decoded HOA signal is generated using an algorithm based on matching pursuit (MP) coding; converting the decoded HOA signal into a plurality of full-range audio channels using a spherical harmonics (SPH) matrix based on the metadata; combining the LFE channel and the plurality of full-range audio channels into a plurality of surround sound channels in a surround sound format based on the metadata; as well as A plurality of speakers are driven using the surround sound channels according to the surround sound format.
27. A method for encoding audio data performed by an encoder-side device, the method comprising: receiving audio content in a surround sound format including metadata based on the surround sound format, wherein the audio content includes a plurality of surround sound audio channels; separating the plurality of surround sound audio channels into a plurality of full-range audio channels and at least one low frequency effects (LFE) channel based on the metadata; retrieving a transformation matrix from a codebook stored in a memory of the encoder-side device based on the metadata; converting the plurality of full-range audio channels into a higher-order stereo ambidextrous (HOA) representation using the conversion matrix; as well as The HOA representation is encoded into a bitstream using a matching pursuit (MP) coding algorithm for transmission to a decoder side device.
28. A method for decoding audio data performed by a decoder-side device, the method comprising: Receive multiple high-order stereo audiophile (HOA) signals, low-frequency effects (LFE) channels and metadata as encoded bitstreams; extracting the plurality of HOA signals, the LFE channel, and the metadata from the encoded bitstream, wherein the plurality of HOA signals are extracted by using an algorithm based on matching pursuit (MP) coding; retrieving an HOA-to-channel conversion matrix from a codebook stored in a memory of the decoder-side device based on the metadata; generating a plurality of full-range audio channels by applying the HOA-to-channel conversion matrix to the extracted plurality of HOA signals; as well as Multi-channel audio is generated by combining the plurality of full-range audio channels with the LFE channel.
29. A method for encoding audio data performed by an encoder-side device, the method comprising: receiving audio content in a surround sound format including metadata based on the surround sound format; separating a plurality of audio channels and a low frequency effects (LFE) channel from the audio content based on the metadata; converting the plurality of audio channels into converted audio based on an identity matrix, the identity matrix being based on the metadata, wherein the converted audio comprises a plurality of channels, the plurality of channels comprising the plurality of audio channels and one or more additional channels such that a total number of channels is equal to an order of a stereo high-fidelity reverberation, wherein the one or more additional channels do not include the audio content; as well as The converted audio is encoded into the bitstream using an algorithm based on matching pursuit (MP) coding.
30. A method for decoding audio data performed by a decoder-side device, the method comprising: receiving audio content of a low frequency effects (LFE) channel of a segment of surround sound audio content and metadata associated with the audio content and the LFE channel as an encoded bitstream, wherein the audio content includes a plurality of channels generated by applying a transformation matrix based on an identity matrix to a plurality of non-LFE surround sound channels of the segment of surround sound audio content, the identity matrix being based on the metadata, wherein the plurality of channels includes the plurality of non-LFE surround sound channels and at least one additional channel that does not include audio data; Generating a transposed matrix of the transformation matrix; reconstructing the plurality of non-LFE surround sound channels and the at least one additional channel by applying the transposed matrix of the transformation matrix to the audio content; as well as A plurality of speakers are driven using the plurality of non-LFE surround sound channels and the LFE channel.
31. A method for encoding audio data performed by an encoder-side device, the method comprising: receiving audio content in a surround sound format including metadata based on the surround sound format; separating a plurality of surround sound audio channels and a low frequency effects (LFE) channel from the audio content based on the metadata; encoding the plurality of surround sound audio channels using an algorithm based on matching pursuit (MP) coding; encoding the LFE channel and the metadata; as well as The encoded multiple surround sound audio channels, LFE channel and metadata are transmitted to a decoder side device as a code stream.
32. A method for decoding audio data performed by a decoder-side device, the method comprising: receiving an encoded bitstream including a plurality of surround sound channels, a low frequency effects (LFE) channel and metadata; extracting the channel and the metadata by executing one or more decoding algorithms including an algorithm based on matching pursuit (MP) coding; Based on the metadata, a surround sound format including the plurality of surround sound channels and the LFE channel is generated.
33. A method for encoding audio data performed by an encoding device, the method comprising: Receiving a set of one or more audio objects, each audio object having an audio signal and associated metadata; determining a number of audio objects in the group based on the associated metadata; determining a transformation matrix from a transformation matrix codebook based on the number of audio objects in the group, wherein the transformation matrix is used to 1) map each audio signal of the group to one of a plurality of surround sound speaker positions, and 2) transform the group of one or more audio objects into a higher order stereo ambisonics (HOA) representation according to the plurality of surround sound speaker positions, wherein the number of speaker positions of the transformation matrix is greater than or equal to the number of audio signals of the group; generating the HOA representation of the group by applying the transformation matrix to the group of one or more audio objects; as well as The HOA representation is encoded into a codestream including the associated metadata using an algorithm based on Matching Pursuit (MP) coding.
34. A method for decoding audio data performed by a decoder-side device, the method comprising: Receiving an encoded bitstream, the encoded bitstream comprising 1) a higher-order stereo ambisonics (HOA) representation of audio content and 2) metadata associated with the audio content; determining a number of audio objects of the audio content based on the metadata; determining a transformation matrix from a transformation matrix codebook based on the number of audio objects, wherein the transformation matrix is used to 1) transform the HOA representation into the plurality of surround sound audio channels, and 2) reconstruct the one or more audio objects from the plurality of surround sound audio channels; generating the one or more audio objects from the HOA representation using the transformation matrix; as well as A loudspeaker or headphone drive signal is spatially rendered using the one or more audio objects and the metadata.
35. A method for encoding audio data performed by an encoder-side device, the method comprising: generating a plurality of spherical projection grids, each spherical projection grid having a different number of speaker positions on a surface of the spherical projection grid according to a spherical T-shaped design; generating a transformation matrix for each of the plurality of spherical projection grids and storing the transformation matrix in a codebook, the transformation matrix being used to transform one or more audio objects into a higher order stereo ambisonics (HOA) using spherical harmonics for a corresponding number of speaker positions; Receiving a set of one or more audio objects, each audio object having an audio signal and associated metadata; determining a number of audio objects in the group based on the associated metadata; retrieving, based on the number of audio objects and from the codebook, a particular transformation matrix associated with a number of speaker positions, the number of speaker positions being greater than or equal to the number of audio objects in the group; generating an HOA representation of the set of one or more audio signals by applying the set of one or more audio signals to the particular transformation matrix; as well as The HOA representation is encoded into a codestream including the associated metadata using an algorithm based on Matching Pursuit (MP) coding.
36. A method for decoding audio data performed by a decoder-side device, the method comprising: receiving a bitstream, the bitstream comprising encoded higher-order stereo high-fidelity reverberation (HOA) data of a set of one or more audio objects and encoded audio object metadata; decoding the encoded HOA data using an algorithm based on matching pursuit (MP) coding; determining a number of the set of audio objects using the audio object metadata from the bitstream; selecting from a transformation matrix codebook comprising a plurality of transformation matrices, each transformation matrix in the plurality of transformation matrices being used to transform the HOA data into a different number of audio objects; generating a set of one or more audio objects reconstructed from the decoded HOA data by applying the selected transformation matrix to the decoded HOA data; as well as A loudspeaker or headphone drive signal is spatially rendered using the one or more audio objects and the metadata.
37. A method for encoding audio data performed by an encoder-based device, the method comprising: generating a plurality of spherical projection grids, each spherical projection grid having a different number of speaker positions on a surface of the spherical projection grid according to a Fliege-Maier set; generating a transformation matrix for each of the plurality of spherical projection grids and storing the transformation matrix in a codebook, the transformation matrix being used to transform one or more audio objects into a higher order stereo ambisonics (HOA) using spherical harmonics for a corresponding number of speaker positions; Receiving a set of one or more audio objects, each audio object having an audio signal and associated metadata; determining a number of audio objects in the group based on the associated metadata; retrieving, based on the number of audio objects and from the codebook, a particular transformation matrix associated with a number of speaker positions, the number of speaker positions being greater than or equal to the number of audio objects in the group; generating an HOA representation of the group by applying each audio signal of the group to the particular transformation matrix; as well as The HOA representation is encoded into a codestream including the associated metadata using an algorithm based on Matching Pursuit (MP) coding.
38. A method for encoding audio data performed by an encoder-side device, the method comprising: receiving a plurality of audio object signals and associated metadata for a plurality of audio objects; determining a number of audio objects in the plurality of audio objects based on the associated metadata; converting the plurality of audio object signals into an HOA representation based on an identity matrix, the identity matrix being based on the associated metadata, wherein the HOA representation comprises a plurality of HOA signals, the plurality of HOA signals comprising the plurality of audio object signals and one or more additional audio signals, such that a total number of audio signals is equal to a number of HOA signals of an order of a stereo high-fidelity reverberation of the HOA representation, wherein the one or more additional audio signals do not include audio data of the plurality of audio objects; as well as The HOA representation is encoded into a codestream including the associated metadata using an algorithm based on matching pursuit (MP) coding.
39. A method for decoding audio data performed by a decoder-side device, the method comprising: Receiving an encoded bitstream, the encoded bitstream comprising 1) a plurality of higher-order stereo ambisonic (HOA) channels and 2) metadata associated with a plurality of audio objects; determining a number of audio objects in the plurality of audio objects based on the metadata; determining a transformation matrix based on an identity matrix based on the number of audio objects, the identity matrix including an order equal to the number of audio objects; converting the plurality of HOA channels into a plurality of audio signals of the plurality of audio objects and one or more additional audio signals, wherein the number of audio signals based on the conversion is equal to the number of HOA channels in the plurality of HOA channels; generating a plurality of drive signals by rendering the plurality of audio signals based on the metadata; as well as The plurality of driving signals are used to drive a plurality of microphones or a plurality of speakers of the headphone.
40. The method of claim 39, wherein: The size of the transformation matrix is based on the number of HOA channels in the plurality of HOA channels.
41. A method for encoding audio data performed by an encoder-side device, the method comprising: receiving a set of one or more audio objects, each audio object comprising an audio signal and associated metadata; encoding each audio signal of the set of one or more audio objects using an algorithm based on matching pursuit (MP) coding; as well as The encoded audio signal together with the associated metadata is transmitted to a decoder-side device as a code stream.
42. A method for decoding audio data performed by a decoder-side device, the method comprising: receiving an encoded bitstream, the encoded bitstream comprising a plurality of audio signals of a plurality of audio objects and associated metadata; Decoding the plurality of audio signals using an algorithm based on matching pursuit (MP) coding; Determining a speaker layout of a plurality of speakers of the decoder-side device; as well as generating a plurality of drive signals by rendering the plurality of audio signals based on the speaker layout and the metadata; as well as The plurality of speakers are driven using the plurality of drive signals.
43. A method for encoding an audio channel performed by an encoder-side device, the method comprising: take over: Multiple audio signals in surround sound format, a set of one or more audio objects, each audio object comprising an audio signal, a higher-order ambisonic (HOA) representation of the sound field, the HOA representation comprising a plurality of HOA signals, and metadata associated with at least some of the received signals; combining the received signals into a mixed audio; Encoding the mixed audio using an algorithm based on matching pursuit (MP) coding; as well as The encoded mixed audio in the code stream including the metadata is transmitted to a decoder-side device.
44. A method for decoding audio content performed by a decoder-side device, the method comprising: receiving a bitstream comprising an encoded audio mix and metadata associated with the encoded audio mix; generating a decoded audio mix by decoding the encoded audio mix using an algorithm based on matching pursuit (MP) coding; extracting one or more surround sound audio channels, the one or more audio objects, and the one or more HOA signals from the decoded audio mix; Based on the metadata and an output channel layout of one or more speakers, 1) generate a first set of one or more drive signals by rendering the one or more surround sound audio channels, 2) generate a second set of one or more drive signals by rendering the one or more audio objects, and 3) generate a third set of one or more drive signals by rendering the one or more HOA signals; generating a mixed set of one or more drive signals by mixing the first set, the second set, and the third set together; as well as The one or more speakers of the decoder-side device are driven using the set of mixed one or more driving signals.
