Coding and decoding method, device and system for directional sound source

Abstract

The invention relates to a method, a device and a system for encoding and decoding a directional sound source. Some encoding methods may involve receiving a monaural signal corresponding to an audio object and a representation of a radiation pattern corresponding to the audio object. The radiation pattern may include sound levels corresponding to a plurality of sampling times, a plurality of frequency bands, and a plurality of directions. The method may involve encoding the monaural audio signal and encoding the source radiation pattern to determine radiation pattern metadata. Encoding the radiation pattern may involve determining a spherical harmonic transformation of the representation of a radiation pattern and compressing the spherical harmonic transformation to obtain encoded radiation pattern metadata.

Description

[0001] Information about divisional applications

[0002] This case is a divisional application of the invention patent application with the application date of April 15, 2019, application number 201980013721.7, and invention name "Method, device and system for encoding and decoding of directional sound sources".

[0003] CROSS REFERENCE TO RELATED APPLICATIONS

[0004] This application claims the benefit of priority to U.S. Patent Application No. 62 / 658,067, filed on April 16, 2018; U.S. Patent Application No. 62 / 681,429, filed on June 6, 2018, and U.S. Patent Application No. 62 / 741,419, filed on October 4, 2018, the entire contents of which are hereby incorporated by reference herein. Technical Field

[0005] The invention relates to encoding and decoding of directional sound sources and auditory scenes based on multiple dynamic and / or moving directional sources Background Art

[0006] Real-world sound sources, whether natural or artificial (speakers, musical instruments, voices, mechanical devices) radiate sound in a non-isotropic manner. Characterizing the radiation pattern (or "directivity") of a sound source can be critical for correct rendering, especially in the context of interactive environments such as video games and virtual / augmented reality (VR / AR) applications. In these environments, users typically interact with directional audio objects by walking around them, thereby changing their auditory stereoscopic perception of the sound produced (also known as 6 degrees of freedom [DoF] rendering). Users can also grab and dynamically rotate virtual objects, which also requires rendering different directions in the radiation pattern of the corresponding sound source. In addition to more realistically rendering the direct propagation effect from the source to the listener, the radiation characteristics will also play a major role in the high-order acoustic coupling between the source and its environment (e.g., the virtual environment in the game), thereby affecting the reverberant sound (i.e., sound waves that travel back and forth as in an echo). Therefore, such reverberation may affect other spatial cues, such as perceived distance.

[0007] Most audio game engines provide some way to represent and render directional sound sources, but are typically limited to simple directional gain, which relies on a simple first-order cosine function or "sound cone" (e.g., a power cosine function) and a simple high-frequency roll-off filter. These representations are not adequate to represent real-world radiation patterns, and are also not well suited to simplified / combined representations of multiple directional sound sources. Summary of the invention

[0008] Various audio processing methods are disclosed herein. Some such methods may involve encoding directional audio data. For example, some methods may involve receiving a monophonic audio signal corresponding to an audio object and a representation of a radiation pattern corresponding to the audio object. The radiation pattern may, for example, include sound levels corresponding to multiple sampling times, multiple frequency bands, and multiple directions. Some such methods may involve encoding the monophonic audio signal and encoding a source radiation pattern to determine radiation pattern metadata. The encoding of the radiation pattern may involve determining a spherical harmonic transform of the representation of the radiation pattern and compressing the spherical harmonic transform to obtain encoded radiation pattern metadata.

[0009] Some such methods may involve encoding a plurality of directional audio objects based on a cluster of audio objects. The radiation pattern may represent a centroid reflecting an average sound level value for each frequency band. In some such implementations, the plurality of directional audio objects are encoded as a single directional audio object whose directivity corresponds to a time-varying energy-weighted average of spherical harmonic coefficients of each audio object. The encoded radiation pattern metadata may indicate a location of a cluster of audio objects, which is an average of the locations of each audio object.

[0010] Some methods may involve encoding group metadata about the radiation patterns of the group of directional audio objects. In some examples, the source radiation pattern may be rescaled to the amplitude of the input radiation pattern in each frequency-based direction to determine a normalized radiation pattern. According to some implementations, compressing the spherical harmonic transform may involve singular value decomposition methods, principal component analysis, discrete cosine transforms, data-independent bases, and / or eliminating spherical harmonic coefficients of the spherical harmonic transform that are above a threshold order of spherical harmonic coefficients.

[0011] Some alternative methods may involve decoding audio data. For example, some such methods may involve receiving an encoded core audio signal, encoded radiation pattern metadata, and encoded audio object metadata, and decoding the encoded core audio signal to determine a core audio signal. Some such methods may involve decoding the encoded radiation pattern metadata to determine a decoded radiation pattern, decoding the audio object metadata, and rendering the core audio signal based on the audio object metadata and the decoded radiation pattern.

[0012] In some cases, the audio object metadata may include at least one of time-varying 3 degrees of freedom (3DoF) or 6 degrees of freedom (6DoF) source orientation information. The core audio signal may include a plurality of directional objects based on clusters of objects. The decoded radiation pattern may represent a centroid reflecting an average value for each frequency band. In some examples, the rendering may be based on applying sub-band gains based at least in part on the decoded radiation data to the decoded core audio signal. The encoded radiation pattern metadata may correspond to a set of time-varying and frequency-varying spherical harmonic coefficients.

[0013] According to some embodiments, the encoded radiation pattern metadata may include audio object type metadata. The audio object type metadata may, for example, indicate parameter directional pattern data. The parameter directional pattern data may include a cosine function, a sine function, or a cardioid function. In some instances, the audio object type metadata may indicate database directional pattern data. Decoding the encoded radiation pattern metadata to determine the decoded radiation pattern may involve querying a directional data structure that includes an audio object type and corresponding directional pattern data. In some instances, the audio object type metadata indicates dynamic directional pattern data. The dynamic directional pattern data may correspond to a set of time-varying and frequency-varying spherical harmonic coefficients. Some methods may involve receiving the dynamic directional pattern data before receiving the encoded core audio signal.

[0014] Some or all of the methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. This non-transitory media may include memory devices such as those described herein, including (but not limited to) random access memory (RAM) devices, read-only memory (ROM) devices, etc. Therefore, various innovative aspects of the subject matter described in this disclosure may be implemented in one or more non-transitory media on which software is stored. The software may, for example, include instructions for controlling at least one device to process audio data. The software may, for example, be executed by one or more components of a control system such as those disclosed herein. The software may, for example, include instructions for performing one or more of the methods disclosed herein.

[0015] At least some aspects of the present invention may be implemented via an apparatus. For example, one or more devices may be configured to at least partially perform the methods disclosed herein. In some embodiments, an apparatus may include an interface system and a control system. The interface system may include one or more network interfaces, one or more interfaces between the control system and a memory system, one or more interfaces between the control system and another device, and / or one or more external device interfaces. The control system may include at least one of a general-purpose single-chip or multi-chip processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic, or a discrete hardware component. Therefore, in some embodiments, the control system may include one or more processors and one or more non-transitory storage media operably coupled to the one or more processors.

[0016] According to some such examples, the control system may be configured to receive audio data corresponding to at least one audio object via the interface system. In some examples, the audio data may include a monophonic audio signal, audio object position metadata, audio object size metadata, and rendering parameters. Some such methods may involve determining whether the rendering parameters indicate a position mode or a directional mode, and upon determining that the rendering parameters indicate a directional mode, rendering the audio data for reproduction via at least one speaker according to the directional mode indicated by the position metadata and / or the size metadata.

[0017] In some examples, rendering the audio data may involve interpreting the audio object position metadata into audio object orientation metadata. The audio object position metadata may, for example, include x, y, z coordinate data, spherical coordinate data, or cylindrical coordinate data. In some cases, the audio object orientation metadata may include roll, pitch, and roll data.

[0018] According to some examples, rendering the audio data may involve interpreting the audio object size metadata as directional metadata corresponding to the directional pattern. In some embodiments, rendering the audio data may involve querying a data structure containing multiple directional patterns, and mapping the position metadata and / or the size metadata to one or more of the directional patterns. In some cases, the control system may be configured to receive the data structure via the interface system. In some examples, the data structure may be received before the audio data. In some embodiments, the audio data may be received in a Dolby Atmos format. The audio object position metadata may, for example, correspond to world coordinates or model coordinates.

[0019] Details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages will become apparent from the description, drawings, and claims. It should be noted that the relative dimensions of the following drawings may not be drawn to scale. Like reference numerals and designations in the various drawings generally indicate like elements. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1A is a flowchart showing blocks of an audio encoding method according to an example.

[0021] Figure 1B Process blocks that may be implemented by an encoding system for dynamically encoding per-frame directionality information for directional audio objects according to one example are described.

[0022] Figure 1C Process blocks implemented by a decoding system according to one example are described.

[0023] Figure 2A and 2B Represents the radiation pattern of an audio object in two different frequency bands.

[0024] Figure 2C is a graph showing examples of normalized and non-normalized radiation patterns according to one example.

[0025] Figure 3 An example showing a hierarchy containing audio data and various types of metadata.

[0026] Figure 4 is a flowchart showing blocks of an audio decoding method according to one example.

[0027] Figure 5A Depicts drums and cymbals.

[0028] Figure 5B Demonstrates an example of a loudspeaker system.

[0029] Figure 6 is a flowchart showing blocks of an audio decoding method according to one example.

[0030] Figure 7 An example of encoding multiple audio objects is described.

[0031] Figure 8 is a block diagram showing an example of components of an apparatus that may be configured to perform at least some of the methods disclosed herein.

[0032] In the various drawings, like reference numbers and numerals indicate like elements. DETAILED DESCRIPTION

[0033] Aspects of the present invention relate to the representation and efficient decoding of complex radiation patterns. Some such implementations may include one or more of the following:

[0034] 1. Represent the general sound radiation pattern as the time- and frequency-dependent Nth-order coefficients of a real-valued spherical harmonic function (SPH) decomposition (N>=1). This representation can also be extended to depend on the level of the playback audio signal. In contrast to the case where the directional source signal is itself a HOA-like PCM representation, the monophonic target signal can be encoded separately from its directionality information, which is represented as a set of time-dependent scalar SPH coefficients in a subband.

[0035] 2. An efficient coding scheme to reduce the bit rate required to represent this information

[0036] 3. A solution to dynamically combine radiation patterns so that a scene consisting of several radiating sound sources can be represented by an equivalent reduced number of sources while maintaining their perceptual quality at rendering time.

[0037] Aspects of the present invention relate to representing a general radiation pattern so as to supplement metadata for each monophonic audio object by a set of time / frequency related coefficients representing the directionality of the monophonic audio object projected in an Nth order spherical harmonic basis (N>=1).

[0038] The first order radiation pattern can be represented by a set of 4 scalar gain coefficients for a set of predefined frequency bands (e.g., 1 / 3 octave). The set of frequency bands may also be referred to as groups or subbands. The groups or subbands may be determined based on a short time Fourier transform (STFT) or a perceptual filter bank for a single frame of data (e.g., 512 samples as in Dolby Atmos). The resulting pattern may be rendered by evaluating the spherical harmonic decomposition at the desired direction around the object.

[0039] In general, this radiation pattern is characteristic of the source and may be constant over time. However, to represent dynamic scenes where objects rotate or change, or to ensure random access to the data, it may be beneficial to update this set of coefficients at regular time intervals. In the case of dynamic auditory scenes with moving objects, the result of the object's rotation may be encoded directly in the time-varying coefficients, without the need for explicit separate encoding of the object's orientation.

[0040] Each type of sound source has a characteristic radiation / emission pattern, which typically varies with frequency band. For example, a violin may have a very different radiation pattern than a trumpet, drum, or bell. Furthermore, a sound source such as a musical instrument may radiate differently at very soft and very strong playing levels. Thus, the radiation pattern may also be a function of not only the direction around the sound-emitting object, but also the pressure level of the audio signal it radiates, where the pressure level may also be time-varying.

[0041] Therefore, instead of simply representing the sound field at a point in space, some embodiments involve encoding audio data corresponding to the radiation pattern of an audio object so that it can be rendered from different vantage points. In some cases, the radiation pattern may be a time-varying and frequency-varying radiation pattern. In some cases, the audio data input to the encoding process may include multiple channels (e.g., 4, 6, 8, 20 or more channels) of audio data from directional microphones. Each channel may correspond to data from a microphone at a specific position in space surrounding a sound source from which the radiation pattern can be obtained. Assuming that the relative direction from each microphone to the source is known, this can be achieved by numerically fitting a set of spherical harmonic coefficients so that the resulting spherical function best matches the observed energy levels in different subbands of each input microphone signal. See, for example, the methods and systems described in conjunction with Nicolas Tsingos and Pradeep Kumar Govindaraju, Method, Systems and Apparatus for Determining Audio Representations, PCT / US2017 / 053946, which is hereby incorporated by reference herein. In other examples, the radiation pattern of an audio object may be determined via numerical simulation.

[0042] Instead of simply encoding the audio data from the directional microphone at the sample level, some implementations involve encoding the mono audio object signals with corresponding radiation pattern metadata representing the radiation pattern for at least some of the encoded audio objects. In some implementations, the radiation pattern metadata may be represented as spherical harmonic data. Some such implementations may involve a smoothing process and / or a compression / data reduction process.

[0043] Figure 1A 1 is a flowchart showing a block diagram of an audio encoding method according to one example. Method 1 may be performed, for example, by a control system including one or more processors and one or more non-transitory memory devices (e.g., as described below with reference to Figure 8 As with other disclosed methods, not all blocks of method 1 must be implemented in accordance with the control system 815 described herein. Figure 1A Additionally, alternative methods may include more or fewer boxes.

[0044] In this example, block 5 involves receiving a mono audio signal corresponding to an audio object and also receiving a representation of a radiation pattern corresponding to the audio object. According to this implementation, the radiation pattern includes sound levels corresponding to multiple sample times, multiple frequency bands, and multiple directions. According to this example, block 10 involves encoding the mono audio signal.

[0045] exist Figure 1A In the example shown in , block 15 involves encoding the source radiation pattern to determine radiation pattern metadata. According to this implementation, encoding the representation of the radiation pattern involves determining a spherical harmonic transform of the representation of the radiation pattern and compressing the spherical harmonic transform to obtain encoded radiation pattern metadata. In some implementations, the representation of the radiation pattern may be rescaled to the amplitude of the input radiation pattern in each frequency-based direction to determine a normalized radiation pattern.

[0046] In some cases, compressing the spherical harmonic transform may involve discarding some high-order spherical harmonic coefficients. Some such examples may involve eliminating spherical harmonic coefficients of the spherical harmonic transform that are above a threshold order of spherical harmonic coefficients (e.g., above the 3rd order, above the 4th order, above the 5th order, etc.).

[0047] However, some implementations may involve alternative and / or additional compression methods. According to some such implementations, the compressed spherical harmonic transform may involve singular value decomposition methods, principal component analysis, discrete cosine transforms, data-independent bases, and / or other methods.

[0048] According to some examples, method 1 may also involve encoding multiple directional audio objects as a group or "cluster" of audio objects. Some implementations may involve encoding group metadata about the radiation pattern of the group of directional audio objects. In some cases, multiple directional audio objects are encoded as a single directional audio object whose directionality corresponds to the time-varying energy-weighted average of the spherical harmonic coefficients of each audio object. In some such examples, the encoded radiation pattern metadata may represent a centroid corresponding to the average sound level value for each frequency band. For example, the encoded radiation pattern metadata (or related metadata) may indicate the position of a cluster of audio objects, which is the average of the position of each directional audio object in the cluster.

[0049] Figure 1B 1 illustrates a process block that may be implemented by the encoding system 100 for dynamically encoding per-frame directional information for a directional audio object according to one example. For example, the process may be performed by, for example, referring to Figure 8The encoding system 100 may be implemented by the control system of the control system 815 described below. The encoding system 100 may receive a mono audio signal 101, which may correspond to a mono object signal as discussed above. The mono audio signal 101 may be encoded at block 111 and provided to a serialization block 112.

[0050] At block 102, static or time-varying directional energy samples at different sound levels in a set of frequency bands relative to a reference coordinate system may be processed. The reference coordinate system may be determined in some coordinate space, such as a model coordinate space or a world coordinate space.

[0051] At block 105, a frequency-dependent rescaling of the time-varying directed energy samples from block 102 may be performed. In one example, the Figures 2A to 2C The example illustrated in performs a frequency-dependent rescaling. The normalization may be based on, for example, a rescaling of the amplitude of high frequencies relative to the direction of low frequencies.

[0052] The frequency-dependent rescaling may be renormalized based on a core audio assumed capture direction. This core audio assumed capture direction may represent a listening direction relative to a sound source. For example, this listening direction may be referred to as a viewing direction, where the viewing direction may be in a certain direction relative to a coordinate system (e.g., a forward direction or a backward direction).

[0053] At block 106, the rescaled directivity output of 105 may be projected onto a spherical harmonic function basis to obtain coefficients of the spherical harmonic function.

[0054] At block 108, the spherical coefficients of block 106 are processed based on the instantaneous sound level 107 and / or information from a rotation block 109. The instantaneous sound level 107 may be measured in a certain direction at a certain time. The information from the rotation block 109 may indicate an (optional) rotation of the time-varying source orientation 103. In one example, at block 109, the spherical coefficients may be adjusted to account for time-dependent modifications in the source orientation relative to the original recorded input data.

[0055] At block 108, a target level determination may be further performed based on the equalization of the direction determination relative to the assumed capture direction of the core audio signal.Block 108 may output a set of rotated spherical coefficients that have been equalized based on the target level determination.

[0056] At block 110, encoding of the radiation pattern may be based on a projection onto a smaller subspace of spherical coefficients associated with the source radiation pattern, resulting in encoded radiation pattern metadata. Figure 1A , at block 110, an SVD decomposition and compression algorithm may be performed on the spherical coefficients output by block 108. In one example, the SVD decomposition and compression algorithm of block 110 may be performed according to the principles described in conjunction with equations 11-13 described below.

[0057] Alternatively, block 110 may involve utilizing other methods, such as principal component analysis (PCA) and / or a data-independent basis, such as a 2D DCT, to represent the spherical harmonics. Projected into a space that facilitates lossy compression. The output of 110 may be a matrix T representing the projection of the data into a smaller subspace of the input, i.e., the encoded radiation pattern T. The encoded radiation pattern T, the encoded core mono audio signal 111, and any other object metadata 104 (e.g., x, y, z, optional source orientation, etc.) may be serialized at a serialization block 112 to output an encoded bitstream. In some examples, the radiation structure may be represented by the following bitstream syntax structure in each encoded audio frame:

[0058] Byte freqBandModePreset (e.g., wideband, octave, wideband, 1 / 3 octave, normal).

[0059] This determines the number of subbands N and the center frequency value)

[0060] Byte order (spherical harmonic order N)

[0061] Int*coefficient((N+1)*(N+1)*K value)

[0062] This syntax can cover different sets of coefficients for different pressure / intensity levels of a sound source. Alternatively, if the directionality information is available at different signal levels, and if the level of the source cannot be further determined at playback time, a single set of coefficients can be generated dynamically. For example, such coefficients can be generated by interpolating between low-level coefficients and high-level coefficients based on the time-varying level of the object audio signal at encoding time.

[0063] The input radiation pattern relative to the mono audio object signal may also be 'normalized' to a given direction, such as the principal response axis (which may be the direction from which the mono audio object signal was recorded or an average of multiple recordings), and the encoded directionality and final rendering may need to be consistent with this 'normalization'. In one example, this normalization may be specified as metadata. In general, it is desirable to encode a core audio signal that will convey a good representation of the object timbre without applying directionality information.

[0064] Directional Coding

[0065] Aspects of the present invention are directed to implementing an efficient encoding scheme for directional information, since the number of coefficients grows quadratically in the order of decomposition. An efficient encoding scheme for directional information can be implemented for final transmission delivery of auditory scenes, such as over a limited bandwidth network to an endpoint rendering device.

[0066] Assuming 16 bits are used to represent each coefficient, a 4th order spherical harmonic representation in a 1 / 3 octave band would require 25*31~=12 kbits per frame. Refreshing this information at 30 Hz would require a transmission bit rate of at least 400 kbps, which is higher than the bit rate currently required by current object-based audio codecs to transmit both audio and object metadata. In one example, the radiation pattern can be represented by:

[0067] G(θ i ,φ i ,ω)Equation (1)

[0068] In equation (1), (θ i ,φ i ), i∈{1…P} represents the discrete co-latitude angles θ∈[0,π] and azimuth angles φ∈[0,2π) relative to the sound source, P represents the total number of discrete angles, and ω represents the spectral frequency. Figure 2A and 2B Represents the radiation pattern of an audio object in two different frequency bands. For example, Figure 2A can represent the radiation pattern of an audio object in the frequency band from 100 to 300 Hz, while Figure 2B It can, for example, represent the radiation pattern of the same audio object in the frequency band from 1kHz to 2kHz. Low frequencies tend to be relatively more omnidirectional, so Figure 2A The radiation pattern shown in Figure 2B The radiation pattern shown in is relatively more circular. Figure 2A , G(θ0, φ0, ω) represents the radiation pattern in the direction of the main response axis 200 , and G(θ1, φ1, ω) represents the radiation pattern in an arbitrary direction 205 .

[0069] In some examples, the radiation pattern may be captured and determined by multiple microphones physically placed around a sound source corresponding to an audio object, while in other examples, the radiation pattern may be determined via numerical simulation. In the example of multiple microphones, the radiation pattern may be a time-varying reflection, such as a real-time recording. The radiation pattern may be captured at various frequencies, including low frequencies (e.g., <100 Hz), mid frequencies (100 Hz < and >1 kHz), and high frequencies (>10 KHz). The radiation pattern may also be referred to as a spatial representation.

[0070] In another example, the radiation pattern can be based on the direction G(θ i ,φ i ,ω) to reflect the normalization in terms of the captured radiation pattern at a certain frequency, such as (for example):

[0071]

[0072] In equation (2), G(θ0, φ0, ω) represents the radiation pattern in the direction of the main response axis. Figure 2B In one example, one can see the radiation pattern G(θ i ,φ i ,ω) and the normalized radiation pattern H(θ i ,φ i ,ω). Figure 2C is a graph showing an example of normalized and non-normalized radiation patterns according to one example. In this example, the normalized radiation pattern in the direction of the main response axis (which is Figure 2C denoted as H(θ0, φ0, ω) in FIG. 2 has substantially the same amplitude across the illustrated range of the frequency band. In this example, the normalized radiation pattern in direction 205 (in Figure 2A (shown in Figure 2C denoted as H(θ1,φ1,ω)) and the unnormalized radiation pattern (which is Figure 2C The radiation pattern is assumed to be constant for a given frequency band, but in practice it may vary with time, for example with different bowing techniques used on a string instrument.

[0073] The radiation pattern or its parametric representation may be transmitted. Preprocessing of the radiation pattern may be performed prior to its transmission. In one example, the radiation pattern or its parametric representation may be preprocessed by a computational algorithm relative to Figure 1A An example of the computational algorithm is shown. After preprocessing, the radiation pattern can be decomposed on an orthogonal spherical basis, for example, based on:

[0074]

[0075] In equation (3), H(θ i ,φ i ,ω) represents the spatial representation, and represents a spherical harmonic function representation with fewer elements than the spatial representation. H(θ i ,φ i ,ω) and The conversion between can be based on, for example, using the real fully normalized spherical harmonics:

[0076]

[0077] In equation (4), represents the associated Legendre polynomial, order m∈{-N…N}, degree n∈{0…N}, and

[0078]

[0079] Other spherical bases may also be used. Any method for performing spherical harmonic transformations on discrete data may be used. In one example, the transformation matrix may be defined by first defining To use the least squares method:

[0080]

[0081] This relates the spherical harmonic function representation to the spatial representation as

[0082]

[0083] In equation (7), The spherical harmonic representation and / or the spatial representation may be stored for further processing.

[0084] Pseudo-inverse The weighted least squares solution can be of the following form:

[0085]

[0086] For cases where the distribution of spherical samples contains a lot of missing data, the regularization solution may also be applicable. The missing data may correspond to areas or directions for which no directional samples are available (e.g., due to uneven microphone coverage). In many cases, the distribution of spatial samples is uniform enough that the identity weight matrix W produces acceptable results. It can also be generally assumed that P>>(N+1) 2 , so the spherical harmonic function is expressed as Contains fewer elements than the spatial representation H(ω), thereby producing a first-order lossy compression that smoothes the radiation pattern data.

[0087] Now consider the discrete frequency band ω k , k∈{1…K}. The matrices H(ω) can be stacked so that each frequency band is represented by a column of matrices

[0088]

[0089] That is, the spatial representation H(ω) may be determined based on frequency groupings / bands / groups. Thus, the spherical harmonic representation may be based on:

[0090]

[0091] In equation (10), represents the radiation pattern for all discrete frequencies in the spherical harmonic domain. As can be expected, Adjacent columns of are highly correlated, leading to redundancy in the representation. Some embodiments involve further decomposing by matrix factorization of the form

[0092]

[0093] Some embodiments may involve performing a singular value decomposition (SVD), where and denotes the left and right singular matrices and represents a matrix with reduced singular values ​​along its diagonal. The matrix V information can be received or stored. Alternatively, principal component analysis (PCA) and a data-independent basis (e.g., 2D DCT) can be used to convert Project into a space that aids lossy compression.

[0094] Let O = (N + 1) 2 In some examples, to achieve compression, the encoder may discard components corresponding to smaller singular values ​​by computing the product based on:

[0095] T=U∑′, equation (12)

[0096] In equation (12), represents a truncated copy of Σ. The matrix T may represent the projection of the data into a smaller subspace of the input. T represents the encoded radiation pattern data that is then transmitted for further processing. On the decoding, receiving side, in some examples, the matrix T may be received and the pair may be reconstructed based on A low-rank approximation of :

[0097]

[0098] In equation (13), represents a truncated copy of V. The matrix V can be transmitted or stored on the decoder side.

[0099] Here are three examples for transporting truncated decomposition and truncated right singular vectors:

[0100] 1. The transmitter may transmit the coded radiance T and the truncated right singular vector V′ for each object independently.

[0101] 2. Objects may be grouped, for example, by similarity measures, and U and V may be computed as representative bases for multiple objects. Thus, the coded radiance T may be transmitted per object, and U and V may be transmitted per group of objects.

[0102] 3. The left and right singular matrices U and V may be pre-computed on a large database of representative data (e.g., training data), and information about V may be stored on the receiver's side. In some such instances, only the coded radiance T may be transmitted per object. DCT is another example of a basis that may be stored on the receiver's side.

[0103] Spatial encoding of oriented objects

[0104] When encoding and transmitting complex auditory scenes including multiple objects, it may be possible to apply spatial coding techniques in which individual objects are replaced by a smaller number of representative clusters in a manner that best preserves the auditory perception of the scene. In general, replacing groups of sound sources by representative "centroids" requires computing aggregate values / averages for each metadata field. For example, the position of a cluster of sound sources may be the average of the positions of each source. By using the spherical harmonic decomposition as outlined above to represent the radiation pattern of each source (e.g., with reference to equations 1 to 12), the coefficient groups in each subband for each source may be linearly combined to construct an average radiation pattern for a cluster of sources. By computing a loudness or energy-weighted average of the spherical harmonic coefficients that vary over time, a time-varying perceptually optimized representation that better preserves the original scene may be constructed.

[0105] Figure 1C A block showing a process that can be implemented by a decoding system according to one example. For example, Figure 1C The blocks shown in FIG. 1 may be controlled by a control system (such as the one described below with reference to FIG. 1 ) of a decoding device including one or more processors and one or more non-transitory memory devices. Figure 8 The system 815 described in the foregoing may be implemented. At block 150, metadata and an encoded core mono audio signal may be received and deserialized. The deserialized information may include object metadata 151, the encoded core audio signal, and the encoded spherical coefficients. At block 152, the encoded core audio signal may be decoded. At block 153, the encoded spherical coefficients may be decoded. The encoded radiation pattern information may include an encoded radiation pattern T and / or a matrix V. The matrix V will depend on the matrix used to convert The method of projecting into space. If Figure 1B If the SVD algorithm is used at box 110 of , the matrix V can be received or stored by the decoding system.

[0106] Object metadata 151 may include information about the relative direction of the source to the listener. In one example, metadata 151 may include information about the distance and direction of the listener and one or more object distances and directions relative to 6DoF space. For example, metadata 151 may include information about the relative rotation, distance, and direction of the source in 6DoF space. In the example of multiple objects in a cluster, the metadata field may reflect information about a representative "center of mass" that reflects the aggregate value / average of the cluster of objects.

[0107] The renderer 154 may then render the decoded core audio signal and the decoded spherical harmonic function coefficients. In one example, the renderer 154 may render the decoded core audio signal and the decoded spherical harmonic function coefficients based on the object metadata 151. The renderer 154 may determine the subband gains for the spherical coefficients of the radiation pattern based on information from the metadata 151 (e.g., source to listener relative direction). The renderer 154 may then render the core audio object signal based on the determined subband gains, source and / or listener posture information (e.g., x, y, z, roll, pitch, roll) 155 corresponding to the decoded radiation pattern. The listener posture information may correspond to the position and viewing direction of the user in the 6DoF space. The listener posture information may be received from a source local to the VR playback system, such as an optical tracking device. The source posture information corresponds to the position and orientation of the sound-emitting object in the space. It can also be inferred from the local tracking system, for example, whether the user's hands are tracked and interactively manipulated virtual sound-emitting objects, or whether tracked physical props / proxy objects are used.

[0108] Figure 3 An example of a hierarchy containing audio data and various types of metadata is shown. As with other diagrams provided herein, this is provided by way of example only. Figure 3 The amount and type of audio data and metadata shown in the . Some encoders can provide Figure 3 The complete set of audio data and metadata (data set 345) shown, while other encoders may only provide Figure 3 , for example, only data group 315 , only data group 325 , or only data group 335 .

[0109] In this example, the audio data includes a mono audio signal 301. The mono audio signal 301 is an example of an audio signal sometimes referred to herein as a “core audio signal.” However, in some examples, the core audio signal may include audio signals corresponding to multiple audio objects included in a cluster.

[0110] In this example, the audio object position metadata 305 is expressed as Cartesian coordinates. However, in alternative examples, the audio object position metadata 305 may be expressed via other types of coordinates, such as spherical or polar coordinates. Thus, the audio object position metadata 305 may include three degrees of freedom (3DoF) position information. According to this example, the audio object metadata includes audio object size metadata 310. In alternative examples, the audio object metadata may include one or more other types of audio object metadata.

[0111] In this embodiment, the data set 315 includes a mono audio signal 301, audio object position metadata 305, and audio object size metadata 310. The data set 315 may be, for example, Dolby Atmos TM Audio data format provided.

[0112] In this example, data set 315 also includes an optional rendering parameter R. According to some disclosed implementations, optional rendering parameter R may indicate whether at least some of the audio object metadata of data set 315 should be interpreted in its "normal" sense (e.g., as position or size metadata), or should be interpreted as directional metadata. In some disclosed implementations, the "normal" mode may be referred to herein as a "position mode," and the alternative mode may be referred to herein as a "directional mode." hereinafter, with reference to Figures 5A to 6 Describe some examples.

[0113] According to this example, the orientation metadata 320 includes angular information for expressing the roll, pitch, and roll of the audio object. In this example, the orientation metadata 320 indicates the roll, pitch, and roll as ф, θ, and ψ. The data set 325 includes sufficient information to orient the audio object for six degrees of freedom (6DoF) applications.

[0114] In this example, data group 335 includes audio object type metadata 330. In some implementations, audio object type metadata 330 may be used to indicate corresponding radiation pattern metadata. Encoded radiation pattern metadata may be used to determine a decoded radiation pattern (e.g., by a decoder or a device receiving audio data from a decoder). In some examples, audio object type metadata 330 may substantially indicate "I am a trumpet," "I am a violin," etc. In some examples, a decoding device may access a database of audio object types and corresponding directional patterns. According to some examples, the database may be provided with the encoded audio data, or provided prior to transmission of the audio data. This audio object type metadata 330 may be referred to herein as "database directional pattern data."

[0115] According to some examples, the audio object type metadata may indicate parametric directional pattern data. In some examples, the audio object type metadata 330 may indicate a directional pattern corresponding to a cosine function of a specified power, may indicate a cardioid function, and the like.

[0116] In some examples, the audio object type metadata 330 may indicate that the radiation pattern corresponds to a set of spherical harmonic coefficients. For example, the audio object type metadata 330 may indicate that spherical harmonic coefficients 340 are being provided in data set 345. In some such examples, the spherical harmonic coefficients 340 may be a time-varying and / or frequency-varying set of spherical harmonic coefficients, e.g., as described above. Figure 3, this information may require the largest amount of data. Therefore, in some such examples, the spherical harmonic coefficients 340 may be provided separately from the mono audio signal 301 and the corresponding audio object metadata. For example, the spherical harmonic coefficients 340 may be provided at the beginning of the transmission of the audio data, before initiating a real-time operation (e.g., a real-time rendering operation for a game, movie, music performance, etc.).

[0117] According to some embodiments, a device on the decoder side (e.g., a device that provides audio to a reproduction system) may determine the capabilities of the reproduction system and provide the directional information according to the capabilities. For example, in some such embodiments, even if the entire data set 345 is provided to the decoder, only a usable portion of the directional information may be provided to the reproduction system. In some examples, the decoding device may determine which type(s) of directional information to use according to the capabilities of the decoding device.

[0118] Figure 4 4 is a flowchart showing a block diagram of an audio decoding method according to one example. The method 400 may be performed, for example, by a control system of a decoding device including one or more processors and one or more non-transitory memory devices (e.g., as described below with reference to Figure 8 As with other disclosed methods, not all blocks of method 400 must be implemented in accordance with the control system 815 described herein. Figure 4 Additionally, alternative methods may include more or fewer boxes.

[0119] In this example, block 405 involves receiving an encoded core audio signal, encoded radiation pattern metadata, and encoded audio object metadata. The encoded radiation pattern metadata may include audio object type metadata. The encoded core audio signal may, for example, include a mono audio signal. In some examples, the audio object metadata may include 3 DoF position information, 6 DoF position and source orientation information, audio object size metadata, etc. In some cases, the audio object metadata may be time-varying.

[0120] In this example, block 410 involves decoding the encoded core audio signal to determine the core audio signal. Here, block 415 involves decoding the encoded radiation pattern metadata to determine the decoded radiation pattern. In this example, block 420 involves decoding at least some of the other encoded audio object metadata. Here, block 430 involves rendering the core audio signal based on the audio object metadata (e.g., audio object position, orientation, and / or size metadata) and the decoded radiation pattern.

[0121] Block 415 may involve various types of operations depending on the particular implementation. In some cases, the audio object type metadata may indicate database directional pattern data. Decoding the encoded radiation pattern metadata to determine the decoded radiation pattern may involve querying a directional data structure that includes the audio object type and the corresponding directional pattern data. In some examples, the audio object type metadata may indicate parametric directional pattern data, such as directional pattern data corresponding to a cosine function, a sine function, or a cardioid function.

[0122] According to some implementations, the audio object type metadata may indicate dynamic directional pattern data, such as a time-varying and / or frequency-varying set of spherical harmonic coefficients. Some such implementations may involve receiving the dynamic directional pattern data prior to receiving the encoded core audio signal.

[0123] In some cases, the core audio signal received in box 405 may include audio signals corresponding to multiple audio objects included in the cluster. According to some such examples, the core audio signal may be based on a cluster of audio objects that may include multiple directional audio objects. The decoded radiation pattern determined in box 415 may correspond to the centroid of the cluster and may represent the average value of each frequency band of each of the multiple directional audio objects. The rendering process of box 430 may involve applying subband gains to the decoded core audio signal based at least in part on the decoded radiation data. In some examples, after the core audio signal is decoded and the directional processing is applied to the core audio signal, the signal may be further virtualized to its expected positioning relative to the listener position using audio object position metadata and a known rendering process (e.g., binaural rendering through headphones, rendering using speakers of a reproduction environment, etc.).

[0124] As mentioned above Figure 3 As discussed, in some implementations, audio data may be represented by rendering parameters (in Figure 3 The rendering parameters may indicate whether at least some audio object metadata (e.g., Dolby Atmos metadata) should be interpreted in a normal manner (e.g., as position or size metadata) or as directional metadata. The normal mode may be referred to as "position mode," and the alternative mode may be referred to herein as "directional mode." Thus, in some examples, the rendering parameters may indicate whether at least some audio object metadata is to be interpreted as directional relative to a speaker or as positional relative to a room or other reproduction environment. Such implementations are particularly useful for directional rendering using smart speakers with multiple drivers, for example, as described below.

[0125] Figure 5AIn this example, a drum cymbal 505 is shown emitting a sound having a directional pattern 510 with a generally vertical primary response axis 515. Directivity pattern 510 itself is also primarily vertical, and has some spread from primary response axis 515.

[0126] Figure 5B An example of a speaker system is shown. In this example, the speaker system 525 includes multiple speakers / transducers configured to emit sound in various directions, including upward. In some cases, the top-most speakers can be used in a conventional Dolby Atmos manner ("position mode") to render positions, such as causing sounds to reflect off the ceiling to simulate height / ceiling speakers (z=1). In some such cases, the corresponding Dolby Atmos rendering may include additional height virtualization processing that enhances the perception of audio objects having specific positions.

[0127] In other use cases, the same upward-firing speakers can be operated in "directional mode", for example to simulate drums, symbols, or sounds similar to Figure 5A Another audio object with the directional pattern of the directional pattern 510 shown in . Some speaker systems 525 may be capable of beamforming, which can help construct the desired directional pattern. In some examples, virtualization processing will not be involved in order to reduce the perception of audio objects with specific locations.

[0128] Figure 6 600 may be performed, for example, by a control system of a decoding device including one or more processors and one or more non-transitory memory devices (e.g., as described below with reference to Figure 8 As with other disclosed methods, not all blocks of method 600 must be implemented in accordance with the control system 815 described herein. Figure 6 Additionally, alternative methods may include more or fewer boxes.

[0129] In this example, block 605 involves receiving audio data corresponding to at least one audio object, the audio data comprising a mono audio signal, audio object position metadata, audio object size metadata, and rendering parameters. In this implementation, block 605 involves receiving, via an interface system (e.g., a decoding device) Figure 8 In some cases, Dolby Atmos TM The audio data is received in a format. Depending on the particular implementation, the audio object position metadata may correspond to world coordinates or model coordinates.

[0130] In this example, block 610 involves determining whether the rendering parameters indicate a position mode or a directional mode. Figure 6 In the example shown in FIG. 6 , if it is determined that the rendering parameters indicate a directional mode, then in block 615, the audio data is rendered for reproduction (e.g., via at least one speaker, via headphones, etc.) according to the directional mode indicated by at least one of the position metadata or the size metadata. For example, the directional mode may be similar to Figure 5A The directional pattern shown in .

[0131] In some examples, rendering the audio data may involve interpreting the audio object position metadata into audio object orientation metadata. The audio object position metadata may be Cartesian / x, y, z coordinate data, spherical coordinate data, or cylindrical coordinate data. The audio object orientation metadata may be roll, pitch, and roll metadata.

[0132] According to some embodiments, rendering the audio data may involve interpreting the audio object size metadata into directional metadata corresponding to the directional pattern. In some such instances, rendering the audio data may involve querying a data structure containing a plurality of directional patterns and mapping at least one of the position metadata or the size metadata to one or more of the directional patterns. Some such embodiments may involve receiving the data structure via an interface system. According to some such embodiments, the data structure may be received prior to the audio data.

[0133] Figure 7 An example of encoding multiple audio objects is described. In one example, object 1-n information 701, 702, 703, etc. can be encoded. In one example, a representative cluster for audio objects 701 to 703 can be determined at box 710. In one example, the group of sound sources can be aggregated and represented by a representative "center of mass" involving calculating an aggregate value / average for a metadata field. For example, the location of the cluster of sound sources can be the average of the location of each source. At box 720, the radiation pattern for the representative cluster can be encoded. In some examples, the above reference can be used to obtain the radiation pattern of the representative cluster. Figure 1A or Figure 1B The principle described is used to encode the radiation pattern for a cluster.

[0134] Figure 8 is a block diagram showing an example of components of an apparatus that can be configured to perform at least some of the methods disclosed herein. For example, apparatus 805 can be configured to perform the above reference Figures 1A to 1C, 4, 6 and / or 7. In some examples, the device 805 may be or may include a personal computer, a desktop computer, or other local device configured to provide audio processing. In some examples, the device 805 may be or may include a server. According to some examples, the device 805 may be a client device configured to communicate with the server via a network interface. The components of the device 805 may be implemented via hardware, via software stored on non-transitory media, via firmware, and / or by a combination thereof. Figure 8 The types and numbers of components shown in the other figures disclosed herein are shown by way of example only. Alternative implementations may include more, fewer and / or different components.

[0135] In this example, the device 805 includes an interface system 810 and a control system 815. The interface system 810 may include one or more network interfaces, one or more interfaces between the control system 815 and the memory system, and / or one or more external device interfaces (e.g., one or more universal serial bus (USB) interfaces). In some embodiments, the interface system 810 may include a user interface system. The user interface system may be configured to receive input from a user. In some embodiments, the user interface system may be configured to provide feedback to the user. For example, the user interface system may include one or more displays with corresponding touch and / or gesture detection systems. In some examples, the user interface system may include one or more microphones and / or speakers. According to some examples, the user interface system may include equipment for providing tactile feedback, such as motors, vibrators, etc. The control system 815 may, for example, include a general-purpose single-chip or multi-chip processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic and / or a discrete hardware component.

[0136] In some examples, the apparatus 805 may be implemented in a single device. However, in some embodiments, the apparatus 805 may be implemented in more than one device. In some such embodiments, the functionality of the control system 815 may be included in more than one device. In some examples, the apparatus 805 may be a component of another device.

[0137] Various example embodiments of the present invention may be implemented in hardware or dedicated circuits, software, logic or any combination thereof. Some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software executable by a controller, microprocessor or other computing device. In general, the present invention should be understood to also encompass devices suitable for performing the methods described above, such as devices having a memory and a processor coupled to the memory (spatial renderer), wherein the processor is configured to execute instructions and perform methods according to embodiments of the present invention.

[0138] Although various aspects of example embodiments of the present invention are illustrated and described as a block diagram, flow diagram, or using some other graphical representation, it will be appreciated that the blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controller, or other computing devices, or some combination thereof, as non-limiting examples.

[0139] In addition, the various blocks shown in the flowchart may be viewed as method steps, and / or operations resulting from the operation of computer program code, and / or a plurality of coupled logic circuit elements constructed to implement the associated functions. For example, embodiments of the present invention include a computer program product comprising a computer program tangibly embodied on a machine-readable medium, wherein the computer program contains program code configured to implement the method as described above.

[0140] In the context of the present invention, a machine-readable medium may be any tangible medium that may contain or store a program used by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media would include an electrical connection with one or more wires, a portable computer floppy disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable reader read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0141] The computer program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These computer program codes can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code causes the function / operation specified in the flow chart and / or block diagram to be implemented when the processor of the computer or other programmable data processing device executes. The program code can be executed completely on the computer, partially on the computer as an independent software package, partially on the computer and partially on a remote computer, or completely on a remote computer or server.

[0142] In addition, although the operations are depicted in a particular order, this should not be construed as requiring that such operations be performed in the particular order shown or in a sequential order, or that all of the described operations be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Similarly, although the above discussion contains several specific implementation details, these details should not be interpreted as limitations on the scope of any invention or possible claims, but should be regarded as descriptions of features that may be specific to a particular embodiment of a particular invention. Certain features described in this specification in the context of a separate embodiment may also be implemented in combination in a single embodiment. On the contrary, the various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable sub-combination.

[0143] It should be noted that the description and drawings illustrate only the principles of the proposed method and apparatus. Therefore, it will be appreciated that those skilled in the art will be able to design various arrangements that, although not explicitly described or shown herein, embody the principles of the present invention and are included in its spirit and scope. In addition, all examples listed herein are primarily intended to be used explicitly for teaching purposes only to help the reader understand the principles of the proposed method and apparatus and the concepts contributed by the inventors to further develop the field, and should be interpreted as having no limitation on such specifically described examples and conditions. In addition, all statements herein describing the principles, aspects and embodiments of the present invention and specific examples thereof are intended to cover their equivalents.

Claims

1. A method for decoding audio data, comprising: Receiving an encoded core audio signal, encoded radiation pattern metadata, and encoded audio object metadata, wherein the audio object metadata includes 6DoF source orientation information; decoding the encoded core audio signal to determine a core audio signal; decoding the encoded radiation pattern metadata to determine a decoded radiation pattern; Decoding the audio object metadata; and The core audio signal is rendered based on the audio object metadata and the decoded radiance pattern. 2 . The method of claim 1 , wherein the core audio signal comprises a plurality of directional objects based on a cluster of objects, and wherein the decoded radiation pattern represents a centroid reflecting an average value for each frequency band.

3. The method of claim 1, wherein the encoded radiation pattern metadata corresponds to a time-varying and frequency-varying set of spherical harmonic coefficients. The method of claim 1 , wherein the encoded radiation pattern metadata comprises audio object type metadata.

5. The method of claim 4, wherein the audio object type metadata indicates parametric directional pattern data, and wherein the parametric directional pattern data comprises one or more functions selected from a list of functions consisting of a cosine function, a sine function, or a cardioid function.

6. The method of claim 4, wherein the audio object type metadata indicates dynamic directional pattern data, and wherein the dynamic directional pattern data corresponds to a time-varying and frequency-varying set of spherical harmonic coefficients. 7 . The method of claim 6 , further comprising receiving the dynamic directional pattern data prior to receiving the encoded core audio signal.

8. The method of claim 1, wherein the rendering is based on applying a subband gain based at least in part on the decoded radiation pattern to the decoded core audio signal.

9. The method of claim 4, wherein the audio object type metadata indicates database directional pattern data, and wherein decoding the encoded radiation pattern metadata to determine the decoded radiation pattern comprises querying a directional data structure that includes audio object types and corresponding directional pattern data.

10. A non-transitory computer readable medium having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to perform the method of claim 1.

11. An audio decoding device, comprising: Interface system; and A control system configured to: receiving, via the interface system, audio data corresponding to at least one audio object, the audio data comprising a mono audio signal, audio object position metadata, audio object size metadata, and rendering parameters, wherein the audio object position metadata comprises 6DoF source orientation information; determining whether the rendering parameters indicate a position mode or a directional mode; and after determining that the rendering parameters indicate a directional mode, rendering the audio data for reproduction via at least one speaker according to the directional mode indicated by at least one of the audio object position metadata or the audio object size metadata.

12. A method comprising: receiving first audio object information and second audio object information; determining a representative cluster for the first audio object information and the second audio object information; and The radiation pattern for the representative cluster is encoded based at least in part on performing a singular value decomposition.

13. A speaker system, comprising: First speaker; a second speaker disposed above the first speaker; and Wherein the second speaker is configured to operate in a positional mode to render a position, and wherein the positional mode involves a high degree of virtualization processing.

Images