Method and apparatus for encoding and decoding a series of frames of an ambisonic representation of a two-dimensional or three-dimensional sound field.

Abstract

Representing spatial audio scenes using higher-order ambisonics (HOA) technology typically requires a large number of coefficients per time unit. This data rate is too high for most practical applications that require real-time transmission of audio signals. [Solution] According to the present invention, compression is performed in the spatial domain instead of the HOA domain. (N+1) 2 The number of input HOA coefficients in the spatial domain is (N+1) 2 It is converted into individual equivalent signals, resulting in (N+1) 2 Individual time-domain signals are input into a parallel bank of perceptual codecs. On the decoder side, each spatial-domain signal is decoded, the spatial-domain coefficients are converted back to the original HOA domain, and the original HOA representation is restored.

Description

[Technical Field]

[0001] The present invention relates to a method and apparatus for encoding and decoding a series of frames of a higher-order ambisonic representation of a two-dimensional or three-dimensional sound field. [Background technology]

[0002] Ambisonics generally uses specific coefficients based on spherical harmonics to provide a sound field description that is independent of any particular speaker or microphone setup. This leads to a description that does not require information about speaker positions when recording or generating the sound field of a composite scene. The accuracy of reproduction in an ambisonics system can be modified by its order N. This order determines the number of audio information channels required to describe the sound field for a 3D system, as this depends on the number of spherical harmonic bases. The number of coefficients or channels O is O=(N+1). 2 That is the case.

[0003] Representing complex spatial audio scenes using higher-order Ambisonics (HOA) techniques (i.e., order 2 or higher) typically requires a large number of coefficients per time. Each coefficient should have considerable resolution, typically 24 bits / coefficient or more. Therefore, the data rate required to transmit audio scenes in raw HOA format is high. As an example, a third-order HOA signal, for example, recorded using an EigenMike recording system, would require (3+1) 2 This requires a bandwidth of 16.15 Mbit / s (megabits per second) due to the coefficient multiplied by 44100Hz × 24 bits / coefficient. Currently, this data rate is too high for most practical applications requiring real-time transmission of audio signals. Therefore, compression techniques are desired for practically meaningful HOA-related audio processing systems.

[0004] Higher-order ambisonics is a mathematical paradigm that allows for the capture, manipulation, and storage of audio scenes. The sound field is approximated by a Fourier-Bessel series at and around a certain reference point in space. Because the HOA coefficients have this particular underlying mathematics, specific compression techniques must be applied to achieve optimal encoding efficiency. Both redundancy and psychoacoustic aspects should be taken into account, and it can be expected to function differently for complex spatial audio scenes than for conventional mono or multi-channel signals. One specific difference from established audio formats is that all "channels" in the HOA representation are calculated with the same reference position in space. Therefore, considerable coherence can be expected between the HOA coefficients, at least for audio scenes with few dominant sound objects.

[0005] There are very few publicly available techniques for lossy compression of HOA signals. Most of these cannot be categorized as perceptual coding because they typically do not utilize psychoacoustic models to control the compression. In contrast, some existing methods use the decomposition of the audio scene into the parameters of an underlying model.

[0006] <Early approaches for primary or tertiary ambisonic transmission> Ambisonics theory has been used for audio production and consumption since the 1960s, but to this day, its use has been largely limited to primary or secondary content. Several distribution formats have been used, particularly the following:

[0007] • B-format: This format is a standard, professional raw signal format used for content exchange among researchers, producers, and enthusiasts. Typically, it relates to first-order ambisonics with specific coefficient normalizations, but standards up to third order also exist.

[0008] In recent higher-order variations of the B format, modified normalization schemes and special weighting rules, such as SN3D (Furse-Malham, also known as FuMa or FMH sets), typically result in downscaling of the amplitude of parts of the ambisonic coefficient data. On the receiving end, the reverse upscaling operation is performed by a table lookup before decoding.

[0009] • UHJ format (also known as C format): This is a hierarchical encoded signal format applicable to delivering primary ambisonic content to consumers via existing mono or 2-channel stereo paths. With two channels, left and right, a full horizontal surround representation of the audio scene is achievable, although not with full horizontal resolution. An optional third channel improves spatial resolution in the horizontal plane, and an optional fourth channel adds a height dimension.

[0010] • G Format: This format was created to make content produced in the Ambisonics format available to anyone without requiring a specific Ambisonics decoder for home use. Decoding to a standard 5-channel round setup is already performed by the producer. Because the decoding operation is not standardized, reliable reconstruction of the original B Format Ambisonics content is not possible.

[0011] • D format: This format refers to a set of decoded speaker signals produced by any ambisonics decoder. The decoded signals depend on the specific speaker geometry and the individual circumstances of the decoder design. The G format is a subset of the D format definition as it refers to a specific 5-channel surround setup.

[0012] None of the approaches described above were designed with compression in mind. Some of the formats mentioned above are adapted to utilize existing, low-capacity transmission paths (e.g., stereo links), and therefore implicitly reduce the data rate for transmission. However, the downmixed signal lacks a significant portion of the original input signal information. Thus, the flexibility and universality of the ambisonics approach are lost.

[0013] Directional audio coding Around 2005, the DirAC (directional audio coding) technique was developed. This is based on scene analysis of a target to decompose a scene into a single dominant sound object and ambient sounds per time and frequency. Scene analysis is based on an evaluation of the instantaneous intensity vector of the sound field. The two parts of the scene are transmitted along with positional information about where the direct sound is coming from. At the receiver, a single dominant sound source per time-frequency pane is reproduced using vector-based amplitude panning (VBAP). Furthermore, decorrelated ambient sounds are generated as side information according to the transmitted ratio. The DirAC process is illustrated in Figure 1, where the input signal has a B-format. DirAC can be interpreted as a specific method of parametric coding using a single-source + ambient signal model. The transmission quality strongly depends on whether the model's assumptions hold true for that particular compressed audio scene. Furthermore, any misdetection of the direct sound and / or ambient sounds during the sound analysis stage can affect the quality of playback of the decoded audio scene. To date, DirAC has only described primary ambisonics content.

[0014] <Direct compression of HOA coefficients> In the late 2000s, perceptual and reversible (lossless) compression of HOA signals was proposed.

[0015] ·For reversible symbolization, as described in Non-Patent Documents 1 and 2, the cross-correlation between different ambisonic coefficients is utilized to reduce the redundancy of the HOA signal. Backward adaptive prediction is used to predict the current coefficient of a particular order from the weighted combination of the preceding coefficients up to the order of the coefficient to be encoded. Groups of coefficients that are expected to exhibit strong cross-correlation are found by evaluating the characteristics of real-world content. Compression acts in a hierarchical manner. The neighborhood analyzed for potential cross-correlation of a coefficient includes only the coefficients up to the same order at the same time and at preceding times. Therefore, compression is scalable at the bitstream level.

[0016] ·Perceptual coding is described in Non-Patent Document 3 and the above-mentioned Non-Patent Document 1. Existing MPEG AAC compression techniques are used to encode the individual channels (i.e., coefficients) of the HOA B format representation. By adjusting the bit allocation depending on the order of the channel, a non-uniform spatial noise distribution is obtained. In particular, by allocating more bits to the low-order channels and fewer bits to the high-order channels, excellent accuracy is obtained near the reference point. Also, as the distance from the origin increases, the effective quantization noise increases.

[0017] Figure 2 shows the principle of such direct encoding and decoding of a B format audio signal. Here, the upper path shows the compression by Hellerud et al. of the above non-patent document, and the lower path shows the compression to a normal D format signal. In either case, the decoded receiver output signal has the D format.

[0018] The problem of directly searching for redundancy and irrelevancy in the HOA domain is that any spatial information is generally "smeared" across several HOA coefficients. In other words, information that is well-localized and concentrated in the spatial domain spreads out around it. As a result, it is extremely difficult to perform consistent noise assignment that conforms to the acoustic-psychological masking constraints in a reliable manner. Furthermore, important information is incorporated in a differential manner in the HOA domain, and subtle differences in large-scale coefficients can have a strong influence in the spatial domain. Therefore, a high data rate may be required to retain the details of such differences.

[0019] 〈Spatial squeezing〉 More recently, B. Cheng, Ch. Ritz, and I. Burnett developed the "spatial squeezing" technique: Non-Patent Documents 4 to 6.

[0020] Audio scene analysis is performed to decompose the sound field into the selection of the most dominant sound objects for each time / frequency bin. Then, a 2-channel stereo downmix containing these dominant sound objects is generated at new positions intermediate between the left and right channel positions. Since the same analysis can be done for stereo signals, the operation can be partially reversed by remapping the objects detected in the 2-channel stereo downmix to a full 360° sound field.

[0021] Figure 3 depicts the principle of spatial squeezing. Figure 4 shows the related encoding process.

[0022] This concept is strongly related to DirAC. This is because it relies on the same kind of audio scene analysis. However, in contrast to DirAC, the downmix always generates two channels and it is not necessary to transmit side information about the positions of the dominant sound objects.

[0023] Although psychoacoustic principles are not explicitly used, this method leverages the premise that a reasonable level of quality can already be achieved by transmitting only the most prominent sound objects in terms of time-frequency tiles. In this respect, it is a stronger counterpart to the premise of DirAC. Similar to DirAC, any error in the parameterization of the audio scene leads to artifacts in the decoded audio scene. Furthermore, it is difficult to predict the effect of any perceptual coding of the 2-channel stereo downmix signal on the quality of the decoded audio scene. Due to the general structure of this spatial squeeze, it cannot be applied to three-dimensional audio signals (i.e., signals with height dimension). Also, obviously, it does not work for ambisonic orders greater than 1.

[0024] <Ambisonics format and mixed-degree representation> Non-patent document 7 proposes constraining spatial sound information to a subspace of the entire sphere, for example, to cover only the upper hemisphere or an even smaller portion of the sphere. Ultimately, a complete scene can consist of several such constrained "sectors" on the sphere. These sectors are rotated to specific positions to collect the target audio scene. This creates a kind of mixed-order composition of complex audio scenes. Perceptual coding is not mentioned.

[0025] Parametric coding The “classic” approach to describing and transmitting content intended for playback in a wave-field synthesis (WFS) system is through parametric coding of individual sound objects in an audio scene. Each sound object consists of an audio stream (mono, stereo, or something else) with metadata about its role within the full audio scene, most importantly, its position. This object-oriented paradigm was refined for WFS playback during the CARROUSO process in Europe. See Non-Patent Literature 8.

[0026] One example of compressing individual sound objects independently is the integrated encoding of multiple objects in a downmix scenario, as described in Non-Patent Document 9. In this document, simple psychoacoustic cues are used to generate a meaningful downmix signal. From this downmix signal, the receiver can decode a multi-object scene using secondary information. Rendering of objects within the audio scene to their local speaker setup is also performed at the receiver.

[0027] In object-oriented formats, recording is particularly sophisticated. Theoretically, it requires a completely "dry" recording of individual sound objects, that is, a recording that captures only the direct sound emitted by the sound object. This approach has two drawbacks: firstly, dry capture is difficult in natural "live" recordings because there is considerable crosstalk between microphone signals. Secondly, the audio scene collected from a dry recording lacks naturalness and the "atmosphere" of the room where the recording took place.

[0028] <Parametric coding and ambisonics> Some researchers have proposed combining ambisonic signals with several discrete sound objects. The motivation is to capture ambient sounds and sound objects that cannot be well localized through ambisonic representations and to add several discrete, well-localized sound objects via a parametric approach. For the object-oriented portion of the scene, the same encoding mechanism as for the purely parametric representation (see previous section) is used. That is, these individual sound objects typically appear with a mono sound track as well as information about their position and potential motion. See the introduction of ambisonic playback into the MPEG-4 AudioBIFS standard. In that standard, it is up to the audio scene creator how to transmit the raw ambisonic and object streams to the (AudioBIFS) rendering engine. This means that any audio codec defined in MPEG-4 can be used to directly encode ambisonic coefficients.

[0029] <Wave field encoding> Instead of using an object-oriented approach, wave field coding transmits pre-rendered speaker signals from a wave field synthesis (WFS) system. The encoder performs all the rendering to a specific set of speakers. Multidimensional space-time-to-frequency conversion is performed on windowed, quasi-linear segments of the speaker curve. Frequency coefficients (for both time and spatial frequencies) are encoded using some psychoacoustic model.

[0030] In addition to normal time-frequency masking, spatial frequency masking can also be applied. That is, the masking phenomenon is assumed to be a function of spatial frequency. On the decoder side, the encoded speaker channels are decompressed and played back.

[0031] Figure 5 shows the principle of wave-field coding using a pair of microphones in the upper part and a pair of speakers in the lower part. Figure 6 shows the encoding process based on Non-Patent Literature 10. Published experiments on perceptual wave-field coding have shown that the space-time to frequency conversion saves approximately 15% of the data rate compared to separate perceptual compression of the rendered speaker channels for a two-source signal model. Nevertheless, this process does not have the compression efficiency obtained by the object-oriented paradigm. This is probably because it does not capture the sophisticated cross-correlation characteristics between speaker channels, as sound waves arrive at each speaker at different times. A further drawback is the tight tie to the specific speaker layout of the target system.

[0032] <Universal spatial clues> Starting with classical multi-channel compression, the concept of a universal audio codec capable of handling various speaker scenarios has also been considered. In contrast to MP3 surround or MPEG surround, which have fixed channel assignments and relationships, the representation of spatial cues is designed to be independent of specific input speaker configurations. See Non-Patent Documents 11, 12, and 13.

[0033] Following the frequency domain transformation of discrete input channel signals, principal component analysis is performed on each time-frequency tile to distinguish the main sound from the environmental components. The result is the derivation of direction vectors to positions on a circle with a unit radius centered on the listener, using Gerzon vectors for scene analysis. Figure 7 depicts the corresponding system for spatial audio coding with downmixing and transmission of spatial cues. The (stereo) downmix signal consists of separated signal components and is transmitted along with metadata about object locations. The decoder reconstructs the main sound and some environmental components from the downmix signal and sub-information. The main sound is then panned to local speaker positions. This can be interpreted as a multi-channel version of the DirAC process described above, because the transmitted information is very similar. [Prior art documents] [Non-patent literature]

[0034] [Non-Patent Document 1] E. Hellerud, A. Solvang, UP Svensson, "Spatial Redundancy in Higher Order Ambisonics and Its Use for Low Delay Lossless Compression", Proc. of IEEE Intl. Conf. on Acoustics, Speech, and Signal Processing (ICASSP), April 2009, Taipei, Taiwan [Non-Patent Document 2] E. Hellerud, UP Svesson, "Lossless Compression of Spherical Microphone Array Recordings", Proc. of 126th AES Convention, Paper 7668, May 2009, Munich, Germany [Non-Patent Document 3] T. Hirvonen, J. Ahonen, V. Pulkki, "Perceptual Compression Methods for Metadata in Directional Audio Coding Applied to Audiovisual Teleconference", Proc. of 126th AES Convention, Paper 7706, May 2009, Munich, Germany [Non-Patent Document 4] B. Cheng, Ch. Ritz, I. Burnett, "Spatial Audio Coding by Squeezing: Analysis and Application to Compressing Multiple Soundfields", Proc. of European Signal Processing Conf. (EUSIPCO), 2009 [Non-Patent Document 5] B. Cheng, Ch. Ritz, I. Burnett, "A Spatial Squeezing Approach to Ambisonic Audio Compression", Proc. of IEEE Intl. Conf. on Acoustics, Speech, and Signal Processing (ICASSP), April 2008 [Non-Patent Document 6] B. Cheng, Ch. Ritz, I. Burnett, "Principles and Analysis of the Squeezing Approach to Low Bit Rate Spatial Audio Coding", Proc. of IEEE Intl. Conf. on Acoustics, Speech, and Signal Processing (ICASSP), April 2007 [Non-Patent Document 7] F. Zotter, H. Pomberger, M. Noisternig, "Ambisonic Decoding with and without Mode-Matching: A Case Study Using the Hemisphere", Proc. of 2nd Ambisonics Symposium, May 2010, Paris, France [Non-Patent Document 8] S. Brix, Th. Sporer, J. Plogsties, "CARROUSO-An European Approach to 3D-Audio," Proc. of 110th AES Convention, Paper 5314, May 2001, Amsterdam, The Netherlands [Non-Patent Document 9] Ch. Faller, "Parametric Joint-Coding of Audio Sources", Proc. of 120th AES Convention, Paper 6752, May 2006, Paris, France [Non-Patent Document 10] F. Pinto, M. Vetterli, "Wave Field Coding in the Spacetime Frequency Domain", Proc. of IEEE Intl. Conf. on Acoustics, Speech and Signal Processing (ICASSP), April 2008, Las Vegas, NV, USA [Non-Patent Document 11] MM Goodwin, J.-M. Jot, "A Frequency-Domain Framework for Spatial Audio Coding Based on Universal Spatial Cues", Proc. of 120th AES Convention, Paper 6751, May 2006, Paris, France [Non-Patent Document 12] MM Goodwin, J.-M. Jot, "Analysis and Synthesis for Universal Spatial Audio Coding", Proc. of 121st AES Convention, Paper 6874, October 2006, San Francisco, CA, USA [Non-Patent Document 13] MM Goodwin, J.-M. Jot, "Primary-Ambient Signal Decomposition and Vector-Based Localization for Spatial Audio Coding and Enhancement", Proc. of IEEE Intl. Conf. on Acoustics, Speech and Signal Processing (CIASSP), April 2007, Honolulu, HI, USA. [Non-Patent Document 14] M. Kahrs, KH Brandenburg, "Applications of Digital Signal Processing to Audio and Acoustics", Kluwer Academic Publishers, 1998 [Non-Patent Document 15] J. Fliege, U. Maier, "The Distribution of Points on the Sphere and Corresponding Cubature Formulae", IMA Journal of Numerical Analysis, vol.19, no.2, pp.317-334, 1999 [Non-Patent Document 16] J. Blauert, "Spatial Hearing: The Psychophysics of Human Sound Localization," The MIT Press, 1996 [Overview of the project] [Problems that the invention aims to solve]

[0035] The problem that this invention aims to solve is to provide an improved lossy compression of the HOA representation of an audio scene that takes into account psychoacoustic phenomena such as perceptual masking. [Means for solving the problem]

[0036] This problem is solved by the methods disclosed in aspects 1 and 15. Apparatus utilizing these methods is disclosed in aspects 8 and 20.

[0037] According to the present invention, compression is performed in the spatial domain rather than the HOA domain (whereas in the above wave-field encoding, the masking phenomenon is assumed to be a function of spatial frequency, whereas the present invention uses the masking phenomenon as a function of spatial position). (N+1) 2 The input HOA coefficients are, for example, obtained by plane wave decomposition, (N+1) in the spatial domain. 2 It is converted into individual equivalent signals. Each of these equivalent signals represents a set of plane waves coming from the relevant direction in space. In a simplified way, the resulting signal can be interpreted as a virtual beamforming microphone signal that captures any plane waves from the input audio scene representation that enter the region of the relevant beam.

[0038] The resulting (N+1) 2 The set of individual signals is a normal time-domain signal, which can be input into a bank of parallel perceptual codecs. Any existing perceptual compression technique can be applied. On the decoder side, the individual spatial-domain signals are decoded, and the spatial-domain coefficients are converted back to the original HOA domain to recover the original HOA representation.

[0039] This type of processing has significant advantages.

[0040] • Psychoacoustic masking: When each spatial domain signal is treated separately from other spatial domain signals, the encoding error has the same spatial distribution as the masking signal. Therefore, after converting the decoded spatial domain representation back to the original HOA domain, the spatial distribution of the instantaneous power density of the encoding error is located according to the spatial distribution of the power density of the original signal. Advantageously, this ensures that the encoding error remains masked at all times. Even in sophisticated playback environments, the encoding error always propagates strictly along with the corresponding masking signal. However, it should be noted that something similar to "stereo unmasking" (Non-Patent Literature 14) can occur for sound objects originally located between two (in the case of 2D) or three (in the case of 3D) reference positions. However, the probability and severity of this potential pitfall decrease as the order of the HOA input material increases, because the angular distance between different reference positions in the spatial domain decreases. This potential problem can be mitigated by adapting the conversion from HOA to space according to the position of the dominant sound object (see specific embodiments below).

[0041] • Spatial Decorrelation: Audio scenes are typically sparse in the spatial domain and are usually assumed to be a mixture of a few discrete sound objects on top of an underlying ambient sound field. By transforming such an audio scene into the HOA domain—which is essentially a transformation to spatial frequencies—the spatially sparse, i.e., decorrelational scene representation is transformed into a highly correlated set of coefficients. Any information about the discrete sound objects is "smeared" to some extent across all frequency coefficients. Generally, the aim of compression methods is to reduce redundancy by choosing a coordinate system that is decorrelational, ideally according to the Karhunen-Lo'eve transformation. For time-domain audio signals, the frequency domain typically provides a more decorrelational signal representation. However, this is not true for spatial audio, because the spatial domain is closer to the KLT coordinate system than the HOA domain.

[0042] • Concentration of Temporally Correlated Signals: Another important aspect of converting HOA coefficients to the spatial domain is that signal components that are likely to exhibit strong temporal correlation (because they originate from the same physical sound source) are concentrated into a single or a few coefficients. This means that any subsequent processing steps involving the compression of spatially distributed time-domain signals can take advantage of the maximum possible time-domain correlation.

[0043] • Clarity: Audio content encoding and perceptual compression are well-known for time-domain signals. In contrast, understanding redundancy and psychoacoustics in complex transformed domains such as higher-order ambisonics (i.e., order 2 or higher) is far behind and requires much mathematics and research. As a result, when using compression techniques that operate in the spatial domain rather than the HOA domain, many existing insights and techniques can be applied and adapted much more easily. Advantageously, reasonable results can be obtained quickly by leveraging existing compression codecs for the parts of the system.

[0044] In other words, the present invention includes the following advantages. • Better utilization of psychoacoustic masking effects; • Easier to understand and implement It is more suitable for typical spatial audio scene compositions; • Better decorrelation attributes than existing approaches.

[0045] In principle, the encoding method of the present invention is suitable for encoding a series of frames of an ambisonic representation of a two-dimensional or three-dimensional sound field, denoted as HOA coefficients, and this method: • Frame O=(N+1) 2 The input HOA coefficients are converted into O spatial domain signals representing a regular distribution of reference points on a sphere, where N is the order of the HOA coefficients, and each of the spatial domain signals represents a set of plane waves coming from a related direction in space; Each of the spatial domain signals is encoded using perceptual encoding steps or stages, using encoding parameters selected so that the encoding error is imperceptible to the ear; This includes multiplexing the resulting bitstream of the frames into a unified bitstream.

[0046] In principle, the decoding method of the present invention is suitable for decoding a series of frames of an encoded higher-order ambisonic representation of a two-dimensional or three-dimensional sound field encoded according to Embodiment 1, and the decoding method is: • Demultiplex the received integrated bitstream in O=(N+1) 2 Convert each encoded spatial domain signal into individual; Each of the encoded spatial domain signals is decoded using a perceptual decoding step or stage corresponding to the selected encoding type and using decoding parameters that match the encoding parameters, to obtain a corresponding decoded spatial domain signal, the decoded spatial domain signal representing a regular distribution of reference points on a sphere; ·Converting the decoded spatial region signal into O output HOA coefficients of a frame, where N is the order of the HOA coefficients.

[0047] In principle, the encoding device of the present invention is suitable for encoding a series of frames of a high-order ambisonics representation of a two-dimensional or three-dimensional sound field represented by HOA coefficients. The device includes: ·O = (N + 1) 2 Conversion means adapted to convert O input HOA coefficients of a frame into O spatial region signals representing a regular distribution of reference points on a sphere, where N is the order of the HOA coefficients, and each of the spatial region signals represents a set of plane waves coming from a related direction in space; ·Means adapted to encode each of the spatial region signals using a perceptual encoding step or stage, using encoding parameters selected such that the encoding error is inaudible; ·Means adapted to multiplex the resulting bitstream of a frame into an integrated bitstream.

[0048] In principle, the encoding device of the present invention is suitable for decoding a series of frames of an encoded high-order ambisonics representation of a two-dimensional or three-dimensional sound field encoded according to Aspect 1. The device includes: ·Means adapted to demultiplex the received integrated bitstream into O = (N + 1) 2 encoded spatial region signals; ·Means for decoding each of the encoded spatial region signals using a perceptual decoding step or stage corresponding to the selected encoding type and using decoding parameters matching the encoding parameters to obtain corresponding decoded spatial region signals, where the decoded spatial region signals represent a regular distribution of reference points on a sphere; The system comprises a conversion means adapted to convert the decoded spatial domain signal into O output HOA coefficients of a frame, where N is the order of the HOA coefficients.

[0049] Advantageous additional embodiments of the present invention are disclosed in their respective dependent claims. [Brief explanation of the drawing]

[0050] Exemplary embodiments of the present invention are described with reference to the accompanying drawings. [Figure 1] This figure shows directional audio coding with B-format input. [Figure 2] This figure shows the direct encoding of a B-format signal. [Figure 3] This is a diagram illustrating the principle of spatial squeeze. [Figure 4] This diagram shows an encoding process that involves spatial squeezing. [Figure 5] This diagram illustrates the principle of wave field coding. [Figure 6] This is a diagram showing the wave field encoding process. [Figure 7] This figure shows spatial audio coding with downmixing and transmission of spatial cues. [Figure 8] This figure shows exemplary embodiments of the encoder and decoder of the present invention. [Figure 9] This figure shows the binaural masking level difference as a function of the binaural phase difference or time difference of various signals. [Figure 10] This figure shows an integrated psychoacoustic model incorporating BMLD modeling. [Figure 11] This figure illustrates an example of the maximum expected occupancy scenario, namely a movie theater with 7x5 seating (arbitrarily chosen for the example). [Figure 12] This figure shows the derivation of the maximum relative delay and attenuation for the scenario in Figure 11. [Figure 13]This figure shows the HOA component of the sound field and the compression of the two sound objects A and B. [Figure 14] This figure shows an integrated psychoacoustic model of the sound field HOA component and two sound objects A and B. [Modes for carrying out the invention]

[0051] Figure 8 shows a block diagram of the encoder and decoder of the present invention. In this basic embodiment of the present invention, a series of frames of the input HOA representation or signal IHOA are converted in the conversion step or stage 81 to a spatial domain signal according to a regular distribution of reference points on a three-dimensional sphere or a two-dimensional circle. With respect to the conversion from the HOA domain to the spatial domain, in ambisonics theory, a sound field at a particular point in space and around it is described by a truncated Fourier-Bessel series. Generally, the reference point is assumed to be at the origin of the chosen coordinate system. In three-dimensional applications using spherical coordinates, the coefficient A for all definable indices n=0,1,…,N and m=-n,…,n n m A Fourier series with this property describes the pressure of the sound field at an azimuth angle φ, inclination θ, and distance r from the origin.

[0052]

number

number

number

[0053] For two-dimensional applications using circular coordinates, the kernel function depends only on the azimuth angle φ. All coefficients where m≠n have a value of 0 and can be omitted. Therefore, the number of HOA coefficients is reduced to just O=2N+1. Furthermore, the slope θ=π / 2 is fixed. For the 2D case, the perfectly uniform distribution of sound objects on the circle, i.e., φ i For =2π / O, the mode vector in Ψ is identical to the well-known kernel function of the discrete Fourier transform (DFT).

[0054] The conversion from HOA to the spatial domain derives a driver signal for a virtual speaker (emitting a plane wave at infinite distance) that needs to be applied to accurately reproduce the desired sound field as described by the input HOA coefficients.

[0055] All mode coefficients can be combined into the mode matrix Ψ, where the i-th column is the mode vector Y, according to the direction of the i-th virtual speaker. n m ( i ,θ i ), including n=0,...,N, and m=-n,...,n. The number of desired signals in the spatial domain is equal to the number of HOA coefficients. Thus, there exists a unique solution to the transformation / decoding problem, which is the inverse Ψ of the mode matrix Ψ. -1 Defined by: s=Ψ -1 A.

[0056] This conversion relies on the assumption that the virtual speaker emits a plane wave. Real-world speakers have various reproduction characteristics, and the decoding rules for reproduction should take these various characteristics into account.

[0057] An example of a reference point is the sampling point based on Non-Patent Document 15. The spatial domain signal obtained by this transformation is input to O independent parallel known perceptual encoder steps or stages 821, 822, ..., 820. These steps or stages operate, for example, according to the MPEG-1 Audio Layer III (also known as mp3) standard, where O corresponds to the number of parallel channels. Each of these encoders is parameterized so that the encoding error is imperceptible to the ear. The resulting parallel bitstreams are multiplexed into an integrated bitstream BS in a multiplexer step or stage 83 and sent to the decoder side. Instead of mp3, any other suitable type of audio codec, such as AAC or Dolby AC-3, may be used. On the decoder side, the demultiplexer step or stage 86 multiplexes and demultiplexes the received integrated bitstream to derive the individual bitstreams of the parallel perceptual codecs. Each bitstream is decoded in known decoder steps or stages 871, 872, ..., 870 (corresponding to the selected encoding type and matching the encoding parameters, i.e., using decoding parameters selected so that the decoding error is imperceptible to the ear). This restores the uncompressed spatial domain signal. The resulting vector of the signal is, for each time step or stage 88, converted to the HOA domain, thereby restoring the decoded HOA representation or signal HOA, which is output in successive frames. Such processing or systems result in a considerable reduction in data rate. For example, the input HOA representation from a third-order recording of Eigenmike is (3+1) 2 The raw data rate is 16.9344 Mbit / s (coefficient × 44100Hz × 24 bits / coefficient). The result of the conversion to the spatial domain has a sample rate of 44100Hz (3+1). 2These are individual signals. Each of these (mono) signals, representing a data rate of 44100 × 24 = 1.0584 Mbit / s, is independently compressed to an individual data rate of 64 kbit / s using the mp3 codec (which means it is virtually transparent for the mono signals). Then the combined data rate of the integrated bitstream is (3 + 1) 2 The number of signals multiplied by 64 kbit / s per signal equals approximately 1 Mbit / s.

[0058] This evaluation is conservative because it assumes that the entire sphere surrounding the listener is uniformly filled with sound, completely ignoring the mutual masking effects between sound objects at different spatial locations. For example, an 80dB masking signal (masker signal) will mask weaker tones (e.g., 40dB) that are only a few degrees apart. Higher compression ratios can be achieved by taking such spatial masking effects into account, as shown below. Furthermore, the above evaluation completely ignores the correlation between adjacent positions in a pair of spatial domain signals. Again, higher compression ratios can be achieved if better compression processes utilize such correlations. Lastly, and this should never be underestimated, further compression efficiency can be expected when time-varying bitrates are present, especially for film audio, where the number of objects in a sound scene fluctuates significantly. The sparseness of sound objects can be used to further reduce the resulting bitrate.

[0059] <Transformation: Psychoacoustics> In the embodiment shown in Figure 8, minimal bitrate control is assumed. All individual perceptual codecs are expected to run at the same data rate. As mentioned above, considerable improvement can be achieved by using a more sophisticated bitrate that takes the full spatial audio scene into account. More specifically, a combination of time-frequency masking and spatial masking characteristics plays a key role. Regarding its spatial dimension, the masking phenomenon is a function of the absolute angular position of the sound event in relation to the listener, rather than the spatial frequency (note that this understanding differs from the understanding in Non-Patent Document 10 described in the section on Wave Field Coding). The difference between the masking threshold observed for spatial presentation compared to monodic presentation of the masking side [masker] and the masked side [maskee] is called the Binaural Masking Level Difference (BMLD). See Section 3.2.2 of Non-Patent Document 16. Generally, BMLD depends on several parameters, such as signal composition, spatial position, and frequency range. The masking threshold in spatial presentation can be lowered by about 20 dB compared to monody presentation. Therefore, the use of a masking threshold across the spatial domain should take this into consideration.

[0060] A) In one embodiment of the present invention, a psychoacoustic masking model is used that gives a multidimensional masking threshold curve that depends on the angle of sound occurrence on the entire circumference of a circle or sphere, depending on the (time-)frequency and audio scene dimensions, respectively. This masking threshold is (N+1) through operation by a spatial "spreading function" that takes BMLD into account. 2 This can be obtained by combining individual (time-)frequency masking curves obtained for each reference position. This allows us to utilize the influence of the masker on signals located nearby, i.e., at a small angular distance from the masker.

[0061] Figure 9 shows the BMLD for various signals (broadband noise masker and sine waves or 100 μs impulse trains as desired signals) as a function of the phase difference or time difference (i.e., phase angle and time delay) between the two ears of the signal, as disclosed in Non-Patent Literature 16.

[0062] The inverse of the worst-case characteristic (i.e., the one with the highest BMLD value) can be used as a conservative "blurring" function to determine the effect of a masker in one direction on a masky in another direction. This worst-case requirement can be mitigated if the BMLD for a particular case is known. The most interesting case is when the masker is spatially narrow but (time-)frequency broad noise.

[0063] Figure 10 shows how the BMLD model can be incorporated into psychoacoustic modeling to derive the integrated masking threshold MT. Individual MTs for each spatial direction are calculated in psychoacoustic model steps or stages 1011, 1012, ..., 1010 and input into the corresponding spatial spreading function SSF steps or stages 1021, 1022, ..., 1020. This spatial spreading function is, for example, the inverse of one of the BMLDs shown in Figure 9. Thus, an MT covering the entire sphere / circle (in the case of 3D / 2D) is calculated for all signal contributions from each direction. The maximum of all individual MTs is calculated in step / stage 103, providing the integrated MT (congruent MT in Figure 10) for the full audio scene.

[0064] B) Further extensions of this embodiment require a model of sound propagation in the target listening environment, such as a movie theater or other venue with a large audience, because sound perception depends on the listening position relative to the speakers. Figure 11 shows an exemplary movie theater scenario with 7 x 5 = 35 seats. When spatial audio signals are reproduced in a movie theater, audio perception and levels depend on the size of the auditorium and the position of individual listeners. A "perfect" rendering is typically achieved only in the sweet spot, which is usually in the center of the auditorium or at reference position 110. When a seat position located at the left edge of the audience is considered, for example, sound arriving from the right is likely to be attenuated and delayed compared to sound arriving from the left, because the direct line of sight to the right speaker is longer than the direct line of sight to the left speaker. To prevent spatially unmasking effects, this potential direction-dependent attenuation and delay resulting from sound propagation for suboptimal listening positions should be taken into consideration in worst-case scenarios. To prevent such effects, time delays and level changes are taken into account in the psychoacoustic model of the perceptual codec. To derive mathematical expressions for modeling the modified BMLD values, the maximum expected relative time delay and signal attenuation are modeled for any combination of masker and masky directions. Below, this is done for a two-dimensional exemplary setting. A possible simplification of the cinema example in Figure 11 is shown in Figure 12. The audience is at radius r A It is expected to exist within the circle. See the corresponding circle drawn in Figure 11. Two signal directions are possible. Masker S is shown arriving as a plane wave from the left (forward direction in the movie theater), and Maskee N is a plane wave arriving from the lower right of Figure 12, corresponding to the left rear in the movie theater.

[0065] The line representing the simultaneous arrival times of the two plane waves is drawn by a dashed bisector. The two circumferential points with the greatest distance to this bisector are the locations within the audience room where the greatest time / level difference occurs. Before reaching point 120 in the lower right of the diagram, these sound waves have reached the periphery of the listening area and then travel an additional distance d S and d N Proceed along this path.

[0066]

number

number

[0067] To determine the difference in propagation loss, a simple model is assumed below with a loss of k = 3...6 dB per double-distance (the exact figure depends on the speaker technology). Furthermore, the actual sound source is located d outside the listening area. LS It is assumed that the distance is as follows. Then the maximum propagation loss will be as follows:

[0068]

number

number

[0069] C) The same considerations introduced in the above sections can be applied to spatial audio formats that combine one or more discrete sound objects with one or more HOA components. Estimation of the psychoacoustic masking threshold is performed for the full audio scene and optionally includes consideration of the properties of the target environment as described above. Then, the individual compression of the discrete sound objects and the compression of the HOA components take the aforementioned integrated psychoacoustic masking threshold into account for bit allocation.

[0070] Compression of more complex audio scenes, which have both HOA sections and several distinct individual sound objects, is performed similarly to the integrated psychoacoustics model described above. The relevant compression process is illustrated in Figure 13. In parallel with the above considerations, the integrated psychoacoustics model should take all sound objects into account. The same motivations and structures introduced above can be applied. A high-level block diagram of the corresponding psychoacoustics model is shown in Figure 14.

[0071] Several aspects are described below. [Aspect 1] A method for encoding a series of frames of a higher-order ambisonic representation of a two- or three-dimensional sound field, denoted as the HOA coefficient: • Frame O=(N+1) 2 The input HOA coefficients are converted into O spatial domain signals representing a regular distribution of reference points on a sphere, where N is the order of the HOA coefficients, and each of the spatial domain signals represents a set of plane waves coming from a related direction in space; Each of the spatial domain signals is encoded using perceptual encoding steps or stages, using encoding parameters selected so that the encoding error is imperceptible to the ear; • Including multiplexing the resulting bitstream of the frame into a unified bitstream, method. [Aspect 2] The method according to embodiment 1, wherein the masking used in the encoding is a combination of time-frequency masking and spatial masking. [Aspect 3] The method according to embodiment 1 or 2, wherein the transformation is plane wave decomposition. [Aspect 4] The method according to embodiment 1, wherein the perceptual encoding corresponds to the MPEG-1 Audio Layer III or AAC or Dolby AC-3 standard. [Aspect 5] The method according to embodiment 1, wherein, in order to prevent the demasking of encoding errors from spatially distinct directions, direction-dependent attenuation and delay due to sound propagation for non-optimal listening positions are taken into consideration when calculating the masking threshold applied in the encoding. [Aspect 6] The method according to Embodiment 1, wherein each of the individual masking thresholds used in the encoding step or stage is modified by combining it with a spatial spread function that takes into account the interaural masking level difference BMLD, so that the maximum of these individual masking thresholds is formed to obtain an integrated masking threshold for all sound directions. [Aspect 7] The method according to embodiment 1, wherein separate sound objects are encoded individually. [Aspect 8] A device for encoding a series of frames of a higher-order ambisonic representation of a two- or three-dimensional sound field, denoted as the HOA coefficient: • Frame O=(N+1) 2 A conversion means adapted to convert O input HOA coefficients into O spatial domain signals representing a regular distribution of reference points on a sphere, where N is the order of the HOA coefficients, and each of the spatial domain signals represents a set of plane waves coming from a related direction in space; A means adapted to encode each of the spatial domain signals using perceptual encoding steps or stages, using encoding parameters selected so that the encoding error is inaudible to the ear; The frame has means adapted to multiplex the resulting bitstream into a unified bitstream. Device. [Aspect 9] The apparatus according to embodiment 8, wherein the masking used in the encoding is a combination of time-frequency masking and spatial masking. [Aspect 10] The apparatus according to embodiment 8 or 9, wherein the conversion is plane wave resolution. [Aspect 11] The apparatus according to aspect 8, wherein the perceptual encoding corresponds to the MPEG-1 Audio Layer III or AAC or Dolby AC-3 standard. [Aspect 12] The apparatus according to embodiment 8, wherein, in order to prevent the demasking of coding errors from spatially distinct directions, direction-dependent attenuation and delay due to sound propagation for non-optimal listening positions are taken into consideration when calculating the masking threshold applied in the encoding. [Aspect 13] The apparatus according to embodiment 8, wherein each of the individual masking thresholds used in the encoding step or stage is modified by combining it with a spatial spread function that takes into account the interaural masking level difference BMLD, so that the maximum of these individual masking thresholds is formed to obtain an integrated masking threshold for all sound directions. [Aspect 14] The apparatus according to embodiment 8, wherein separate sound objects are encoded individually. [Aspect 15] A method for decoding a series of frames of an encoded higher-order ambisonic representation of a two-dimensional or three-dimensional sound field encoded according to Embodiment 1, the method being: • Demultiplex the received integrated bitstream in O=(N+1) 2 Convert each encoded spatial domain signal into individual; Each of the encoded spatial domain signals is decoded using a perceptual decoding step or stage corresponding to the selected encoding type and using decoding parameters that match the encoding parameters, to obtain a corresponding decoded spatial domain signal, the decoded spatial domain signal representing a regular distribution of reference points on a sphere; This includes converting the decoded spatial domain signal into O output HOA coefficients of the frame, where N is the order of the HOA coefficients. method. [Aspect 16] The perceptual decoding method according to aspect 15, wherein the perceptual decoding corresponds to the MPEG-1 Audio Layer III or AAC or Dolby AC-3 standard. [Aspect 17] The method according to embodiment 15, wherein, in order to prevent the demasking of coding errors from spatially distinct directions, direction-dependent attenuation and delay due to sound propagation for non-optimal listening positions are taken into consideration when calculating the masking threshold applied in the decoding. [Aspect 18] The method according to embodiment 15, wherein each of the individual masking thresholds used in the decoding step or stage is modified by combining it with a spatial spread function that takes into account the interaural masking level difference BMLD, so that the maximum of these individual masking thresholds is formed to obtain an integrated masking threshold for all sound directions. [Aspect 19] The method according to aspect 15, wherein separate sound objects are decoded individually. [Aspect 20] A device for decoding a series of frames of an encoded higher-order ambisonic representation of a two-dimensional or three-dimensional sound field encoded according to Embodiment 1, the device comprising: • Demultiplex the received integrated bitstream in O=(N+1) 2 Means adapted to convert into individual encoded spatial domain signals; A means adapted to decode each of the encoded spatial domain signals using a perceptual decoding step or stage corresponding to a selected encoding type and decoding parameters that match the encoding parameters, so as to a corresponding decoded spatial domain signal, wherein the decoded spatial domain signal represents a regular distribution of reference points on a sphere; A conversion means adapted to convert the decoded spatial domain signal into O output HOA coefficients of a frame, wherein N is the order of the HOA coefficients, Device. [Aspect 21] The apparatus according to embodiment 20, wherein the perceptual decoding corresponds to the MPEG-1 Audio Layer III or AAC or Dolby AC-3 standard. [Aspect 22] The apparatus according to embodiment 20, wherein direction-dependent attenuation and delay due to sound propagation for non-optimal listening positions are taken into consideration when calculating the masking threshold applied in the decoding, in order to prevent the demasking of coding errors from spatially distinct directions. [Aspect 23] The apparatus according to embodiment 20, wherein each of the individual masking thresholds used in the decoding step or stage is modified by combining it with a spatial spread function that takes into account the interaural masking level difference BMLD, so that the maximum of these individual masking thresholds is formed to obtain an integrated masking threshold for all sound directions. [Aspect 24] The apparatus according to embodiment 20, wherein separate sound objects are decoded individually.

Claims

1. - A step of receiving a bitstream containing an encoded higher-order ambisonics (HOA) representation, wherein the encoded HOA representation is an encoded spatial domain signal obtained by converting the HOA coefficients of a two-dimensional or three-dimensional sound field HOA representation into a spatial domain signal and perceptually encoding the spatial domain signal; - The step of obtaining the encoded HOA representation by multiplexing the bitstream; - A step of perceptually decoding the encoded HOA representation to obtain a corresponding decoded spatial domain signal, wherein the perceptual decoding is a decoding corresponding to the perceptual encoding, and the decoded spatial domain signal represents a regular distribution of reference points on a sphere; - Including the step of individually decoding separate sound objects, method.

2. A receiver that receives a bitstream containing an encoded higher-order ambisonics (HOA) representation, wherein the encoded HOA representation is an encoded spatial domain signal obtained by converting the HOA coefficients of a two-dimensional or three-dimensional sound field HOA representation into a spatial domain signal and perceptually encoding the spatial domain signal; - A demultiplexer that demultiplexes the bitstream to obtain the encoded HOA representation; A decoder that perceptually decodes the encoded HOA representation to obtain a corresponding decoded spatial domain signal, wherein the perceptual decoding is a decoding corresponding to the perceptual encoding, and the decoded spatial domain signal represents a regular distribution of reference points on a sphere; It has a decoder that decodes each separate sound object individually, Device.

Images