Audio Processing

Abstract

According to an exemplary embodiment, a method for processing an input audio signal (101) based on spatial metadata (103) so as to play back the spatial audio signal in a device (50) based on at least one sound reproduction characteristic (105) of the device is provided, the method comprising: obtaining the input audio signal (101) and the spatial metadata (103); obtaining the at least one sound reproduction characteristic (105) of the device; rendering a first portion of the spatial audio signal using a first type of playback process applied to the input audio signal based on the spatial metadata (103), wherein the first portion includes sound directions within a front area of ​​the spatial audio signal; and rendering a second portion of the spatial audio signal using a second type of playback process applied to the input audio signal based on the spatial metadata (103) and based on the at least one sound reproduction characteristic (105), wherein the second portion includes sound directions not included in the first portion, and wherein the second type of playback process is different from the first type of playback process and involves crosstalk cancellation processing.

Description

Technical Field

[0001] Example and non-limiting embodiments of the present invention relate to audio signal processing. In particular, various embodiments of the present invention relate to device-specific rendering of spatial audio signals, such as stereo signals with associated spatial metadata. Background Art

[0002] Many portable handheld devices (such as mobile phones, portable media player devices, tablet computers, laptop computers, etc.) have a pair of speakers that enable stereo playback. Typically, the two speakers are located at opposite ends or opposite sides of the device to maximize the distance between them, thereby facilitating the reproduction of stereo audio. However, due to the small size of such devices, the two speakers are typically still relatively close to each other, resulting in a loss of spatial audio image in the reproduced stereo audio in many cases. In particular, the perceived spatial audio image may be different from the spatial audio image that would be perceived by playing back (e.g., via the speakers of a home stereo system) the same stereo audio signal, wherein the two speakers can be arranged in a suitable position relative to each other (e.g., far enough apart from each other) to ensure that the spatial audio image is reproduced with its full width or via headphones that enable the sound to be reproduced at a substantially fixed position relative to the listener's ears.

[0003] While a two-channel stereo signal serves as a conventional example of multi-channel sound reproduction that involves spatial characteristics to some extent, more advanced spatial audio reproduction can be provided via a parameterized spatial audio signal. In the present disclosure, the term "parameterized spatial audio signal" refers to an audio signal that is provided together with associated spatial metadata. The audio signal may include a single-channel audio signal or a multi-channel audio signal, and it may be provided as a time domain audio signal (e.g., such as linear PCM with a given number of bits per sample and a given sample rate) or an encoded audio signal that has been encoded using an audio encoder known in the art (and therefore needs to be decoded using a corresponding audio decoder before playback). The spatial metadata conveys information defining at least some characteristics of the spatial rendering of the audio signal, for example, provided as a set of spatial audio parameters. The spatial audio parameters may, for example, include one or more sound direction parameters defining the direction of sound in the corresponding one or more frequency subbands and one or more energy ratio parameters defining the ratio between the energy of the directional sound component in the corresponding frequency subband and the total energy.

[0004] During the audio rendering phase, spatial metadata is applied to control the processing of the audio signal to form an output audio signal in a desired spatial audio rendering format. The applicable spatial audio rendering format depends on the audio hardware intended for (and / or available for) rendering the spatial audio signal. Non-limiting examples of spatial audio rendering formats include (two-channel) binaural audio signals, Ambisonic (spherical harmonics) audio formats, or (specified) multi-speaker audio formats (such as 5.1 channel or 7.1 surround sound). Processes suitable for converting parameterized spatial audio signals into a spatial audio rendering format of interest are well known in the art. In this regard, see, for example, [1] Audio rendering using Ambisonic-based audio rendering, [2] Audio rendering for binaural output, and [3] Audio rendering for multi-speaker output. In a typical scenario, the audio signal is processed (separately) based on spatial metadata in multiple frequency subbands (e.g., in those frequency subbands for which associated spatial metadata is provided). Various other audio processing processes may be applied to the parameterized spatial audio signal prior to conversion to the spatial audio rendering format of interest, and / or such audio processing processes may be provided as part of the conversion from the parameterized spatial audio signal to the spatial audio rendering format of interest. Non-limiting examples of such audio processing processes include (automatic) gain control, audio equalization, noise processing, audio focus processing, and dynamic range processing.

[0005] For example, a parameterized spatial audio signal may be derived based on two or more microphone signals obtained from corresponding two or more microphones of a capture device or via conversion from a spatial audio signal provided in another audio format (e.g., in a spatial audio rendering format such as a given multi-speaker audio format). The obtained parameterized spatial audio signal may rely on spatial metadata comprising corresponding sound direction parameters and energy ratio parameters for a plurality of frequency subbands based on the two or more microphone signals obtained from corresponding two or more microphones of the capture device. For example, for microphone signals originating from a microphone array of a portable consumer device (such as a mobile phone, a tablet computer, or a digital camera, where the size and / or shape of these devices imposes restrictions on placing two or more microphones in the device), obtaining such a parameterized spatial audio signal may be an advantageous option. Practical experiments have shown that conventional "linear" audio capture techniques generally have significant limitations in capturing high-quality spatial audio from typical microphone arrays available in such devices, while audio capture techniques that (directly) operate based on microphone signals to record parameterized spatial audio signals generally enable high-quality spatial audio.

[0006] Crosstalk cancellation is an audio processing technique that is typically used to advantage in binaural audio reproduction using a pair of loudspeakers, in order to enable controlled sound reproduction to the left and right ears of a listener, thereby enabling binaural playback from the loudspeakers rather than from headphones. Another application in which crosstalk cancellation is typically applied is stereo widening, in which an input audio signal is processed into a signal that transmits a widened stereo image that typically spans the width of a physical loudspeaker setup, thereby enabling enhanced spatial sound reproduction, particularly in devices in which the loudspeakers used for stereo playback are placed close to each other. Crosstalk cancellation addresses the acoustic situation in which sound arrives at the ears of a listener from two loudspeakers: the crosstalk cancellation process aims to ensure that the sound is reproduced from the loudspeakers in a controlled manner so that acoustic signal cancellation occurs, at least over a certain frequency range, so that the sound can be reproduced to the user's ears in a manner similar to a scenario in which the user wears headphones to listen to binaural or stereo audio. For example, crosstalk cancellation techniques have been proposed in the context of stereo widening (e.g., in [4] and [5]), but crosstalk cancellation is also applicable, for example, to sound reproduction systems using more than two loudspeakers.

[0007] Referring back to the audio rendering stage, spatial audio rendering formats (e.g., binaural audio, Ambisonic, and multi-channel speaker formats mentioned above) do not themselves take into account audio reproduction characteristics specific to the audio hardware used for sound reproduction. However, this can be an important factor affecting the perceived sound quality, especially when reproducing spatial sound via speakers of mobile devices (such as mobile phones, portable media player devices, tablet computers, laptop computers, etc.). Typically, when reproducing parameterized spatial audio signals using speakers of such devices, one of the following options can be applied.

[0008] - Converting the parametric spatial audio signal into a "traditional" two-channel stereo format for playback via a pair of speakers of a mobile device, which typically results in a narrow spatial audio image limited by the width of the playback device.

[0009] - Converting the parameterized spatial audio signal into a binaural audio signal and applying a crosstalk cancellation process known in the art to the binaural audio signal. While this approach generally provides acceptable sound reproduction in devices with two loudspeakers having substantially identical sound reproduction characteristics arranged symmetrically with respect to a (presumed) listening position, it can lead to poor sound quality in scenarios where assumptions such as symmetry or similarity of the sound reproduction characteristics do not apply - which is the case in many (multi-purpose) mobile devices (whose primary purpose is not audio playback).

[0010] Therefore, there is room for improvement in spatially parametric spatial audio signal reproduction via two or more loudspeakers so that its sound quality can be (more) easily comparable to the sound quality obtainable via headphone listening.

[0011] refer to:

[0012] [1] International Patent Publication No. WO 2018 / 060550A1;

[0013] [2] Laitinen, Mikko-Ville; Pulkki, Ville, “Binaural reproduction for directional audio coding”, IEEE Symposium on Applications of Audio and Acoustic Signal Processing, 2009, pp. 337-340.

[0014] [3] Vilkamo, Juha; Pulkki, Ville, “Minimization of decorrelator artifacts in directional audio coding by covariance domain rendering”, Audio Engineering Society Journal, Vol. 61, No. 9, pp. 637-646.

[0015] [4] Kirkeby, O; Nelson, A; Hamada, H; Orduna-Bustamante, F, "Fast deconvolution of multichannel systems using regularization", IEEE Transactions on Speech and Audio Processing, Vol. 6, No. 2, pp. 189-194, 1998;

[0016] [5] Bharitkar, S; Kyriakis, C, “Immersive Audio Signal Processing”, ch.4, Springer, 2006;

[0017] [6] Vilkamo, J. T; Kuntz, A., “Optimized covariance domain framework for time-frequency processing of spatial audio”, Audio Engineering Society Journal, Vol. 61, No. 6, pp. 103-411, 2013. Summary of the invention

[0018] According to an example embodiment, a method is provided for processing an input audio signal according to spatial metadata so as to play back the spatial audio signal in a device based on at least one sound reproduction characteristic of the device, the method comprising: obtaining the input audio signal and the spatial metadata; obtaining the at least one sound reproduction characteristic of the device; rendering a first portion of the spatial audio signal using a first type of playback process applied to the input audio signal based on the spatial metadata, wherein the first portion includes sound directions within a front area of ​​the spatial audio signal; and rendering a second portion of the spatial audio signal using a second type of playback process applied to the input audio signal based on the spatial metadata and based on the at least one sound reproduction characteristic, wherein the second portion includes sound directions not included in the first portion, and wherein the second type of playback process is different from the first type of playback process and involves crosstalk cancellation processing.

[0019] According to another example embodiment, a device is provided for processing an input audio signal according to spatial metadata so as to play back the spatial audio signal in a device based on at least one sound reproduction characteristic of the device, the device being configured to: obtain the input audio signal and the spatial metadata; obtain the at least one sound reproduction characteristic of the device; render a first portion of the spatial audio signal using a first type of playback process applied to the input audio signal based on the spatial metadata, wherein the first portion includes sound directions within a front area of ​​the spatial audio signal; and render a second portion of the spatial audio signal using a second type of playback process applied to the input audio signal based on the spatial metadata and based on the at least one sound reproduction characteristic, wherein the second portion includes sound directions not included in the first portion, and wherein the second type of playback process is different from the first type of playback process and involves crosstalk cancellation processing.

[0020] According to another example embodiment, an apparatus is provided for processing an input audio signal according to spatial metadata so as to play back the spatial audio signal in a device according to at least one sound reproduction characteristic of the device, the apparatus comprising: a component for obtaining the input audio signal and the spatial metadata; a component for obtaining the at least one sound reproduction characteristic of the device; a component for rendering a first portion of the spatial audio signal using a first type of playback process applied to the input audio signal according to the spatial metadata, wherein the first portion includes sound directions within a front area of ​​the spatial audio signal; and a component for rendering a second portion of the spatial audio signal using a second type of playback process applied to the input audio signal according to the spatial metadata and according to the at least one sound reproduction characteristic, wherein the second portion includes sound directions not included in the first portion, and wherein the second type of playback process is different from the first type of playback process and involves crosstalk cancellation processing.

[0021] According to another example embodiment, an apparatus is provided for processing an input audio signal according to spatial metadata so as to play back the spatial audio signal in a device according to at least one sound reproduction characteristic of the device, wherein the apparatus comprises at least one processor and at least one memory comprising computer program code, which, when executed by the at least one processor, causes the apparatus to: obtain the input audio signal and the spatial metadata; obtain the at least one sound reproduction characteristic of the device; render a first portion of the spatial audio signal using a first type of playback process applied to the input audio signal according to the spatial metadata, wherein the first portion includes sound directions within a front area of ​​the spatial audio signal; and render a second portion of the spatial audio signal using a second type of playback process applied to the input audio signal according to the spatial metadata and according to the at least one sound reproduction characteristic, wherein the second portion includes sound directions not included in the first portion, and wherein the second type of playback process is different from the first type of playback process and involves crosstalk cancellation processing.

[0022] According to another exemplary embodiment, a computer program is provided comprising a computer readable program code configured to cause the execution of at least one method according to the exemplary embodiments described above when said program code is executed on a computing device.

[0023] The computer program according to the exemplary embodiments may be embodied on a volatile or non-volatile computer-readable recording medium, for example, embodied as a computer program product including at least one computer-readable non-transitory medium having program code stored thereon, which, when executed by an apparatus, causes the apparatus to perform at least the operations described above for the computer program according to the exemplary embodiments of the present invention.

[0024] The exemplary embodiments of the present invention proposed in this patent application should not be interpreted as limiting the applicability of the attached claims. The verb "comprise" and its derivatives are used as open limitations in this patent application, which do not exclude the existence of unlisted features. Unless otherwise explicitly stated, the features described below can be combined with each other arbitrarily.

[0025] Certain features of the invention are set forth in the appended claims. However, the invention in its various aspects both as to its construction and its method of operation, together with additional objects and advantages thereof, will be best understood from the following description of certain exemplary embodiments when read in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Embodiments of the invention are illustrated by way of example and not by way of limitation in the accompanying drawings, in which:

[0027] Figure 1 A block diagram illustrating some elements of an audio processing system according to an example;

[0028] Figure 2 A block diagram illustrating some elements of an apparatus for implementing an audio processing system according to an example;

[0029] Figure 3 A block diagram illustrating some elements of a signal decomposer according to an example;

[0030] Figure 4 A block diagram illustrating some elements of a spatial portion processor according to an example;

[0031] Figure 5 A block diagram illustrating some elements of an audio processing system according to an example;

[0032] Figure 6 A block diagram illustrating some elements of an audio processing system according to an example;

[0033] Figure 7 illustrates a flow chart describing a method for audio processing according to an example;

[0034] Figure 8 illustrates examples of performance obtainable via operation of an audio processing system according to an example;

[0035] Fig. 9 A block diagram illustrating some elements of an apparatus according to an example. DETAILED DESCRIPTION

[0036] Figure 1A block diagram of some components and / or entities of an audio processing system 100 is illustrated, which audio processing system 100 can be used as a framework for various embodiments of the audio processing techniques described in the present disclosure. The audio processing system 100 receives an input audio signal 101 and spatial metadata 103, which together constitute a parameterized spatial audio signal. The audio processing system 100 further receives at least one sound reproduction characteristic 105, which is used as a control input for controlling some aspects of audio processing in the audio processing system 100. The audio processing system 100 enables processing of the parameterized spatial audio signal into an output audio signal 115 of the audio processing system 100.

[0037] The input audio signal 101 comprises a single channel audio signal or a multi-channel audio signal, and it may be provided as a time domain audio signal (e.g., such as linear PCM with a given number of bits per sample and a given sample rate) or as an encoded audio signal that has been encoded using an audio encoder known in the art. In the scenario where the input audio signal 101 comprises a corresponding encoded audio signal, the audio processing system 100 operates using a corresponding audio decoder to decode the encoded audio signal into a corresponding time domain audio signal.

[0038] The spatial metadata 103 transmits information defining at least some characteristics of the spatial reproduction of the input audio signal 101, for example, provided as a set of spatial audio parameters. The following description assumes that the spatial audio parameters include one or more sound direction parameters defining the direction of sound in the corresponding one or more frequency sub-bands and one or more energy ratio parameters defining the ratio of the energy of the directional sound component to the total energy in the corresponding frequency sub-band (or the ratio of the energy of multiple directional sound components to the total energy). However, this is a non-limiting example selected for editorial clarity of the description, and in other examples a different set of spatial audio parameters for transmitting information defining the direction of sound and / or the relationship between the directional sound component and the diffuse sound component may be applied.

[0039] The parameterized spatial audio signal defined by the input audio signal 101 and the spatial metadata 103 defines a spatial audio image representing a sound scene, which may include one or more directional sounds in certain sound directions relative to an assumed listening point and ambient sounds and reverberations around the assumed listening point. In this regard, directional sounds may, for example, represent corresponding different sound sources in corresponding sound directions relative to the assumed listening point. In other examples, directional sounds may represent reflections or reverberations, a combination of multiple different sound sources, and / or ambient sounds around the assumed listening point. Therefore, the sound direction indicated in the spatial metadata for a certain frequency subband indicates the main sound direction in the frequency subband, and it does not necessarily indicate the direction (or even the existence) of different sound sources in the certain frequency subband.

[0040] At least one sound reproduction characteristic 105 includes information defining at least some characteristics of the sound rendering capabilities of a device implementing the audio processing system 100 and / or another device intended to play back the output audio signal 115. An example of information included in at least one sound reproduction characteristic 105 is information obtained based on acoustic measurements and / or acoustic simulations performed on the device (such as (complex-valued) crosstalk cancellation gains for one or more frequency subbands), which measurements or simulations may rely at least in part on the use of a simulated human head positioned at a typical listening (or viewing) distance relative to the device, wherein the simulated human head has corresponding microphones arranged at positions corresponding to corresponding ear positions. Other examples of information included in at least one sound reproduction characteristic 105 include an indication of the number of speakers in the indicating device and an indication of the speaker position of the device related to a reference position relative to the device. In this article, the reference position refers to the assumed listening (or viewing) position of the user relative to the device when listening to the sound reproduced via the speaker of the device (or viewing the visual content from the display of the device). The information defining the speaker position may include one or more of the following for each speaker of the device:

[0041] - the corresponding loudspeaker direction relative to a reference direction (e.g., assumed forward direction), for example, defined as the corresponding loudspeaker angle ∝ relative to the reference direction i .

[0042] - The corresponding loudspeaker distances relative to a reference position of the device.

[0043] As a non-limiting example in this regard, at least one sound reproduction characteristic 105 may define a speaker relative to a forward direction at a reference point at a speaker angle ∝ 1 = -15 degrees and ∝ 2 =15 degrees is positioned, the reference point is in the middle between the loudspeakers, assuming a listening distance of 30cm.

[0044] The output audio signal 115 may include an audio signal that provides a listener (located at or approximately located at a reference point relative to the device) with sound having binaural characteristics when reproduced via the speaker arrangement defined in the at least one reproduction characteristic 105. On the other hand, if reproduced via headphones, the reproduction of the output audio signal 115 does not provide the listener with sound having suitable binaural characteristics.

[0045] The audio processing system 100 enables processing of an input audio signal 101 in accordance with spatial metadata 103 so as to playback the spatial audio signal in a device in accordance with at least one sound reproduction characteristic 105 of the device. The processing performed by the audio processing system 100 includes: rendering a first portion of the spatial audio signal using a first type of playback process applied to the input audio signal 101 in accordance with the spatial metadata 103, wherein the first portion includes directions of sounds in a front area; and rendering a second portion of the spatial audio signal using a second type of playback process applied to the input audio signal 101 in accordance with the spatial metadata 103 and in accordance with the at least one sound reproduction characteristic 105, wherein the second portion includes directions of sounds not included in the first portion, and wherein the second type of playback process is different from the first playback process and involves crosstalk cancellation processing.

[0046] According to non-limiting examples, the first type of playback process may include or it may be based on an amplitude translation process. In other non-limiting examples, instead of amplitude translation, the first type of playback process may include, for example, delay translation, Ambisonics translation, or any combination or sub-combination of amplitude translation, delay translation and Ambisonics translation. In contrast, the first type of playback process does not involve any crosstalk cancellation processing, or the first type of playback process provides a substantially smaller crosstalk cancellation effect than the crosstalk cancellation processing involved in the second type of playback process. In the example, the first type of playback process may be further performed based on at least one sound reproduction characteristic.

[0047] In the following, without loss of generality, the operation of the audio processing system is described by way of example, wherein a first type of playback process involves an amplitude panning process further performed in accordance with at least one sound reproduction characteristic. Each of the first and second types of playback processes may further involve corresponding one or more audio signal processing techniques. As a non-limiting example in this regard, as described in more detail in the following examples, the first type of playback process may include audio equalization, while the second type of playback process may include binauralization.

[0048] As a brief overview, according to Figure 1The audio processing system 100 of the example shown in the figure comprises: a transform entity (or transformer) 102, for converting an input audio signal 101 from a time domain into a transform domain audio signal 107; a signal decomposer 104, for obtaining, based on the transform domain audio signal 107, a first signal component 109-1 representing a first part of a spatial audio image and a second signal component 109-2 representing a second part of the spatial audio image according to spatial metadata 103 and at least one sound reproduction characteristic 105; a first part processor 106, for obtaining, based on the first signal component 109-1 and based on the at least one sound reproduction characteristic 105, a first signal component 109-1 representing a first part of a spatial audio image and a second signal component 109-2 representing a second part of the spatial audio image. , obtaining a modified first signal component 111-1; a second part processor 108, for obtaining a modified second signal component 111-2 based on the second signal component 109-2 and in accordance with at least one sound reproduction characteristic 105; a signal combiner 110, for combining the modified first signal component 111-1 and the modified second signal component 111-2 into a transform domain output audio signal 113 suitable for speaker reproduction; and an inverse transform entity 112, for converting the transform domain output audio signal 113 into a (time domain) output audio signal 115 for use as an output audio signal of the audio processing system 100.

[0049] In other examples, the audio processing system 100 may include Figure 1 entities other than those shown in, and / or Figure 1 Some of the entities depicted in can be combined with other entities while providing the same or corresponding functions. Figure 1 The entities shown in and in the subsequent Figures 2 to 4 The entities shown in the figure are used to represent logical components of the audio processing system 100, which are arranged to perform corresponding functions but do not impose structural limitations on implementing the corresponding entities. Thus, for example, corresponding hardware components, corresponding software components, or corresponding combinations of hardware components and software components can be applied to implement the audio processing system 100 separately from other entities. Figures 1 to 4 Any one of the entities shown in the corresponding one of Figures 1 to 4 Any subcombination of two or more entities shown in a corresponding one of Figures 1 to 4 All entities shown in the corresponding one.

[0050] The audio processing system 100 may be arranged (in view of the spatial metadata 103) to process an input audio signal 101 arranged as a sequence of input frames, each input frame comprising a respective segment of a digital audio signal for each channel, provided as a respective time sequence of input samples at a predefined sampling frequency. In a typical example, the audio processing system 100 uses a fixed predefined frame length. In other examples, the frame length may be a selectable frame length selectable from a plurality of predefined frame lengths, or the frame length may be an adjustable frame length selectable from a predefined range of frame lengths. The frame length may be defined as the number of samples L included in a frame for each channel of the input audio signal 101, which is mapped to a corresponding duration at a predefined sampling frequency. As an example in this regard, the audio processing system 100 may use a fixed frame length of 20 milliseconds (ms), which produces frames of L=160, L=320, L=640, and L=960 samples per channel at sampling frequencies of 8, 16, 32, or 48 kHz, respectively. The frames may be non-overlapping, or they may partially overlap. However, these values ​​are used as non-limiting examples, and frame lengths and / or sampling frequencies different from these examples may be used instead, eg, depending on the required audio bandwidth, desired frame delay, and / or available processing power.

[0051] The audio processing system 100 can be implemented by one or more computing devices, and the resulting output audio signal 115 can be provided for playback via a speaker of one of these devices. Typically, the audio processing system 100 is implemented in a portable handheld device (such as a mobile phone, a media player device, a tablet computer, a laptop computer, etc.), which is also used to playback the output audio signal 115 via a pair of speakers provided in the device. In another example, the audio processing system 100 is provided in a first device, and playback of the output audio signal 115 is provided in a second device. In another example, a first part of the audio processing system 100 is provided in a first device, and a second part of the audio processing system 100 and playback of the output audio signal 115 are provided in a second device. In the latter two examples, the second device may include a portable handheld device (such as a mobile phone, a media player device, a tablet computer, a laptop computer, etc.), and the first device may include any type of computing device, for example, a portable handheld device, a desktop computer, a server device, etc.

[0052] Figure 2 The block diagram of some components and / or entities of a device 50 that can be used to implement the audio processing system 100 is shown. The device 50 can be provided, for example, as a portable handheld device or other type of mobile device. For the sake of brevity and clarity of description, the reference Figure 2In the following description of the present invention, it is assumed that the elements of the audio processing system 100 and the resulting playback of the output audio signal 115 are provided in a device 50. The device 50 further comprises: a microphone array 52 comprising two or more microphones; an audio preprocessor 54 for processing respective microphone signals captured by the microphone array 52 into a parameterized spatial audio signal comprising the input audio signal 101 and the spatial metadata 103; a memory 56 for storing information, such as the parameterized spatial audio signal and at least one sound reproduction characteristic 105; an audio driver 58; and a pair of speakers 60, wherein the audio driver 58 is arranged to drive the playback of the output audio signal 115 via the speakers 60.

[0053] In the device 50, the audio processing system 100 may receive the parameterized spatial audio signal (comprising the input audio signal 101 and the spatial metadata 103) and the at least one sound reproduction characteristic 105 by reading this information from a memory 56 in or coupled to the device 50. In another example, the device 50 may receive the parameterized spatial audio signal and / or the at least one sound reproduction characteristic 105 via a communication interface (such as a network interface) from another device storing one or both of these pieces of information in a memory provided therein. Instead of or in addition to providing the output audio signal 115 for playback via the audio driver 58 and the speaker 60, the device 50 may be arranged to store the output audio signal 115 in the memory 56 and / or provide the output audio signal 115 to another device via a communication interface for rendering and / or storage therein.

[0054] Return to reference Figure 1 , the transform entity 102 can be arranged to convert the input audio signal 101 from the time domain into a transform domain audio signal 107. Typically, the transform domain involves the frequency domain. In an example, the transform entity 102 uses a short-time discrete Fourier transform (STFT) to convert each channel of the input audio signal 101 into a corresponding channel of the transform domain audio signal 107 using a predefined analysis window length (e.g., 20 milliseconds). In another example, the transform entity 102 uses a (analysis) complex modulated quadrature mirror filter (QMF) bank for time domain to frequency domain conversion. The STFT and QMF banks serve as non-limiting examples in this regard, and any suitable transform technique known in the art can be used in other examples to create the transform domain audio signal 107.

[0055] A portion of the processing performed by the audio processing system 100 (e.g., at least some aspects of the processing performed by the signal decomposer 104) may be performed separately for a plurality of frequency subbands. Thus, the operation of the audio processing system 100 may include (at least conceptually) dividing or decomposing each channel of the transform domain audio signal 107 into a plurality of frequency subbands, thereby providing a corresponding time-frequency representation for each channel of the input audio signal 101. According to a non-limiting example, the (conceptual) frequency subband division may be performed by the transform entity 102 or by the signal decomposer 104, if applicable.

[0056] A given frequency band in a given frame may be referred to as a time-frequency tile. The number of frequency subbands and the corresponding bandwidths of the frequency subbands may be selected, for example, based on the desired frequency resolution and / or available computing power. In an example, the subband structure involves 24 frequency subbands according to the Bark scale, the equivalent rectangular band (ERB) scale, or the 3rd octave band scale known in the art. In other examples, different numbers of frequency subbands with the same or different bandwidths may be used. In this regard, a specific example is a single frequency subband covering the entire input spectrum or a single frequency subband covering a continuous subset of the input spectrum.

[0057] A time-frequency block representing a frequency bin b in a time frame n of a channel i of the transform domain audio signal 107 may be denoted as x(i,b,n). The transform domain audio signal 107 (e.g., the time-frequency block x(i,b,n)) is passed to the signal decomposer 104 to be decomposed into a first signal component 109-1 and a second signal component 109-2 therein. As previously described, a plurality of consecutive frequency bins may be grouped into frequency subbands, thereby providing a plurality of frequency subbands k=0,…,K-1. For each frequency subband k, the lowest bin (i.e., the frequency bin representing the lowest frequency in the frequency subband) may be denoted as b k,low , the highest bin (i.e., the frequency bin representing the highest frequency in the frequency subband) may be denoted as b k,high In the following examples, it is (implicitly) assumed that STFT is used in the transform entity 102. In such an example, the transform entity 102 may transform each frame n of the input audio signal 101 into a corresponding frame of the frequency domain audio signal 107 having one time sample per time frame (for each frequency bin b). In other examples, the transform may produce a plurality of samples in the transform domain audio signal 107 for each time frame (for each frequency bin b).

[0058] Still reference Figure 1, the signal decomposer 104 may be configured to obtain a first signal component 109-1 representing a first portion of a spatial audio image and a second signal component 109-2 representing a second portion of the spatial audio image based on the transform domain audio signal 107 and in accordance with at least one sound reproduction characteristic 105. In this regard, the first portion may include a specified spatial portion or spatial region of the spatial audio image, and the second portion may represent one or more spatial portions or regions of the spatial audio image that do not include the specified spatial portion. In an example, the second portion may include the remaining portion of the spatial audio image, i.e., those portions of the spatial audio image that are not included in the first portion.

[0059] According to a non-limiting example, the first part includes sound directions within the front area of ​​the spatial audio image, and the second part includes sound directions not included in the first part (e.g., those sound directions not included in the front area). Typically, but not necessarily, the second part includes the remaining area relating to those parts of the spatial audio image that are not included in the front area. The remaining area may also be referred to as the "peripheral" area of ​​the spatial audio image. Therefore, in the context of this example, the first signal component 109-1 may also be referred to as the front area signal, and the second signal component 109-2 may also be referred to as the remaining signal. Therefore, the front area may represent those directional sounds of the spatial audio image within the range of a predefined sound direction defining the front area in the spatial audio image, and the remaining area may represent the directional sounds of the spatial audio image outside the predefined range and the ambient (non-directional) sounds of the spatial audio image.

[0060] In a non-limiting example, the first part consists of sound directions within the front area, while the second part does not include sound directions within the front area but consists of sound directions outside the front area and ambient sounds of the spatial audio image. However, it is easy for a person skilled in the art to understand that due to the necessary constraints imposed by the actual implementation of the signal decomposer 104 operating on real-world audio signals, it may not be possible to strictly include only directional sounds within the front area in the first part and / or to strictly remove these sounds from the second part. Therefore, in this non-limiting example, strictly including only directional sounds within the front area in the first part and strictly removing these directional sounds from the second part only states the processing purpose rather than the processing result in all real-life scenarios.

[0061] The signal decomposition process performed by the signal decomposer 104 may include: obtaining a first signal component 109-1 based on the transform domain audio signal 107 using an amplitude translation technique in view of the spatial metadata 103 and in view of the at least one sound reproduction characteristic 105; and obtaining a second signal component 109-2 based on the transform domain audio signal 107 using a binauralization technique in view of the spatial metadata 103 and in view of the at least one sound reproduction characteristic 105. Typically, the decomposition process results in each of the first signal component 109-1 and the second signal component 109-2 having a corresponding audio channel for each speaker of the device implementing the audio processing system 100 (e.g., the speaker 60 of the device 50). Therefore, in the case of processing the parameterized spatial audio signal for playback by two speakers, each of the first signal component 109-1 and the second signal component 109-2 has a corresponding two audio channels regardless of the number of channels of the transform domain audio signal 107. Wherein, according to an example, two channels of the first signal component 109-1 can be used to transmit spatial sound, wherein any directional sound in the front area of ​​the spatial audio image is arranged in the corresponding sound direction via application of amplitude panning technology, and two channels of the second signal component 109-2 can be used to transmit binaural spatial sound, including any directional sound outside the front area and any ambient sound of the spatial audio image. The signal decomposer 104 provides the first signal component 109-1 to the first part processor 106 for corresponding further processing therein in view of at least one sound reproduction characteristic 105, and provides the second signal component 109-2 to the second part processor 108 for corresponding further processing therein in view of at least one sound reproduction characteristic 105.

[0062] Figure 3The block diagram of some components and / or entities of the signal decomposer 104 according to an example is shown, and the signal decomposer 104 includes: a covariance matrix estimator 114 for obtaining a covariance matrix 119 and an energy metric (energy a target matrix estimator 116 for obtaining a first target covariance matrix 121-1 and a second target covariance matrix 121-2 based on the spatial metadata 103 and the energy measure 117, in view of at least one sound reproduction characteristic 105, wherein the first target covariance matrix 121-1 represents the sound included in the first part of the spatial audio image, and the second target covariance matrix 121-2 represents the sound included in the second part of the spatial audio image; a mixing rule determiner 118 for obtaining a first mixing matrix 123-1 and a second mixing matrix 123-2 based on the covariance matrix 119, the first target covariance matrix 121-1 and the second target covariance matrix 121-2; and a mixer 120 for obtaining a first signal component 109-1 and a second signal component 109-2 based on the transform domain audio signal 107 in view of the mixing matrices 123-1 and 123-2. In the following, the method according to the present invention will be described in more detail. Figure 3 In other examples, the signal decomposer 104 may include other entities, and / or Figure 3 Some of the entities depicted in may be omitted or combined with other entities.

[0063] The covariance matrix estimator 114 is arranged to perform a covariance matrix estimation process, which includes obtaining a covariance matrix 119 and an energy metric 117 based on the transform domain audio signal 107. The covariance matrix estimator 114 provides the covariance matrix 119 for the mixing rule determiner 118 and provides the energy metric 117 for the target matrix estimator to perform further processing therein accordingly. Assuming a dual-channel frequency domain audio signal 107, it can be represented in vector form as:

[0064]

[0065] Based on this definition, according to an example, the covariance matrix 119 may be obtained as:

[0066]

[0067] Wherein, E[] denotes the expectation operator and H denotes the Hermitian transpose. In an example, the expectation value derivable via the expectation operator may be provided as an average value over several (continuous) time indices n, while in another example, the instantaneous value of x(b,n) may be directly applied as the expectation value without time averaging over the time index n. The energy metric 117 may, for example, include a covariance matrix C calculated as x The overall energy measure e(k,n) is the sum of the diagonal elements of (k,n).

[0068] The target matrix estimator 116 can be configured to obtain a first target covariance matrix 121-1 and a second target covariance matrix 121-2 based on the spatial metadata 103 and the energy metric 117, possibly in view of at least one sound reproduction characteristic 105. The target matrix estimator 116 provides the first and second target covariance matrices 121-1, 121-2 for the mixing rule determiner 118 for further processing therein. For clarity of description, an example of spatial audio parameters involving one or more sound direction parameters defining the corresponding sound directions in the horizontal plane for one or more frequency subbands is described below. This can be easily analogized to further examples, which, additionally or alternatively, involve spatial audio parameters involving one or more sound direction parameters defining the corresponding elevation angles of the sound directions for one or more frequency subbands.

[0069] In the present example, the spatial audio parameters included in the spatial metadata 103 include one or more azimuth angles θ(k,n), which are used as corresponding sound direction parameters for one or more frequency subbands. In particular, the azimuth angle θ(k,n) indicates the azimuth angle relative to a predefined reference sound direction for frequency subband k for time index n (e.g., the direction just in front of the assumed listening point). In addition, in the present example, the spatial audio parameters included in the spatial metadata 103 include one or more direct-to-total energy ratios r(k,n), which are used as corresponding energy ratio parameters for one or more frequency subbands. In particular, the direct-to-total energy ratio r(k,n) indicates the ratio of directional energy to total energy in frequency subband k at time index n.

[0070] In this regard, the calculation of the target covariance matrix 121-1, 121-2 may include: determining an energy factor value d(k,n) for a frequency subband k for a time index n based on the spatial metadata 103, for example, such that the energy factor value d(k,n) has a value of 1 for those time-frequency image tiles whose indicated direction is within the first part of the spatial audio image (e.g., the sound direction is within the sound direction range defining the front area in the spatial audio image), and has a value of 0 for other time-frequency image tiles. According to a non-limiting example, the energy factor value d(k,n) may be defined as:

[0071]

[0072] Among them, θ d = θ denotes the absolute value of the angle defining the range of sound directions around a predefined reference direction (e.g., the front direction) belonging to the front region in the spatial audio image. Thus, equation (3a) assumes a front region that is symmetrically positioned around the reference direction, such that the front region spans from -θ d to θ d In another example, the front area is not located symmetrically around the reference direction, and the energy factor value d(k,n) may be defined as:

[0073]

[0074] Among them, θ d1 ,θ d2 The indications define corresponding angles of the sound direction range relative to a reference direction (eg, the front direction) belonging to the front region of the spatial audio image.

[0075] According to the example, the angle θ d or angle θ d1 ,θ d2 It can be obtained, for example, based on at least one sound reproduction characteristic 105. As a specific example, the angle θ d or angle θ d1 ,θ d2 The loudspeaker angle α defined in at least one sound reproduction characteristic 105 may be i To obtain, for example, so that θ d1 =α 1 And θ d2 =α 2 In another example, the angle θ d or angle θ d1 ,θ d2 can be included in at least one sound reproduction characteristic 105. In another example, the angle θ d or angle θ d1 ,θ d2is predefined.

[0076] Generally, the energy factor value d(k,n) indicates the degree of directional sound inclusion of the first part of the spatial audio image. In this regard, equations (3a) and (3b) are used to provide a non-limiting example of providing the energy factor value d(k,n) as a "binary" value that indicates, for a frequency subband k and a time index n, one of inclusion in the first part (e.g., d(k,n)=1) and exclusion from the first part (e.g., d(k,n)=0). In another example, the transition between the first part and the second part of the spatial audio image can be made smooth, e.g., by introducing a transition range around the angle θ d or the angle θ d1 , θ d2 such that the energy factor value d(k,n) is set to a value 0 < d(k,n) < 1, and the energy factor value decreases as the distance from a reference direction (e.g., the front direction) increases. Thus, for a frequency subband k and a time index n, the contribution of the directional sound whose sound direction is within the transition range is divided between the first part and the second part according to the energy factor value d(k,n).

[0077] As previously described, the signal decomposition process performed by the signal decomposer 104 may include using an amplitude translation technique to obtain the first signal component 109-1. Thus, the calculation of the first target covariance matrix 121-1 may further include: for each time-frequency map block, determining a corresponding translation gain vector g(k,n) based on the sound direction parameter defined for the corresponding time-frequency map block in the spatial metadata 103, where the translation gain vector g(k,n) includes the corresponding translation gain for each channel of the first signal component 109-1 subsequently obtained by the operation of the signal decomposer 104. In an example, this includes: for a frequency subband k and a time index n, determining the translation gain vector g(θ(k,n)) based on the azimuth angle θ(k,n) defined for the corresponding time-frequency map block. In this example, the translation gain vector g(θ(k,n)) includes a 2x1 corresponding real-valued gain vector, thereby providing corresponding gains for the left and right channels. Any amplitude translation technique known in the art may be used to obtain the translation gain g(k,n), e.g., vector-based amplitude panning (VBAP), tangent panning law, or sine panning law.

[0078] Given the known energy value e(k,n), energy factor value d(k,n), and translation gain g(k,n), the target matrix estimator 116 may continue to obtain the first target covariance matrix 121-1 based on the spatial audio parameters available in the spatial metadata, e.g., as:

[0079] C 1 (k,n) = g(k,n)gT (k,n)d(k,n)r(k,n)e(k,n) (4)

[0080] Therefore, according to the first target covariance matrix C of equation (4) 1 (k,n) represents those directional sounds included in the first part of the spatial audio image (eg, in the front area of ​​the spatial audio image).

[0081] Along the lines described above, the signal decomposition process performed by the signal decomposer 104 may include: using, for example, a binauralization technique as described below, to obtain the second signal component 109-2 as a binaural audio signal. Therefore, as an example, the calculation of the second target covariance matrix 121-2 may include: for each time-frequency block, based on the sound direction parameters defined for the corresponding time-frequency block in the spatial metadata 103, determine the corresponding head-related transfer function (HRTF) vector h(k,n). In the example, this includes: for the time index n, for the frequency subband k, based on the azimuth angle θ(k,n) defined for the corresponding time-frequency block, determine the HRTF vector h(k,θ(k,n)), whereby the HRTF vector h(k,θ(k,n)) includes a 2x1 corresponding complex-valued gain vector and provides corresponding gains for the left and right channels. The HRTF vector h(k,n) can be obtained, for example, from an HRTF database stored in a memory of a device implementing the audio processing system 100 (e.g., in the memory 56 of the device 50).

[0082] The acquisition of the second target covariance matrix 121-2 may further include: obtaining the diffusion field covariance matrix C d (k), which can be obtained by assuming a set of sound direction values ​​θ that (preferably substantially uniformly) spans a predefined range of sound directions m (where m=1,…,M) is predefined as:

[0083]

[0084] Here, the set of sound direction values ​​θ m It may include, for example, from 20 to 60 sound directions that are (pseudo) uniformly spaced to cover the desired spatial portion of the 3D space, thereby modeling the responses from all directions of the desired spatial portion of the 3D space. As previously described, the diffuse field covariance matrix C d (k) may be pre-calculated, for example, according to equation (5) and provided to the signal decomposer 104 for obtaining the second target covariance matrix 121 - 2 .

[0085] Therefore, for example, the second target covariance matrix 121-2 can be obtained as:

[0086] C 2 (k,n)=r(k,n)h(k,n)h H (k,n)(1-d(k,n))e(k,n)+(1-r(k,n))C d (k)e(k,n) (6)

[0087] Therefore, according to the second target covariance matrix C of equation (6) 2 (k,n) represents those directional sounds included in the second portion of the spatial audio image (eg, the remaining area of ​​the spatial audio image) and the non-directional (ambient) sounds of the spatial audio image.

[0088] The mixing rule determiner 118 may be configured to: obtain a first mixing matrix 123-1 and a second mixing matrix 123-2 based on the covariance matrix 119, the first target covariance matrix 121-1 and the second target covariance matrix 121-2; and provide the first and second mixing matrices 123-1, 123-2 to the mixer 120 for further processing therein. The mixing rule determination process performed by the mixing rule determiner 118 may include: obtaining the first mixing matrix 123-1 based on the covariance matrix 119 and the first target covariance matrix 121-1; and obtaining the second mixing matrix 123-2 based on the covariance matrix 119 and the second target covariance matrix 121-2. As an example, the corresponding acquisition of the first mixing matrix 123-1 and the second mixing matrix 123-2 may be performed as described in [6].

[0089] In particular, the formula provided in Appendix [6] can be used to calculate the covariance matrix 119 (eg, the covariance matrix C x (k, n)) and the first target covariance matrix 121-1 (eg, target covariance matrix C 1 (k,n)), get the mixing matrix M 1 (k, n), which can be used as a first mixing matrix 123-1 for the corresponding time-frequency block to obtain the corresponding time-frequency block of the first signal component 109-1 so that it has the same or similar covariance matrix as the first target covariance matrix 121-1. Along similar lines, the process of [6] can be applied to obtain a covariance matrix 119 (e.g., covariance matrix C for frequency subband k based on time index n) for the first signal component 109-1. x (k, n)) and the second target covariance matrix 121-2 (eg, target covariance matrix C 2 (k,n)), get the mixing matrix M 2(k, n), which can be used as a second mixing matrix 123-2 for the corresponding time-frequency map block to obtain the corresponding time-frequency map block of the second signal component 109-2 so that it has a covariance matrix that is the same as or similar to the second target covariance matrix 121-2.

[0090] For the purpose of this acquisition, a prototype matrix Q is defined to guide the mixing matrix M according to the procedure described in detail in [6]. 1 (k,n) and M 2 Generation of (k,n):

[0091]

[0092] The procedure explained in detail in [6] is applied based on the covariance matrix C x (k,n) and the first target covariance matrix C 1 (k,n) Get the mixing matrix M 1 (k,n), so that when the mixing matrix M 1 (k,n) is applied to the covariance matrix C x (k,n) signal, the resulting processed signal is approximated by the least squares optimization method with the covariance matrix C 1 (k,n) signal. Along similar lines, this process can be applied to obtain the mixing matrix M 2 (k,n), which is applied to the covariance matrix C x (k,n) produces a processed signal that is approximated in the sense of least squares optimization to a signal with a covariance matrix C 2 Therefore, a mixing matrix M is provided as the first mixing matrix 123-1. 1 (k, n) is used to obtain the first signal component 109-1 based on the transform domain audio signal 107, and the mixing matrix M used as the second mixing matrix 123-2 is provided. 2 (k, n) for obtaining the second signal component 109-2 based on the transform domain audio signal 107. In this article, the prototype matrix Q is provided as a unit matrix so that the signal content in the channels of the first and second signal components 109-1, 109-2 is similar to the signal content of the corresponding channels of the transform domain audio signal 107 (and therefore similar to the signal content of the corresponding channels of the input audio signal 101).

[0093] The mixer 120 may be configured to obtain a first signal component 109-1 and a second signal component 109-2 based on the transform domain audio signal 107 in view of the mixing matrices 123-1, 123-2, and to provide the first and second signal components 109-1, 109-2 to the first and second partial processors 106, 108, respectively, for further processing therein.

[0094] The mixing process performed by the mixer 120 may include obtaining the first signal component as the product of the first mixing matrix 123-1 and the transform domain audio signal 107, for example, as:

[0095]

[0096] where k denotes the frequency subband in which frequency bin b resides, and where x 1 (1,b,n) and x 1 (2, n, n) respectively denote the left channel and the right channel of the first signal component 109-1. Along similar lines, the mixing process may include: obtaining the second signal component as the product of the second mixing matrix 123-2 and the transform domain audio signal 107, for example, as:

[0097]

[0098] Among them, x 1 (1,b,n) and x 1 (2, b, n) respectively denote the left channel and the right channel of the second signal component 109 - 2 .

[0099] Although the above examples provided in equations (8a) and (8b) apply the first and second mixing matrices M 1 (k,n) and M 2 (k, n), but in another example, before applying them to generate the first and second signal components 109-1, 109-2, the mixing matrix M 1 (k,n) and M 2 One or both of (k, n) may be subjected to temporal smoothing (such as averaging over a predefined number of frames (eg, four frames)).

[0100] Now return to reference Figure 1, the first part processor 106 may be configured to obtain a modified first signal component 111-1 based on the first signal component 109-1 and in accordance with the at least one sound reproduction characteristic 105; and provide the modified first signal component 111-1 to the signal combiner 110 for further processing therein. In this regard, the first part processor 106 may be configured to apply a set of equalization gains to obtain a modified first signal component 111-1 based on the first signal component 109-1, for example, as:

[0101] x′ 1 (i,b,n)=g EQ (i,k)x 1 (i,b,n) (9)

[0102] Among them, x′ 1 (i, b, n) denotes the modified first signal component 111-1 of frequency bin b for time index n in channel i (eg, obtained according to equation (8a) above), g EQ (i, k) denotes the equalization gain for channel i in frequency subband k in which frequency bin b resides. Thus, for each channel i, equation (9) applies the corresponding equalization gain g defined for frequency subband k for each frequency bin b of that frequency subband EQ (i, k), thereby generating an equalization gain that may be different depending on the frequency subband k and channel i.

[0103] Equalization gain g EQ (i, k) may include corresponding predefined gain values ​​reflecting the characteristics of a device (eg, device 50) implementing the audio processing system 100. According to an example, the equalization gain g EQ (i, k) may include corresponding predefined gain values ​​provided as part of the at least one sound reproduction characteristic 105. In another example, the at least one sound reproduction characteristic 105 may include a gain value that may be used as a parameter for obtaining the equalization gain g EQ (i, k) based on the corresponding gain, and / or at least one sound reproduction characteristic 105 may include enabling acquisition of the equalization gain g EQ Other types of equilibrium information for (i,k).

[0104] According to the example, the equalization gain g EQ (i, k) may have been obtained on an experimental basis, for example, by recording a test signal using a microphone located at a reference position relative to a device implementing the audio processing system 100 (eg, the device 50), and obtaining the equalization gain g EQ (i, k) such that they equalize the spectrum of the test signal to the desired degree. In the example, the equalization gain g EQ(i, k) is set so as to avoid over-amplification of the spectrum parts where the signal level is relatively low. In another example, additionally or alternatively, at least some of the equalization gains g EQ (i,k) can be set to unity or a value close to unity.

[0105] Equalization gain g EQ (i, k) is intended to equalize the response of the loudspeakers of the device so that the timbre of the sound is less colored, while at the same time possibly providing different equalization gains g for channel i of the first signal component 109-1. EQ (i, k) aims to mitigate the differences in the respective responses of the loudspeakers. Therefore, applying the equalization gain g EQ (i, k) can be used to ensure that the directional sounds of the spatial audio image conveyed by the parameterized spatial audio signal provided as input to the audio processing system 100 appear in their corresponding intended sound directions in the reproduced spatial audio image, thereby improving the spatial characteristics of the output audio signal 115.

[0106] In an example, the first partial processor 106 is arranged to delay the modified first signal component 111-1 by a predefined time delay so as to align the modified first signal component 111-1 with the modified second signal component 111-2 in time. Thus, if a delay is applied, the predefined time delay is selected so that it matches or substantially matches the delay resulting from the process performed by the second partial processor 108. In an example, the time delay may be applied to the first signal component 109-1 before performing an equalization process, for example according to equation (9), while in another example, the time delay may be applied to the modified first signal component 111-1, for example obtained by equation (9), before providing the signal to the signal combiner 110.

[0107] Still reference Figure 1, the second portion processor 108 may be configured to: obtain a modified second signal component 111-1 based on the second signal component 109-2 and in accordance with at least one sound reproduction characteristic 105; and provide the modified second signal component 111-2 to the signal combiner 110 for further processing therein. In this regard, the at least one sound reproduction characteristic 105 may include information specifying a corresponding acoustic propagation characteristic of each speaker from a device implementing the audio processing system 100 (e.g., the speaker 60 of the device 50). The processing performed by the second portion processor 108 is used to perform a crosstalk cancellation process for the second signal component 109-2. In this regard, the second signal component 109-2 obtained from the signal decomposer 104 may be provided as a binaural signal obtained according to equation (8b). Therefore, the left channel of the second signal component 109-2 is intended to be played back to the left ear of the listener, and the right channel of the second signal component 109-2 is intended to be played back to the right ear of the listener. The crosstalk cancellation process performed by the second spatial processor 108 is intended to provide the modified second signal component 111-2 as an audio signal, wherein the "leakage" of the audio signal content from the left channel of the second signal component 109-2 to the right ear of the listener located at a reference position relative to the device is reduced (e.g., substantially eliminated), and vice versa, wherein the "leakage" of the audio signal content from the right channel of the second signal component 109-2 to the left ear of the listener located at a reference position relative to the device is reduced (e.g., substantially eliminated). Therefore, when played back via the loudspeaker, the spatial characteristics produced by the audio content transmitted by the modified second signal component 111-2 are substantially similar to the spatial characteristics obtained by listening to the second signal component 109-2 via (binaural) headphones, thereby enabling high-quality spatial audio reproduction via the loudspeaker.

[0108] Figure 4 FIG. 1 is a block diagram of some components and / or entities of the second part processor 108 according to an example, including the filter gain H LL (b) H RL (b) H LR (b) and H RR (b) and for obtaining the corresponding filter gain H LL (b) H RL (b) H LR (b) and H RR (b) The filter gain determiner 122. The filter gain H LL (b) H RL (b) H LR (b) and H RR (b) may also be denoted as crosstalk cancellation gain or crosstalk cancellation filter, which may be provided as corresponding complex-valued gains for multiple frequency bins b (eg, for all frequency bins b). Filter gain HLL (b) H RL (b) H LR (b) and H RR (b) are typically based at least in part on measurements performed on loudspeakers of a device implementing the audio processing system 100, and therefore they may further take into account, at least to some extent, device-specific equalization of the loudspeakers.

[0109] The second partial processor 108 may be configured to create the left channel of the modified second signal component 111-1 as the left channel of the second signal component 109-2 multiplied by the filter gain H LL (b) multiply the right channel of the second signal component 109-2 by the filter gain H LR (b) and; and the modified right channel of the second signal component 111-2 is created as the left channel of the second signal component 109-2 multiplied by the filter gain H RL (b) multiply the right channel of the second signal component 109-2 by the filter gain H RR Herein, the left channel and the right channel of the second signal component 109-2 may respectively include, for example, x obtained according to equation (8b). 2 (1, b, n) and x 2 (2, b, n), and the left and right channels of the modified second signal component 111-2 for frequency bin b in channel i for time index n may be denoted as x′ 2 (i, b, n).

[0110] According to the example, the filter gain H LL (b) H RL (b) H LR (b) and H RR The corresponding gains of (b) are predefined and provided as part of the at least one sound reproduction characteristic 105, and the filter gain determiner 122 can therefore be configured to: read the filter gains from a memory in a device implementing the audio processing system 100; and provide these filter gains for use as the filter gains H LL (b) H RL (b) H LR (b) and H RR (b) To achieve crosstalk elimination filtering.

[0111] According to another example, for each speaker, at least one sound reproduction characteristic 105 includes a corresponding transfer function (from the corresponding speaker to the left ear of the user and the right ear of the user located at a reference position relative to the device implementing the audio processing system 100), and the filter coefficient determiner 122 can be configured to obtain the corresponding filter gain H based on the reference frequency response obtained in the at least one sound reproduction characteristic 105.LL (b) H RL (b) H LR (b) and H RR (b) As a non-limiting example in this regard, the filter gain determiner 122 may be configured to obtain the corresponding filter gain H according to the technique described in [4]. LL (b) H RL (b) H LR (b) and H RR (b). An overview of the technique is provided below.

[0112] According to [4], the filter gain for frequency bin b can be obtained as:

[0113] H(b)=(D(b)HD(b)+βI) -1 D(b) H A(b) (10)

[0114] Wherein, H(b) denotes a 2x2 complex-valued filter gain matrix in the transform domain, D(b) denotes a 2x2 transfer function matrix obtained as part of at least one sound reproduction characteristic 105, β denotes a real-valued scalar regularization coefficient, I denotes a 2x2 identity matrix, and A(b) denotes a 2x2 target transfer function matrix. Equation (10) can be "expanded" to:

[0115]

[0116] Among them, D LL (b) Denotes the reference transfer function from the left speaker to the left ear, D LR (b) Denote the reference transfer function from the left speaker to the right ear, D RL (b) Denote the reference transfer function from the right speaker to the left ear, D RR (b) Label the reference transfer function from the right speaker to the right ear, A LL (b) Label the target transfer function from the left speaker to the left ear, A LR (b) Label the target transfer function from the left speaker to the right ear, A RL (b) Label the target transfer function from the right speaker to the left ear, A RR (b) shows the transfer function from the right speaker to the right ear.

[0117] As mentioned before, the transfer function D LL (b) D RL (b) D LR (b) and D RR(b) are available in at least one sound reproduction characteristic 105, and they may be obtained based on experimental data, for example via a process involving recording a test signal using a microphone arrangement located at a reference position relative to a device implementing the audio processing system 100 (e.g., the device 50), and generating a test signal based on the corresponding test recording (e.g., as corresponding to the transfer function D LL (b) D RL (b) D LR (b) and D RR (b)), obtain the transfer function D LL (b) D RL (b) D LR (b) and D RR (b) In this context, the transfer function D can be recorded by slightly changing the position and / or orientation of the microphone used to capture the test signal. LL (b) D RL (b) D LR (b) and D RR (b) in order to account for small differences in the orientation and / or posture of the user relative to the device. The microphone arrangement mentioned in the foregoing may, for example, comprise a simulated human head located in a reference position relative to the device, wherein the simulated human head has corresponding microphones arranged in positions corresponding to the corresponding positions of the ears.

[0118] According to a non-limiting example, for crosstalk cancellation, the second part processor 108 may be configured to: set the target transfer function A from the left speaker to the left ear to LL (b) Set equal to the reference transfer function D from the left speaker to the left ear LL (b) A LL (b) = D LL (b) and the target transfer function A from the right speaker to the right ear RR (b) Set equal to the reference transfer function D from the right speaker to the right ear RR (b) A RR (b) = D RR (b) In another example, the second part processor 108 can be configured to transfer the target function A from the left speaker to the left ear by: LL (b) and the target transfer function A from the right speaker to the right ear RR Each of (b) is set equal to the unit value, i.e., A LL (b) = A RR (b) = 1 to provide crosstalk cancellation. To provide crosstalk cancellation, the target transfer function A from the left speaker to the right ear is LRThe amplitude of (b) is set to be smaller than the amplitude of the reference transfer function from the left speaker to the right ear D LR (b), and / or the target transfer function A from the right loudspeaker to the left ear RL The amplitude of (b) is set to be smaller than the reference transfer function A from the right speaker to the left ear. RL (b) has a magnitude, for example, such that |A LR (b)|<|D LR (b) | and / or | A RL (b)|<|D RL (b) |. In a non-limiting example, this can be accomplished by LR (b) = g LR D LR (b) to set the target transfer function A from the left speaker to the right ear LR (b) and / or pursuant to A RL (b) = g RR D RL (b) to set the target transfer function A from the right speaker to the left ear RL (b) is done, where, at least in some frequency subband b, 0 ≤ g LR <1 and / or 0≤g RL <1. As an example, the second part processor 108 may be configured to set the target transfer function A from the left speaker to the right ear LR (b) and the target transfer function A from the right speaker to the left ear RL (b) is set equal to "zero", that is, A LR (b) = A RL (b)=0.

[0119] Still referring to equations (10) and (11), according to an example, the regularization coefficient β can be set to a predefined constant value that is the same across frequency bins b. In another example, the regularization coefficient β can be set to a predefined frequency-dependent value that can be different across frequency bins b (e.g., according to a predefined frequency function), thereby avoiding the problem of applying the filter gain H LL (b) H RL (b) H LR (b) and H RR (b) results in crosstalk cancellation with a strong signal level increase (e.g., "boost") or signal level reduction (e.g., "cut") in certain frequency ranges of the corresponding frequency response. In this case, the constant regularization coefficient β in equation (10) can be replaced by a frequency bin-dependent regularization coefficient β(b), which has a specific value for the applied filter gain H LL (b) H RL (b) H LR (b) and HRR (b) A relatively high value and application of filter gain H at frequencies that would cause excessive changes in signal level (e.g., “surges” or “clips”) LL (b) H RL (b) H LR (b) and H RR (b) A relatively low value that does not cause excessive changes in signal level (e.g., "boosting" or "clipping").

[0120] Return to reference Figure 1 , the signal combiner 110 may be configured to combine the modified first signal component 111-1 and the modified second signal component 111-2 into a transform domain output audio signal 113 suitable for loudspeaker reproduction; and to provide the transform domain audio signal 113 to the inverse transform entity 112 for further processing therein. As an example in this regard, in the signal combiner 112, the transform domain output audio signal 113 may be obtained as a sum, an average value or another linear combination of the modified first signal component 111-1 and the modified second signal component 111-2.

[0121] Still reference Figure 1 , the inverse transform entity 112 may be arranged to: convert the transform domain output audio signal 113 into a (time domain) output audio signal 115; and provide the output audio signal 115 as an output audio signal of the audio processing system 100. In this regard, the inverse transform entity 112 is arranged to use an applicable inverse transform that reverses the time to transform domain conversion performed in the transform entity 102. As a non-limiting example in this regard, the inverse transform entity 112 may apply an inverse STFT or a (synthesized) QMF bank to provide the inverse transform.

[0122] Figure 5 A block diagram of some components and / or entities of an audio processing system 100′ is illustrated, which audio processing system 100′ can be used as a framework for various embodiments of the audio processing techniques described in the present disclosure. The audio processing system 100′ is a variation of the audio processing system 100 described above via a number of non-limiting examples, and therefore only the differences in its operation from the operation of the audio processing system 100 are described herein. The audio processing system 100′ includes a first subsystem 100a and a second subsystem 100b, which can be provided and / or operated separately from each other. In this regard, the first and second subsystems 100a, 100b can be implemented in the same device (e.g., device 50), or they can be implemented in different devices.

[0123] The first subsystem 100a comprises a transform entity 101 and a signal decomposer 104, each of which is arranged to operate as described hereinbefore in the context of the audio processing system 100. The first subsystem 100a further comprises: a first inverse transform entity 112-1 for transforming a first signal component 109-1 from a transform domain to a time domain, thereby providing a time domain first signal component 109-1'; and an inverse transform entity 112-2 for transforming a second signal component 109-2 from a transform domain to a time domain, thereby providing a time domain second signal component 109-2'. Each of the inverse transform entities 112-1, 112-2 is arranged to operate in the manner described hereinbefore in the context of the inverse transform entity 112, mutatis mutandis.

[0124] Following the ideas described above for the audio processing system 100, when the audio processing system 100' is applied to process the parameterized spatial audio signal for playback by two speakers, each of the first signal component 109-1' and the second signal component 109-2' has corresponding two audio channels, regardless of the number of channels of the transform domain audio signal 107. According to an example, the two channels of the first signal component 109-1' can be used to transmit spatial sound, wherein any directional sound in the front area of ​​the spatial audio image is arranged in the corresponding sound direction by applying the amplitude panning technology, and the two channels of the second signal component 109-2' can be used to transmit binaural spatial sound, which includes any directional sound outside the front area and any ambient sound of the spatial audio image.

[0125] The device implementing the first subsystem 100a may further be arranged to transmit the first and second signal components 109-1', 109-2' to the second subsystem 100b for further processing therein. The first and second signal components 109-1', 109-2' may be accompanied by an audio format indicator, which is used to identify the first and second signal components 109-1', 109-2' as originating from the first subsystem 100a. The transmission from the first subsystem 100a to the second subsystem 100b may, for example, include: the device implementing the first subsystem 100a is arranged to send this information to the device implementing the second subsystem 100b via a communication network or communication channel, and / or the device implementing the first subsystem 100a is arranged to store this information in a memory, which can then be read by the second subsystem 100b.

[0126] Thus, the device implementing the second subsystem 100b may be arranged to receive the first and second signal components 109-1', 109-2' via a network interface, or to read the first and second signal components 109-1', 109-2' from a memory. The second subsystem 100b comprises: a first transform entity 102-1 for transforming the first signal component 109-1' from the time domain to the transform domain, thereby recovering the frequency domain first signal component 109-1; and a second transform entity 102-2 for transforming the second signal component 109-2' from the time domain to the transform domain, thereby recovering the frequency domain second signal component 109-2. Each of the transform entities 102-1, 102-2 is arranged to operate in the manner described above in the context of the transform entity 102, with necessary modifications. The second subsystem 100b further comprises a first partial processor 106 , a second partial processor 108 , a combiner 110 and a transformation entity 112 , each of which is arranged to operate as described hereinbefore in the context of the audio processing system 100 .

[0127] Figure 6 A block diagram of some components and / or entities of an audio processing system 200 is illustrated, which audio processing system 200 may be used as a framework for various embodiments of the audio processing techniques described in the present disclosure. The audio processing system 200 is a variation of the audio processing system 100 described above via a number of non-limiting examples. The audio processing system 200 receives an input audio signal 101 and spatial metadata 103 (which together constitute a parameterized spatial audio signal), and the audio processing system 200 further receives at least one sound reproduction characteristic 105, which serves as a control input for controlling some aspects of the audio processing in the audio processing system 200. As with the audio processing system 100, the audio processing system 200 enables processing of the parameterized spatial audio signal into an output audio signal 215 that constitutes an audio output signal of the audio processing system 200.

[0128] Like the audio processing system 100, the audio processing system 200 also enables processing of a parameterized spatial audio signal for playback by a speaker of a device, wherein the processing is performed in accordance with at least one sound reproduction characteristic 105 of the device, and wherein the processing includes: rendering a first portion of a spatial audio image conveyed by the parameterized spatial audio signal using an amplitude panning process applied to an input audio signal in accordance with the spatial metadata and the at least one sound reproduction characteristic 105; and rendering a second portion of the spatial audio image using a crosstalk cancellation process applied to the input audio signal in accordance with the spatial metadata and the at least one sound reproduction characteristic 105.

[0129] As a brief overview, according to Figure 6The example audio processing system 200 shown in the figure comprises: a transform entity 102 for converting an input audio signal 101 from a time domain into a transform domain audio signal 107; a covariance matrix estimator 114 for obtaining a covariance matrix 119 and an energy metric 117 based on the transform domain audio signal 107 in view of the spatial metadata 103; a target matrix estimator 216 for obtaining an extended target covariance matrix 221 based on the spatial metadata 103 and the energy metric 117 in view of at least one sound reproduction characteristic 105, wherein the extended target covariance matrix 221 is used as a first part of the spatial audio image a target covariance matrix for both the sound included in the first part and the sound included in the second part of the spatial audio image; a mixing rule determiner 218, for obtaining an extended mixing matrix 223 based on the covariance matrix 119 and the extended target covariance matrix 221; a mixer 220, for obtaining a transform domain output audio signal 213 suitable for speaker reproduction based on the transform domain audio signal 107 in view of the extended mixing matrix 223; and an inverse transform entity 112, for converting the transform domain output audio signal 213 into a (time domain) output audio signal 215 for use as an output audio signal of the audio processing system 200.

[0130] In other examples, the audio processing system 200 may include Figure 6 entities other than those shown in, and / or Figure 6 Some of the entities depicted in can be combined with other entities while providing the same or corresponding functions. Figure 6 The entities shown in the figure are used to represent logical components of the audio processing system 200, which are arranged to perform corresponding functions but do not impose structural limitations on implementing the corresponding entities. Thus, for example, corresponding hardware components, corresponding software components, or corresponding combinations of hardware components and software components can be applied to implement the audio processing system 200 separately from other entities. Figure 6 Any of the entities shown in Figure 6 Any subcombination of two or more of the entities shown in Figure 6 All entities shown in .

[0131] The overall operation of the audio processing system 200, for example with respect to the characteristics of the input audio signal 101, the spatial metadata 103 and the at least one sound reproduction characteristic 105, as well as with respect to processing the input audio signal into a sequence of input frames and the implementation of the audio processing system 200 by the device 50, is similar to that described above with respect to the audio processing system 100. Furthermore, the corresponding operations of the transform entity 102, the covariance matrix estimator 114 and the inverse transform entity 112 are similar to those described above in the context of the audio processing system 100 (see Figure 1 and Figure 3) is similar to what is described in .

[0132] Still reference Figure 6 , the target matrix estimator 216 can be configured to: based on the spatial metadata 103 and the energy metric 117, in view of at least one sound reproduction characteristic 105, obtain an extended target covariance matrix 221, wherein the extended target covariance matrix 22 is used as a target covariance matrix for both the sound included in the first part of the spatial audio image and the sound included in the second part of the spatial audio image. The target matrix estimator 216 can be further configured to: provide the extended target covariance matrix 221 to the mixing rule determiner 218 for further processing therein. As in the case of the target matrix estimator 116, for the sake of clarity of description, an example of spatial audio parameters including one or more sound direction parameters defining the corresponding sound directions in the horizontal plane for one or more frequency subbands is described below, which can be easily analogized to further examples, which, additionally or alternatively, involve spatial audio parameters including one or more sound direction parameters defining the elevation angle of the sound direction for one or more frequency subbands.

[0133] The target matrix determiner 216 may be configured to calculate the first and second target covariance matrices 121-1, 121-2 according to the process described above in the context of the target matrix determiner 116. As an example in this regard, the target matrix determiner 216 may obtain a first target covariance matrix C representing the sound included in the first portion of the spatial audio image according to equation (4): 1 (k, n), and obtain a second target covariance matrix C representing the sound included in the second part of the spatial audio image according to equation (6): 2 (k, n). In addition, the target matrix determiner 216 may be further configured to be based on the first target covariance matrix C 1 (k, n), obtains further explanation via the use of the equalization gain g described above in the context of the first part processor 106 of the audio processing system 100 EQ (i, k) and is used to implement the characteristics of the device (eg, device 50) of the audio processing system 200 and the expanded first covariance matrix C′ 1 (k, n), for example, is obtained as:

[0134]

[0135] The target matrix determiner 216 may be further configured to be based on the second target covariance matrix C 2 (k), obtain the expanded second covariance matrix C′ 2 (k, n), for example, is obtained as:

[0136] C′ 2 (k, n) = H(b k，mid )C 2 (k,n)H H (b k，mid ) (13)

[0137] Among them, H(b k，mid ) denotes the 2x2 complex-valued filter coefficient matrix in the transform domain. In this regard, H(b k，mid ) is similar to H(b) defined in the context of equation (10) above, where the index b k，mid refers to the frequency bin closest to the center frequency of frequency subband k. The expanded first and second target covariance matrices C′ 1 (k, n), C′ 2 (k, n)) can be used to obtain the (combined) extended target covariance matrix 221, for example, as:

[0138] C′ y (k, n) = C′ 1 (k,n)+C′ 2 (k,n) (14)

[0139] The mixing rule determiner 218 may be configured to: obtain an extended mixing matrix 223 based on the covariance matrix 119 and the extended target covariance matrix 221; and provide the extended mixing matrix 223 to the mixer 220 for further processing therein. The operation of the mixing rule determiner 218 is similar to the operation of the mixing rule determiner 118 described above, except that a single mixing matrix is ​​obtained, which is suitable for processing both the sound included in the first part of the spatial audio image and the sound included in the second part of the spatial audio image in the mixer 220. In this regard, the mixing rule determiner 218 may be configured to: apply the formula provided in Appendix [6] to obtain, for time index n, for frequency subband k, based on the covariance matrix 119 (e.g., the covariance matrix C x (k, n)) and the expanded target covariance matrix 221 (eg, the expanded target covariance matrix C′ y (k,n)), generating a mixing matrix M(k,n), which can be used as an extended mixing matrix 223 for the corresponding time-frequency tile.

[0140] The mixer 220 may be configured to obtain a transform domain output audio signal 213 based on the transform domain audio signal 107 in view of the extended mixing matrix 223, and to provide the transform domain output audio signal 213 to the inverse transform entity 112 for further processing therein. The mixing process performed by the mixer 220 may include obtaining the transform domain output audio signal 213 as the product of the extended mixing matrix 223 and the transform domain audio signal 107, for example as:

[0141]

[0142] Here, k denotes the frequency sub-band in which the frequency bin b resides.

[0143] The inverse transform entity 112 may be arranged to: convert the transform domain output audio signal 213 into a (time domain) output audio signal 215; and provide the output audio signal 215 as an output audio signal of the audio processing system 200 as described hereinbefore.

[0144] In the above, the operation of the audio processing system 100, 100', 200 has been described by (implicitly and / or explicitly) referring to providing each of the first signal component 109-1, the second signal component 109-2, the modified first signal component 111-1, the modified second signal component 111-2, the transform domain output audio signal 213 and the output audio signal 215 (used as the output audio signal) as a respective two-channel signal in preparation for sound reproduction via two speakers. However, this is a non-limiting example selected for clarity and brevity of description, and the respective operation of each element of the audio processing system 100, 100', 200 is easily analogized to processing involving three or more channels to address a speaker arrangement including three or more speakers.

[0145] Furthermore, the description above relates to processing in a plurality of frequency subbands. For example, this may involve performing the processing described above for the audio processing system 100, 100', 200 for a set of frequency subbands that cover or substantially cover the spectrum represented by the parameterized spatial audio signal as a whole. In another example, the audio processing described above with reference to the audio processing system 100, 100', 200 may be performed in a predefined portion of the spectrum represented by the parameterized spatial audio signal, while for the remainder of the spectrum, an output audio signal 215 may be obtained using audio rendering techniques known in the art. In this regard, the predefined portion of the spectrum may include one or more predefined frequency subbands, for example, such that certain frequency subbands at the low end of the spectrum and / or at the high end of the spectrum are processed using an audio rendering mechanism known in the art, while frequency subbands in between are processed as described above with reference to the audio processing system 100, 100', 200.

[0146] In another example, some aspects of the audio processing described above with reference to the audio processing systems 100, 100', 200 may be replaced with different audio rendering techniques in predefined frequency subbands (e.g., in certain frequency subbands at the low end of the spectrum and / or at the high end of the spectrum). As an example, in the audio processing systems 100, 100', this may be achieved by omitting the crosstalk cancellation processing described above with reference to the second part processor 108 at certain frequency subbands at the low end of the spectrum and / or at the high end of the spectrum, while in the audio processing system 200, this may be achieved by preparing an extended target covariance matrix C' in the target matrix estimator 216 at certain frequency subbands at the low end of the spectrum and / or at the high end of the spectrum. y (k,n) when the crosstalk cancellation filter H(b k,mid ) (e.g., by adding the filter gain H RL (b) and H LR (b) is set to “zero” and the filter gain H is LL (b) and H RR (b) is set to unit value) to provide.

[0147] In another example, in the target matrix estimator 116, 216, regarding generating the second target covariance matrix 121-2 (eg, C 2(k,n)) can be omitted in certain frequency sub-bands at the low end of the spectrum and / or at the high end of the spectrum and replaced by an amplitude shifting technique. This can be achieved, for example, by replacing the HRTFh(k,n) in equation (6) with a suitable amplitude shifting gain and replacing the diffuse field covariance matrix in equation (6) with a unit matrix. Similarly, in this example, in the case of the audio processing systems 100, 100', the crosstalk cancellation processing described above with reference to the second part processor 108 should be omitted in certain frequency sub-bands at the low end of the spectrum and / or at the high end of the spectrum, while in the case of the audio processing system 200, an extended target covariance matrix C′ is prepared in the target matrix estimator 216 y (k,n) should omit the crosstalk cancellation filter H(b) at certain frequency sub-bands at the low end of the spectrum and / or at the high end of the spectrum. k,mid )’s contribution.

[0148] The logic elements of the audio processing system 100, 100', 200 may be arranged, for example, according to Figure 7 The method 300 operates as a method for processing an input audio signal 101 according to spatial metadata 103 for playback of a spatial audio signal in a device, wherein the processing is performed in accordance with at least one sound reproduction characteristic 105 of the device. For example, in view of the examples related to the operation of any one of the audio processing systems 100, 100' and / or 200 described above, the method 300 can be varied in many ways.

[0149] The method 300 comprises: obtaining an input audio signal 101, spatial metadata 103 and at least one sound reproduction characteristic 105 of the device, as shown in block 302. The method 300 further comprises: rendering a first portion of the spatial audio signal using a first type of playback process applied to the input audio signal 101 in accordance with the spatial metadata 103, wherein the first portion comprises directions of sounds within a front area, as shown in block 304; and rendering a second portion of the spatial audio signal using a crosstalk cancellation process applied to the input audio signal 101 in accordance with the spatial metadata 104 and in accordance with the at least one sound reproduction characteristic 105, wherein the second portion comprises directions of sounds not included in the first portion, and wherein the second type of playback process is different from the first type of playback process and involves crosstalk cancellation processing.

[0150] Figure 8An example of performance obtainable via operation of the audio processing system 100, 100', 200 (labeled "Proposed Output" in the illustration) is illustrated in comparison to previously known audio processing techniques involving binaural synthesis combined with general crosstalk cancellation techniques (labeled "HRTF+CTC Output" in the illustration). Figure 8 In the illustration of , the upper figure depicts the amplitude spectrum of the left channel and the lower figure depicts the amplitude spectrum of the right channel, which are obtained by processing an exemplary parametric audio signal (including an impulse as an input audio signal 101 and spatial metadata 103 defining a sound direction of "zero" degrees and a directional and total energy ratio of "one" for all frequency subbands), thus, the exemplary parametric audio signal models a sound source just in front of the assumed listening point under anechoic conditions. Figure 8 The amplitude response of the input audio signal is illustrated as a corresponding solid line curve, the amplitude response of the output audio signal 115, 215 obtained through processing by the audio processing system 100, 100', 200 is illustrated as a corresponding short dash line curve, and the amplitude response of the processed audio signal obtained using previously known audio processing techniques is illustrated as a corresponding dot-dash line curve.

[0151] like Figure 8 As shown in , an input audio signal 101 containing impulses just in front of an assumed listening point under anechoic conditions produces a flat amplitude spectrum in both the left and right channels. Due to the amplitude panning technique applied by the audio processing system 100, 100', 200 to the impulses in the first part of the spatial audio image (e.g., in the front region), the amplitude response of the output audio signal 115, 215 is substantially similar to the amplitude response of the input audio signal 101, except for a slight attenuation (approximately 3 dB) due to the application of the amplitude panning gain. Therefore, no coloration of the signal occurs, thereby enabling good timbre reproduction to the listener. In contrast, previously known audio processing techniques in which the input audio is processed into binaural signals (via the use of HRTFs) (which are further subjected to a crosstalk cancellation process) result in significant distortion in the amplitude spectrum (especially in the high end of the frequency spectrum in both channels), which results in coloration and timbre degradation of the reproduced sound, which can be avoided by using the audio processing system 100, 100', 200.

[0152] Fig. 9 A block diagram of some components of an exemplary apparatus 400 is shown. The apparatus 400 may include Fig. 9 The apparatus 400 may be used, for example, in implementing one or more components described above in the context of the audio processing systems 100, 100', 200. The apparatus 400 may implement, for example, the device 50 or one or more components thereof.

[0153] The apparatus 400 comprises a processor 416 and a memory 415 for storing data and computer program code 417. The memory 415 and a portion of the computer program code 417 stored therein may further be arranged to implement, together with the processor 416, at least some of the operations, processes and / or functions described above in the context of the audio processing systems 100, 100', 200.

[0154] The apparatus 400 includes a communication portion 412 for communicating with other devices. The communication portion 412 includes at least one communication means for enabling wired or wireless communication with other devices. The communication means of the communication portion 412 may also be referred to as corresponding communication components.

[0155] The device 400 may further include a user I / O (input / output) component 418, which may be configured (possibly together with the processor 416 and a portion of the computer program code 417) to provide a user interface for receiving input from a user of the device 400 and / or providing output to a user of the device 400 to control at least some aspects of the operation of the audio processing system 100, 100', 200 implemented by the device 400. The user I / O component 418 may include hardware components such as a display, a touch screen, a touch pad, a mouse, a keyboard, and / or an arrangement of one or more keys or buttons. The user I / O component 418 may also be referred to as a peripheral device. The processor 416 may be configured, for example, according to a portion of the computer program code 417 and possibly further according to user input received via the user I / O component 418 and / or according to information received via the communication portion 412 to control the operation of the device 400.

[0156] Although processor 416 is depicted as a single component, it can be implemented as one or more separate processing components. Similarly, although memory 415 is depicted as a single component, it can be implemented as one or more separate components, some or all of which can be integrated / removable, and / or can provide permanent / semi-permanent / dynamic / cached storage devices.

[0157] The computer program code 417 stored in the memory 415 may include computer executable instructions that, when loaded into the processor 416, control one or more aspects of the operation of the device 400. As an example, the computer executable instructions may be provided as one or more sequences of one or more instructions. The processor 416 is capable of loading and executing the computer program code 417 by reading one or more sequences of one or more instructions contained therein from the memory 415. The one or more sequences of one or more instructions may be configured to cause the device 400 to perform at least some of the operations, processes and / or functions described above in the context of the audio processing systems 100, 100', 200 when executed by the processor 416.

[0158] Thus, the device 400 may include at least one processor 416 and at least one memory 415 including computer program code 417 for one or more programs, the at least one memory 415 and the computer program code 417 being configured to, together with the at least one processor 416, cause the device 400 to perform at least some of the operations, processes and / or functions described hereinabove in the context of the audio processing systems 100, 100', 200.

[0159] The computer program stored in the memory 415 may, for example, be provided as a corresponding computer program product comprising at least one computer-readable non-transitory medium having stored thereon a computer program code 417 which, when executed by the apparatus 400, causes the apparatus 400 to perform at least some of the operations, processes and / or functions described hereinbefore in the context of the audio processing systems 100, 100', 200. The computer-readable non-transitory medium may include a memory device or a recording medium, such as a CD-ROM, a DVD, a Blu-ray disc, or another article of manufacture tangibly embodying the computer program. As another example, the computer program may be provided as a signal configured to reliably transmit the computer program.

[0160] Reference to a processor should not be understood as including only programmable processors, but also special purpose circuits such as field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), signal processors, etc. Features described in the preceding description may be used in combinations other than those explicitly described.

[0161] Although some functions have been described above with reference to certain features and / or elements, these functions may be performed by other features and / or elements whether described or not. Although features have been described with reference to certain embodiments, these features may also exist in other embodiments whether described or not.

Claims

1. A method for processing an input audio signal according to spatial metadata so as to play back a spatial audio signal in a device in dependence on at least one sound reproduction characteristic of the device, the method comprising: obtaining, by the device, the input audio signal and the spatial metadata; obtaining, by the device, the at least one sound reproduction characteristic of the device; rendering, by the device, a first portion of the spatial audio signal using a first type of playback process applied to the input audio signal in accordance with the spatial metadata, wherein the first portion includes directions of sounds in a front region of the spatial audio signal; and A second portion of the spatial audio signal is rendered by the device using a second type of playback process applied to the input audio signal based on the spatial metadata and based on the at least one sound reproduction characteristic, wherein the second portion includes sound directions not included in the first portion, and wherein the second type of playback process is different from the first type of playback process and involves crosstalk cancellation processing.

2. The method according to claim 1, wherein: The processing is performed separately in a plurality of frequency sub-bands.

3. The method according to claim 1, wherein: For one or more frequency subbands, the spatial metadata includes: The corresponding sound direction parameters, and The corresponding energy ratio parameter.

4. The method according to claim 1, wherein: The first portion comprises a first portion of a spatial audio image conveyed by the spatial audio signal, and wherein the second portion comprises a second portion of the spatial audio image, wherein the second portion is a substantially different portion than the first portion.

5. The method according to claim 4, wherein: The first portion represents directional sound of the spatial audio image in a front area of ​​the spatial audio signal, and The second portion represents directional sounds of the spatial audio image outside the front area and non-directional sounds of the spatial audio image.

6. The method according to claim 1, wherein: The front area includes a predefined range of sound directions.

7. The method according to claim 1, in, The at least one sound reproduction characteristic comprises a respective definition of a loudspeaker position relative to a reference position of the device, and The method comprises: defining a range of sound directions belonging to the front area based on the speaker position.

8. The method according to claim 1, wherein: The first type of playback process includes an amplitude translation process.

9. The method according to claim 1, wherein: The first type of playback processing includes at least one of the following: Amplitude translation process; and A process with one of the following: involving crosstalk cancellation processing arranged to provide a substantially less crosstalk cancellation effect than the crosstalk cancellation processing involved in the second type of playback process; and No crosstalk cancellation processing is involved.

10. The method according to claim 1, in, Rendering the first portion comprises: obtaining, using the first type of playback process, a first signal component representing the first portion based on the input audio signal in accordance with the spatial metadata, and The rendering of the second part comprises: obtaining a second signal component representing the second part based on the input audio signal using the second type of playback process according to the spatial metadata and according to the at least one sound reproduction characteristic.

11. The method according to claim 10, comprising: Based on the input audio signal, obtaining a covariance matrix and an energy measure; obtaining a first target covariance matrix representing the first portion and a second target covariance matrix representing the second portion based on the energy metric, the spatial metadata, and the at least one sound reproduction characteristic; Based on the covariance matrix and the first target covariance matrix, obtaining a first mixing matrix, the first mixing matrix when applied to the input audio signal produces a modified audio signal having a covariance matrix similar to the first target covariance matrix; Based on the covariance matrix and the second target covariance matrix, obtaining a second mixing matrix, which when applied to the input audio signal produces a modified audio signal having a covariance matrix similar to the second target covariance matrix; as well as The first signal component is obtained as a product of the input audio signal and the first mixing matrix, and the second signal component is obtained as a product of the input audio signal and the second mixing matrix.

12. The method according to claim 11, wherein: Obtaining the first target covariance matrix includes: obtaining an energy factor value indicating a degree of inclusion in the first portion based on a sound direction parameter included in the spatial metadata; Based on the sound direction parameter, determining a panning gain; and The first target covariance matrix is ​​obtained based on the energy metric, the translation gain, the energy factor value, and an energy ratio parameter included in the spatial metadata.

13. The method according to claim 11, wherein: Obtaining the second target covariance matrix includes: obtaining an energy factor value indicating a degree of inclusion in the first portion based on a sound direction parameter and the at least one sound reproduction characteristic included in the spatial metadata; determining a head related transfer function HRTF based on the sound direction parameter included in the spatial metadata; obtaining a diffuse field covariance matrix based on the HRTFs across a predefined range of sound directions; and The second target covariance matrix is ​​obtained based on the energy metric, the HRTF, the diffuse field covariance matrix, and an energy ratio parameter included in the spatial metadata.

14. The method according to claim 10, wherein: Obtaining the first signal component includes multiplying the first signal component using a gain value, the gain value being based on predefined equalization information included in the at least one sound reproduction characteristic.

15. The method according to claim 10, wherein: Acquiring the second signal component includes: obtaining a set of crosstalk cancellation gains based on a reference transfer function included in the at least one sound reproduction characteristic; and The set of crosstalk cancellation gains is applied to the second signal component.

16. The method according to claim 15, in, The reference transfer function comprises: a reference transfer function from the first loudspeaker to the left ear of a user located at a reference position relative to the device, a reference transfer function from the first loudspeaker to the right ear of the user at the reference position, a reference transfer function from a second loudspeaker to the left ear of the user at the reference position, and a reference transfer function from the second speaker to the user's right ear at the reference position; and Wherein, the set of crosstalk cancellation gains includes: a crosstalk cancellation gain from the first speaker to the left channel of the second signal component, a crosstalk cancellation gain from the first speaker to the right channel of the second signal component, a crosstalk cancellation gain of the left channel from the second speaker to the second signal component, and A crosstalk cancellation gain from the second speaker to the right channel of the second signal component.

17. The method according to claim 10, further comprising: An output audio signal for playback by the device is obtained as a combination of the first signal component and the second signal component.

18. The method according to claim 1, comprising: Based on the input audio signal, obtaining a covariance matrix and an energy measure; obtaining a first target covariance matrix representing the first portion and a second target covariance matrix representing the second portion based on the energy metric, the spatial metadata, and the at least one sound reproduction characteristic; acquiring an extended first target covariance matrix based on the first target covariance matrix and using a gain value based on predefined equalization information included in the at least one sound reproduction characteristic; Based on the second target covariance matrix and the crosstalk cancellation gain, obtaining an extended second target covariance matrix; Acquire a target covariance matrix as a combination of the extended first target covariance matrix and the extended second target covariance matrix; Based on the covariance matrix and the target covariance matrix, obtaining a mixing matrix which, when applied to the input audio signal, produces a modified audio signal having a covariance matrix similar to the target covariance matrix; as well as An output audio signal for playback by the device is obtained as a product of the input audio signal and a corresponding mixing matrix.

19. A computer program product having a computer readable program code stored thereon, wherein when the program code is run on a computing device, the computer readable program code is configured to cause the method according to any one of claims 1 to 18 to be performed.

20. An apparatus for processing an input audio signal according to spatial metadata for playback of a spatial audio signal in a device in dependence on at least one sound reproduction characteristic of the device, the apparatus comprising at least one processor and at least one memory comprising computer program code which, when executed by the at least one processor, causes the apparatus to: Obtaining the input audio signal and the spatial metadata; obtaining the at least one sound reproduction characteristic of the device; rendering a first portion of the spatial audio signal using a first type of playback process applied to the input audio signal in dependence upon the spatial metadata, wherein the first portion includes directions of sounds within a front region of the spatial audio signal; and A second portion of the spatial audio signal is rendered using a second type of playback process applied to the input audio signal based on the spatial metadata and based on the at least one sound reproduction characteristic, wherein the second portion includes sound directions not included in the first portion, and wherein the second type of playback process is different from the first type of playback process and involves crosstalk cancellation processing.

Images