Spatial audio reproduction by positioning at least a portion of a sound field - Patents.com

Abstract

To solve the conventional problem.SOLUTION: A sound field positioning apparatus which is based on a target direction is provided, configure to: obtain an audio signal, speaker setting information, and a processing path parameter including a target direction related to a processing path; process the audio signal on the basis of the processing path parameter; generate an incoherent audio signal from the audio signal; determine a panning gain on the basis of the target direction related to the processing path and the speaker setting information; and apply the panning gain to the incoherent audio signal to generate a multi-channel audio signal, the sound field positioning apparatus comprises means of combining the multi-channel audio signal to generate a combined panning gain application multi-channel audio signal.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present application relates to an apparatus and method for reproducing spatial audio by positioning at least a portion of a sound field, but is not limited to reproducing spatial audio by positioning at least a portion of a sound field in an augmented reality and / or virtual reality device. [Background technology]

[0002] Reverberation refers to the persistence of sound in a space after the actual sound source has stopped. Different spaces have different reverberation characteristics. To convey the spatial impression of the environment, it is important to reproduce the reverberation perceptually accurately. Room acoustics are often represented by a separately synthesized early reflection part and a statistical model of diffuse late reverberation. Figure 1 shows an example of a room impulse response with a direct sound 101 followed by discrete early reflections 103 with a direction of arrival (DOA) and a diffuse late reverberation 105 that can be synthesized without a specific direction of arrival. The delay d1(t) 102 in Figure 1 can be seen as the direct sound arrival delay from the sound source to the listener, and the delay d2(t) 104 can be seen as the delay from the sound source to the listener for one of the early reflections (in this case the first arriving reflection).

[0003] One way to recreate reverberation is to use a set of N loudspeakers (or virtual loudspeakers played binaurally using a set of Head-Related Transfer Functions (HRTFs)). The loudspeakers are more or less evenly spaced around the listener. These loudspeakers play mutually incoherent reverberation signals, giving the perception of a diffuse ambient reverberation.

[0004] Reverberations generated by different loudspeakers must be mutually incoherent. In the simple case, reverberations can be generated using different channels of the same reverberator, where the output channels are uncorrelated but have the same acoustic properties such as RT60 time and level (in particular the diffuse-to-direct or reverberant-to-direct ratios). Such uncorrelated outputs sharing the same acoustic properties can be obtained, for example, from the output taps of a feedback delay network (FDN) reverberator with appropriately adjusted delay line length, or from a reverberator based on using decaying uncorrelated noise sequences, by using different uncorrelated noise sequences in each channel. In this case, the different reverberation signals have effectively the same characteristics and the reverberation is generally perceived as similar for all directions. Summary of the Invention [Problem to be solved by the invention]

[0005] The embodiments of the present application aim to solve the problems associated with the prior art. [Means for solving the problem]

[0006] According to a first aspect, there is provided an apparatus for positioning at least a portion of a sound field based on a target direction, the apparatus comprising: acquiring at least one audio signal; acquiring speaker setup information; acquiring at least one processing path parameter for at least two processing paths, the at least one processing path parameter including a target direction associated with each of the at least two processing paths; processing the at least one audio signal based on the at least one processing path parameter for each of the at least two processing paths to generate a multi-channel audio signal; and for each processing path, means for extracting from the at least one audio signal at least two at least partially mutually incoherent audio signals. the multi-channel audio signal from each processing path is configured to generate a composite panning gain applied multi-channel audio signal; and means configured to generate a composite panning gain applied multi-channel audio signal from each processing path; and means configured to generate a composite panning gain applied multi-channel audio signal from each processing path; and to generate a composite panning gain applied multi-channel audio signal from each processing path; and

[0007] The at least one processing path parameter may further include at least one reverberation parameter associated with each of the at least two processing paths, and the means configured to generate the at least two at least partially mutually incoherent audio signals from the at least one audio signal may be configured to reverberate the at least one audio signal based on the at least one reverberation parameter to generate each of the at least two at least partially mutually incoherent audio signals.

[0008] The means configured to generate at least two at least partially mutually incoherent audio signals from at least one audio signal may be configured to decorrelate the at least one audio signal to generate each of the at least two at least partially mutually incoherent audio signals.

[0009] The means configured to determine at least two panning gains based on a target direction associated with the processing path and the speaker setup information may be configured to apply vector-based amplitude panning based on a target direction associated with the processing path and a direction associated with the speaker setup information.

[0010] The means may be further configured to generate an immersive audio signal based on processing the composite panning gain applied multi-channel audio signal.

[0011] The means configured to generate an immersive audio signal based on processing of the composite panning gains applied multi-channel audio signal may be configured to process, for each channel of the composite panning gains applied multi-channel audio signal, based on a head-related transfer function related to a direction relative to the loudspeaker associated with the channel, to generate a channel binaural panned processed audio signal, and to combine, for all channels, the channel binaural panned processed audio signals to generate the immersive audio signal.

[0012] The means configured to obtain speaker setting information may be configured to perform any of the following: receiving speaker setting information; determining speaker setting information; and obtaining predetermined or default speaker setting information.

[0013] The at least two at least partially mutually incoherent audio signals may be mutually incoherent audio signals.

[0014] According to a second aspect, there is provided a method for an apparatus for positioning at least a portion of a sound field based on a target direction, the method comprising: obtaining at least one audio signal; obtaining speaker setup information; obtaining at least one processing path parameter for at least two processing paths, the at least one processing path parameter including a target direction associated with each of the at least two processing paths; and processing the at least one audio signal based on the at least one processing path parameter for each of the at least two processing paths to generate a multi-channel audio signal, wherein for the processing, obtaining at least two at least partially mutually incoherent audio signals from the at least one audio signal. the at least partially mutually incoherent audio signal to generate a multi-channel audio signal; determining at least two panning gains based on target directions and speaker setting information associated with the processing paths; applying each of the at least two panning gains to an associated one of the at least partially mutually incoherent audio signals to generate an at least partially mutually incoherent audio signal with the at least two panning gains applied; and combining the at least partially mutually incoherent audio signals with the at least two panning gains applied to generate a multi-channel audio signal; and combining the multi-channel audio signals from each processing path to generate a composite panning gain applied multi-channel audio signal.

[0015] The at least one processing path parameter may further include at least one reverberation parameter associated with each of the at least two processing paths, and generating the at least two at least partially mutually incoherent audio signals from the at least one audio signal may include reverberating the at least one audio signal based on the at least one reverberation parameter to generate each of the at least two at least partially mutually incoherent audio signals.

[0016] Generating at least two at least partially mutually incoherent audio signals from at least one audio signal may include decorrelating the at least one audio signal to generate each of the at least two at least partially mutually incoherent audio signals.

[0017] Determining at least two panning gains based on a target direction associated with the processing path and the speaker setup information may include applying vector-based amplitude panning based on a target direction associated with the processing path and a direction associated with the speaker setup information.

[0018] The method may include generating an immersive audio signal based on processing the composite panning gain applied multi-channel audio signal.

[0019] Generating an immersive audio signal based on processing the composite panning gain applied multi-channel audio signal may include, for each channel of the composite panning gain applied multi-channel audio signal, processing the composite panning gain applied multi-channel audio signal based on a head-related transfer function associated with a direction of a loudspeaker associated with the channel to generate a channel binaural panned processed audio signal, and combining the channel binaural panned processed audio signals for all channels to generate the immersive audio signal.

[0020] Obtaining the speaker setting information may include any of receiving speaker setting information, determining speaker setting information, and obtaining predetermined or default speaker setting information.

[0021] The at least two at least partially mutually incoherent audio signals may be mutually incoherent audio signals.

[0022] According to a third aspect, there is provided an apparatus for positioning at least a portion of a sound field based on a target direction, the apparatus comprising at least one processor and at least one memory including computer program code, the at least one memory and the computer program code causing the apparatus to at least: obtain speaker setup information; obtain at least one processing path parameter for at least two processing paths, the at least one processing path parameter including a target direction associated with each of the at least two processing paths; and, for each of the at least two processing paths, process at least one audio signal based on the at least one processing path parameter to generate a multi-channel audio signal, the apparatus comprising: the at least two panning gains are determined based on target directions and speaker setting information associated with the processing paths; applying each of the at least two panning gains to an associated one of the at least partially mutually incoherent audio signals to generate an at least partially mutually incoherent audio signal with the at least two panning gains applied; and combining the at least partially mutually incoherent audio signals with the at least two panning gains applied to generate a multi-channel audio signal; and combining the multi-channel audio signals from each processing path to generate a composite panning gain applied multi-channel audio signal.

[0023] The at least one processing path parameter may further include at least one reverberation parameter associated with each of the at least two processing paths, and the apparatus adapted to generate at least two at least partially mutually incoherent audio signals from the at least one audio signal may be adapted to reverberate the at least one audio signal based on the at least one reverberation parameter to generate the at least two at least partially mutually incoherent audio signals, respectively.

[0024] An apparatus adapted to generate at least two at least partially mutually incoherent audio signals from at least one audio signal may be adapted to decorrelate the at least one audio signal to generate each of the at least two at least partially mutually incoherent audio signals.

[0025] The apparatus configured to determine at least two panning gains based on a target direction associated with the processing path and on the speaker setup information may be configured to apply vector-based amplitude panning based on the target direction associated with the processing path and on the direction associated with the speaker setup information.

[0026] The apparatus may be further adapted to generate an immersive audio signal based on processing the composite panning gain applied multi-channel audio signal.

[0027] An apparatus adapted to generate an immersive audio signal based on processing of a composite panning gain applied multi-channel audio signal may be adapted to process, for each channel of the composite panning gain applied multi-channel audio signal, based on a head-related transfer function associated with the direction of the loudspeaker associated with the channel, to generate a channel binaural panned processed audio signal, and to combine, for all channels, the channel binaural panned processed audio signals to generate the immersive audio signal.

[0028] A device adapted to obtain speaker setting information may be adapted to perform any of the following: receive speaker setting information; determine speaker setting information; and obtain predetermined or default speaker setting information.

[0029] The at least two at least partially mutually incoherent audio signals may be mutually incoherent audio signals.

[0030] According to a fourth aspect, there is provided an acquisition circuit configured to acquire at least one audio signal, the acquisition circuit configured to acquire speaker setup information, the acquisition circuit configured to acquire at least one processing path parameter for at least two processing paths, the at least one processing path parameter including a target direction associated with each of the at least two processing paths, and a processing circuit configured to process the at least one audio signal based on the at least one processing path parameter to generate a multi-channel audio signal, for each of the at least two processing paths, the processing circuit configured to extract from the at least one audio signal at least two at least partially mutually incoherent audio signals. an apparatus is provided that includes a processing circuit configured to: generate a panning gain-applied multi-channel audio signal from each processing path; determine at least two panning gains based on target directions and speaker setting information associated with the processing paths; apply each of the at least two panning gains to an associated one of the at least partially mutually incoherent audio signals to generate an at least partially mutually incoherent audio signal with the at least two panning gains applied; and combine the at least partially mutually incoherent audio signals with the at least two panning gains applied to generate a multi-channel audio signal; and a combining circuit configured to combine the multi-channel audio signals from each processing path to generate a composite panning gain applied multi-channel audio signal.

[0031] According to a fifth aspect, the apparatus includes at least: acquiring speaker setup information; acquiring at least one processing path parameter for at least two processing paths, the at least one processing path parameter including a target direction associated with each of the at least two processing paths; and processing at least one audio signal based on the at least one processing path parameter for each of the at least two processing paths to generate a multi-channel audio signal, wherein for each processing path, the apparatus generates at least two at least partially mutually incoherent audio signals from the at least one audio signal and processes the at least two at least partially mutually incoherent audio signals based on the target direction associated with the processing path and the speaker setup information. In one embodiment, a computer program product is provided that includes instructions [or a computer readable medium containing program instructions] for causing an apparatus to: determine two panning gains, apply each of the at least two panning gains to an associated one of the at least partially mutually incoherent audio signals to generate an at least partially mutually incoherent audio signal with at least two panning gains applied, and combine the at least partially mutually incoherent audio signals with the at least two panning gains applied to generate a multi-channel audio signal; and combine the multi-channel audio signals from each processing path to generate a composite panning gain applied multi-channel audio signal.

[0032] According to a sixth aspect, an apparatus is provided with at least the steps of: obtaining speaker setup information; obtaining at least one processing path parameter for at least two processing paths, the at least one processing path parameter including a target direction associated with each of the at least two processing paths; and processing at least one audio signal based on the at least one processing path parameter for each of the at least two processing paths to generate a multi-channel audio signal, wherein for each processing path, the apparatus generates from the at least one audio signal at least two at least partially mutually incoherent audio signals, and processing the at least two at least partially mutually incoherent audio signals based on the target direction and the speaker setup information associated with the processing paths. a panning gain-applied multi-channel audio signal; and combining the multi-channel audio signals from each processing path to generate a composite panning gain-applied multi-channel audio signal.

[0033] According to a seventh aspect, there is provided a system comprising: means for obtaining speaker setup information; means for obtaining, for at least two processing paths, at least one processing path parameter, the at least one processing path parameter including a target direction associated with each of the at least two processing paths; and means for processing at least one audio signal based on the at least one processing path parameter to generate a multi-channel audio signal, for each of the at least two processing paths, the means for processing comprising: means for generating at least two at least partially mutually incoherent audio signals from the at least one audio signal; and means for obtaining a target direction associated with the processing path. and speaker setup information, means for determining at least two panning gains based on the speaker setup information, means for applying each of the at least two panning gains to an associated one of the at least partially mutually incoherent audio signals to generate an at least partially mutually incoherent audio signal with the at least two panning gains applied, and means for combining the at least partially mutually incoherent audio signals with the at least two panning gains applied to generate a multi-channel audio signal, and means for combining the multi-channel audio signals from each processing path to generate a composite panning gain applied multi-channel audio signal.

[0034] According to an eighth aspect, an apparatus is provided with at least the steps of: acquiring speaker setup information; acquiring at least one processing path parameter for at least two processing paths, the at least one processing path parameter including a target direction associated with each of the at least two processing paths; and processing at least one audio signal based on the at least one processing path parameter for each of the at least two processing paths to generate a multi-channel audio signal, wherein for each processing path, the apparatus generates from the at least one audio signal at least two at least partially mutually incoherent audio signals and processes the at least two audio signals based on the target direction and the speaker setup information associated with the processing paths. and determining at least two panning gains based on the information received from the at least one processing path and applying each of the at least two panning gains to an associated one of the at least partially mutually incoherent audio signals to generate an at least partially mutually incoherent audio signal with the at least two panning gains applied, and combining the at least partially mutually incoherent audio signals with the at least two panning gains applied to generate a multi-channel audio signal; and combining the multi-channel audio signals from each processing path to generate a composite panning gain applied multi-channel audio signal.

[0035] An apparatus comprising means for performing the operations of the methods described above.

[0036] An apparatus configured to perform the operations of the method as described above.

[0037] A computer program comprising program instructions for causing a computer to carry out the method as described above.

[0038] A computer program product stored on the medium can cause an apparatus to perform the methods described herein.

[0039] The electronic device may include an apparatus as described herein.

[0040] The chipset may include an apparatus as described herein. [Brief description of the drawings]

[0041] For a better understanding of the present invention, reference will now be made, by way of example, to the accompanying drawings in which: [Figure 1] FIG. 1 shows a model of the room acoustics and the room impulse response. [Diagram 2] FIG. 2 illustrates generally an exemplary apparatus in which some embodiments may be implemented. [Diagram 3] FIG. 3 illustrates a flow diagram of the operation of an exemplary apparatus such as that shown in FIG. [Figure 4] FIG. 4 illustrates a schematic of an exemplary reverberation panner, such as that shown in FIG. 2, according to some embodiments. [Diagram 5] FIG. 5 is a flow diagram of the operation of an exemplary reverberation panner such as that shown in FIG. [Figure 6] FIG. 6 shows a graph of target direction, panning gain, and reverberation channel mapping illustrating target direction paths and the effect of implementing some embodiments. [Figure 7] FIG. 7 illustrates a schematic diagram of an example of a feedback delay network (FDN) reverberator in accordance with some embodiments. [Figure 8] FIG. 8 is a flow diagram of operations for adjusting parameters of a feedback delay network (FDN) reverberator according to some embodiments. [Figure 9] FIG. 9 is a flow diagram of operations for tuning parameters of three feedback delay network (FDN) reverberators according to some embodiments. [Figure 10] FIG. 10 is a diagram illustrating an implementation of an apparatus such as that shown in FIG. 2 within an exemplary application according to some embodiments. [Figure 11]FIG. 11 illustrates, in schematic form, an exemplary apparatus for microphone audio signals, in which some embodiments may be implemented. [Figure 12] FIG. 12 is a flow diagram of the operation of an exemplary apparatus as shown in FIG. [Figure 13] FIG. 13 is a schematic diagram of an exemplary decorrelator panner as shown in FIG. 12, according to some embodiments. [Figure 14] FIG. 14 shows a flow diagram of the operation of an exemplary decorrelator panner as shown in FIG. [Figure 15] FIG. 15 shows an example of an apparatus suitable for implementing the apparatus shown in the previous figures. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0042] In the following, a preferred apparatus and possible mechanisms for parameterizing and rendering reverberant audio scenes are described in more detail.

[0043] As mentioned above, reproducing reverberation from N incoherent loudspeakers (virtual or real) around a listener often reproduces the perception of diffuse reverberation. However, such implementations cannot output a properly perceived reverberation when the reverberation needs to be rotated, e.g. when the generated reverberation is direction-dependent.

[0044] This can be achieved, for example, in a noise convolution based reverberator by adjusting the decay rates of different channels based on the absorption characteristics of different wall materials, so that each channel has a different RT60 time.

[0045] Binaural playback, i.e. implementations where the loudspeakers are virtual loudspeakers created with HRTFs, is possible in the absence of head tracking, since the correct reverberation characteristics are perceived from the correct direction. However, problems arise when head tracking is implemented.

[0046] As an example of this, a listener can first face forward and show different reverberation times for the left / right and front / back directions. For example, in this situation the RT60 time for the front / back direction is RT60_front_back=1.2 seconds, and the reverberation time for the left / right direction is RT60_left_right=0.7 seconds. If the listener turns their head by 90 degrees, they would expect the reverberation to change to RT60=1.2 seconds reverberation for the left / right direction and RT60=0.7 seconds reverberation for the front / back direction. However, this may not be how the reverberation is implemented.

[0047] A simple solution would be to always select the HRTF that is closest to the desired direction for each reverberation channel after head tracking, but such an approach can introduce artifacts when switching between HRTFs.

[0048] Alternatively, one could perform head tracking and then interpolate HRTF filters between the desired directions for each reverberation channel, but the interpolation step would likely introduce perceptible artifacts.

[0049] An approach that avoids HRTF switching or interpolation would be to position the created reverberation based on head tracking information, for example using the commonly used Vector Based Amplitude Panning (VBAP) method. As a result, when the listener turns his head by 90 degrees, the reverberation that was originally in front is generated from -90 degrees. As a result, the correct characteristics of the reverberation can be reproduced from the correct direction according to the head tracking information. In this way, each virtual loudspeaker is spatialized with the same HRTF filter, so there is no artificiality due to HRTF filter switching or interpolation.

[0050] However, applying VBAP can cause other problems. VBAP positions the audio signal by playing it from 1-3 speakers according to the speaker settings and desired direction, and applies appropriate gain to each speaker. This is suitable for positioning normal audio signals and is often applied to spatial audio processing. However, VBAP has problems reproducing reverberation because it uses 1-3 speakers to generate each reverberation signal coherently. The reverberation generated in this way is perceived as a coherent, unspread reverberation, rather than a diffuse, surrounding sound.

[0051] According to some embodiments, the concept discussed herein relates to the reproduction of diffuse reverberant or ambient audio signals, and a method is proposed that allows the reproduction of rotatable diffuse reverberant or ambient audio, whose reverberant or ambient characteristics may be directionally dependent (i.e., different reverberant characteristics in different directions). This is achieved in some embodiments by generating two audio signals from one audio signal. These two audio signals are less coherent than simply comparing two identical copies of the original audio signal. Thus, in some embodiments, rendering of a (virtual) multi-channel signal for a large number of processing paths (at least three, typically 6-20 paths) is implemented by determining (for example using VBAP) at least two panning gains based on the target directions and the positions of the (virtual) loudspeakers in the (virtual) loudspeaker set, and obtaining for each of the determined gains at least partially mutually incoherent audio signals, in other words less coherent audio signals (and preferably mutually incoherent audio signals). For example, using the outputs of two reverberators that are adjusted to generate at least partially mutually incoherent (or non-coherent or mutually incoherent) reverberant audio signal outputs, or using a decorrelator that generates at least partially mutually incoherent (non-coherent or mutually incoherent) ambient audio signals. The objective of this embodiment is that the processing paths, each of which implements, for example, a reverberator and a decorrelator, generate mutually incoherent audio signals. However, for design and practical reasons, the output of each processing path may not generate a completely mutually incoherent audio signal, but rather a less coherent audio signal or an audio signal that is at least partially mutually incoherent.In the examples below, ideal mutually incoherent audio signals are generated, however it will be understood that the generation of less coherent audio signals, or at least partially mutually incoherent audio signals, is also encompassed by the same methods and apparatus.

[0052] These gains can be determined and applied to the corresponding captured (reverberant) signals to obtain a (reverberant) multi-channel signal.

[0053] The acquired (reverberant) multi-channel signals can then be combined in some embodiments to reproduce a composite (reverberant) multi-channel signal from the corresponding (virtual) loudspeakers.

[0054] A typical use case might employ an ambient virtual loudspeaker set (e.g., 16 virtual loudspeakers more or less evenly spaced around the listener) played back with HRTFs. In such a case, an embodiment might: Determine the initial target directions of the reverberator (e.g. the directions of the virtual loudspeakers, i.e. in this example, 16 target directions); For each target direction, determine three mutually incoherent variations (or incoherent ones) of the reverberation, the reverberation according to the desired reverberation characteristics for that direction; determining a rotated target orientation based on the head orientation and the initial target orientation; Using the present invention to recreate each of the three reverberation sets in a corresponding rotated target direction (e.g., using VBAP as a panning gain determination tool); It can be configured as follows.

[0055] As a result, the sound scenes produced by this embodiment can be perceived as surrounding, enveloping, and diffuse, and because the reverberation is updated based on the listener's orientation, the reverberant characteristics are perceived to originate from the correct direction.

[0056] 2, an embodiment of an exemplary apparatus 299 embodying the present invention is shown. The input to the system is an audio signal 200 to be reverberated.

[0057] The reverberation device shown in Fig. 2 has N reverberation panners 201. In Fig. 2, a first reverberation panner 2011, a second reverberation panner 2012, an Nth reverberation panner 201 N is specifically shown.

[0058] Each reverberation panner 201 is configured to obtain or receive an audio signal 200 , as well as speaker setup information 202 , target direction information 204 , and reverberation parameters 206 .

[0059] For example, the first reverberation panner 2011 is configured to acquire or receive the audio signal 200 and the loudspeaker setting information 202, as well as the first target direction information (or target direction 1) 2041 and the first reverberation parameter (or reverberation parameter 1) 2061.

[0060] The second reverberation panner 2012 is configured to acquire or receive the common audio signal 200 and the speaker setup information 202, as well as the second target direction information (or target direction 2) 2042 and the second reverberation parameters (or reverberation parameters 2) 2062.

[0061] The reverberation device shown in Fig. 2 has N reverberation panners 201. In Fig. 2, there are a first reverberation panner 2011, a second reverberation panner 2012, and an Nth reverberation panner 201 N2 is specifically shown. Each reverberation panner 201 is configured to obtain or receive an audio signal 200, loudspeaker setup information 202, target direction information 204, and reverberation parameters 206. For example, the first reverberation panner 2011 is configured to obtain or receive the audio signal 200 and the loudspeaker setup information 202, and further a first target direction information (or target direction 1) 2041 and a first reverberation parameter (or reverberation parameter 1) 2061. The second reverberation panner 2012 is configured to obtain or receive the common audio signal 200 and the loudspeaker setup information 202, and further a second target direction information (or target direction 2) 2042 and a second reverberation parameter (or reverberation parameter 2) 2062. Furthermore, the Nth reverberation panner 201 N 2 includes an audio signal 200, loudspeaker setting information 202, and Nth target direction information (or target direction N) 204. N , and the Nth reverberation parameter (or reverberation parameter N) 206 N The device is configured to obtain or receive the

[0062] The reverberation process is performed according to the reverberation parameters and the target direction. The input audio signal is inThe loudspeaker setting information 202 in some embodiments is a surround loudspeaker setting that can be used to generate a perception of ambient diffuse reverberation. The setting or loudspeaker configuration can be obtained based on any suitable method. For example, in some embodiments, the loudspeaker setting is pre-determined or default loudspeaker setting information. In some embodiments, the loudspeaker setting information is determined (e.g., a speaker calibration process is performed) or input (e.g., by user input). Furthermore, the setting or loudspeaker configuration can be in any suitable format. The loudspeaker setting information in some embodiments can define the number of loudspeakers and their relative orientation to the listener. Examples of loudspeaker setups or configurations are described, for example, in K. Hiyama, S. Komiyama, and K. Hamasaki, The Minimum Number of Loudspeakers and Its Arrangement for Reproducing the Spatial Impression of Diffuse Sound Field, AES 113th Convention, 2002, and C. Kirch, J Poppitz, T. Wendt, S. van der Par, and S. Ewert, Spatial Resolution of late Reverbouration in Virtual Acoustic Environments, Trends in Hearing (currently available on the website of the Carl von Ossietzky Universitat Oldenburg), posted in 2021.

[0063] This is an example of a speaker configuration or setup with 16 speakers arranged in three layers on the listener's plane: the first layer has eight speakers spaced at an azimuth angle of 45 degrees, the second layer has four speakers spaced at an elevation angle of 30 degrees and an azimuth angle of 90 degrees, and the third layer has four speakers spaced at an elevation angle of -30 degrees and an azimuth angle of 90 degrees. This can be expressed as azimuth and elevation angle values. Azimuth angle θ ls (i): 0, 45, 90, 135, 180, -135, -90, -45, 135, -135, -45, 45, 135, -135 degrees Elevation angle φ ls (i):0, 0, 0, 0, 0, 0, 30, 30, 30, -30, -30, -30, -30° where i is the channel of the speaker. The loudspeaker setup has N channels (in this example, 16 channels).

[0064] The reverberation parameters 206 of each of the reverberation panners (first reverberation parameter 2061, second reverberation parameter 2062, third reverberation parameter 2063, etc.) are respectively set in the target direction 1 (2041) (θ target (1,n),φ target (1,n)), target direction 2(2042)(θ target (2,n),φ target (2,n)), target direction 3(2043)(θ target (3,n),φ target (3,n)) (where the target direction may vary in time). The reverberation parameters and the target direction may be obtained by any suitable method or means. For example, in some embodiments, the initial target direction may be set to the direction of the speaker setup, i.e. θ initial (j)=θ ls (i) φ initial (j)=φ ls (i). where j is the index of the reverberation panner. Next, the target direction θ target (j,n),φ target(j,n) is the listener's orientation and the initial target direction θ initial (j),φ initial Based on (j), it can be determined, for example, using a quaternion or based on the method shown in MV Laitinen, “Binaural reproduction for directional audio coding”, M.Sc. Thesis, TKK, 2008.

[0065] Thus, the reverberation panner is configured to rotate the initial target direction based on the head orientation (available as a quaternion or Euler angles).

[0066] In some embodiments, the reverberation parameters 206 (such as the first reverberation parameter 2061, the second reverberation parameter 2062, and the third reverberation parameter 2063) are obtained as input, for example from an encoder input format file created by a content creator, and may include, in addition to the target direction, parameters such as the desired reverberation time RT60(f), the reverberation-to-direct ratio RDR(f) (or other equivalent representation such as direct-to-total emitted energy ratio), and / or the size of the virtual environment, and / or one or more materials.

[0067] In some embodiments, a first reverberation panner 2011, a second reverberation panner 2012, and an Nth reverberation panner 201 N is then configured to configure or initialize a reverberator based on the reverberation parameters to create a reverberated audio signal having desired reverberation characteristics defined by the reverberation parameters 206 (such as the first reverberation parameter 2061, the second reverberation parameter 2062, and the third reverberation parameter 2063).

[0068] In such an embodiment, the reverberation panner 201 pans the audio signal 201s based on the reverberation parameters 206. in (n) and generate a multi-channel signal according to a speaker setup 202 (or speaker configuration) in which the reverberant signals are positioned in a target direction 204 .

[0069] The output of the reverberation panner 201 is a set of panned reverberant signals 208s. pr,1 The first reverberation panner 2011 is configured to generate a first panned reverberation signal (or reverberation signal 1) 2081, the second reverberation panner 2012 is configured to generate a second panned reverberation signal (or reverberation signal 2) 2082, and the Nth reverberation panner 201 N is the Nth panned reverberation signal (or reverberation signal N) 208 N The panning reverberation signal 208 s pr,1 (n,i) is a multi-channel signal with N channels. An example of a reverberation panner is further described below with respect to FIG.

[0070] Therefore, as shown in FIG. in (n) is forwarded to the reverberation panner block. They work in a similar way, but with respect to the target direction θ target (j,n),φ target The (j,n) and reverberation parameters are independent for each of the reverberation panner blocks. Furthermore, the reverberations generated by different reverberation panner blocks are mutually incoherent. Therefore, the output of each reverberation panner block is a panned reverberation signal s pr,j (n,i), where j is the index of the reverberation panner path.

[0071] In this example, there are as many reverberation panners j as there are channels i in the multi-channel setup, although in other embodiments there could be a different number of panners.

[0072] The apparatus 299 further comprises a loudspeaker signal combiner 203. The loudspeaker signal combiner 203 combines the panned reverberation signal s pr,j (n,i) 208 and combine them into a single multi-channel signal, the panned reverberant signal 210. For example, apply:

number

[0073] The resulting panned reverberation signal 210 is forwarded to the HRTF processor 205, which then outputs the panned reverberation signal 210. i Each channel i of i is passed to.

[0074] Thus, for example, the panned reverberation signal 2101s pr The first channel of (n,1) is forwarded to a first HRTF processor 2051, which also processes a pair of head-related transfer functions “HRTF 1” (one filter for each ear) h hrtf (n,1,k) (where k is the HRTF channel, i.e., left or right) 2121. The orientation of the HRTF pair is determined by the loudspeaker setup θ ls (1), φ ls (1) corresponds to the direction of the corresponding channel in (1). Thus, for the example loudspeaker setup or configuration described above, this results in an azimuth angle of 0 degrees and an elevation angle of 0 degrees. In these embodiments, the HRTF processor 205 is configured to apply an HRTF filter (e.g., via convolution) and the resulting signal is the binaural panned reverberant signal s pr,bin (n,1,k) 214. Thus, the first channel output is the first channel, or channel 1, binaural panned reverberation signal 2141, which is passed to the binaural signal combiner 207.

[0075] The same process is repeated for the panned reverberation signal s pr For each channel in (n,i), the corresponding HRTF filter h hrtf (n,i,k) using the panned binaural reverberation signal s pr,bin (n,i,k) is forwarded to the binaural signal combiner 207.

[0076] In some embodiments, the apparatus 299 includes a binaural combiner 207 configured to receive the binaural panned reverberation signals and combine them into a single binaural signal, for example by applying the following formula:

number

[0077] Reverberant binaural signals rev,bin (n,k) 250 is the output of the processing. The reverberant binaural signal 250 is configured to produce the perception of surround diffuse reverberation. Furthermore, the reverberation characteristics are rendered based on the desired directional reverberation characteristics, which are applied based on head tracking data or any other directional target data.

[0078] With reference to FIG. 3, a flow diagram illustrating an example operation of the apparatus 299 of FIG. 2 is shown.

[0079] Thus, in FIG. 3, as indicated by step 301, the method includes obtaining an audio signal, speaker settings, target directions, and reverberation parameters.

[0080] Then, after obtaining the audio signal, speaker settings, target directions and reverberation parameters, we generate panned reverberation signals (multi-channel) for multiple paths, as shown by step 303 in FIG.

[0081] Next, as indicated by step 305 in FIG. 3, the panned reverberation signals may be combined to generate a loudspeaker channel panned reverberation signal.

[0082] Next, HRTF processing is performed on the channel panning reverberant signal, as indicated by step 307 in FIG.

[0083] The processed signals may then be combined to generate a reverberant binaural signal, as indicated by step 309 in FIG.

[0084] Then, as indicated by step 311 in FIG. 3, a reverberant binaural signal may be output.

[0085] With reference to Fig. 4, the reverberation panner 201 is shown in further detail and diagrammatically. The example shown in Fig. 4 is one of N blocks from the exemplary embodiment shown in Fig. 2, each of which is configured to have an individual target direction 204 and reverberation parameter input 206. Furthermore, in the example shown in Fig. 1, all reverberation panners of different paths j are configured to generate mutually incoherent reverberations. Otherwise, the operation of the reverberation panners of the different paths is identical.

[0086] In the example shown in FIG. in The (n) 200 is passed to a series of reverberators 401 (shown as a first reverberator 4011, a second reverberator 4012, and a third reverberator 4013). Each reverberator 401 is also configured to receive reverberation parameters 206 as an input.

[0087] Based on the reverberation parameters 206, the reverberator 401 is configured to generate a reverberated audio signal 402. For example, the first reverberator 4011 may generate a (first) reverberated audio signal 1 4021s, for example using a feedforward delay network (FDN) reverberator. rev It is configured to output (n,1).

[0088] The second reverberator 4012 outputs a (second) reverberant audio signal 1 4022s rev (n,2), the third reverberator 4013 outputs the (third) reverberant audio signal 3 4023s rev The 3-channel RTK is configured to output (n,3) signals, which have the same reverberation characteristics but are mutually incoherent.

[0089] Loudspeaker setting 202 θ ls (i),φ ls (i), and the target direction 204 θ target (j,n),φ target (j,n) are also inputs to the reverberation panner 201 and are forwarded to a panning gain determiner 405 configured to determine panning gains g(i,j,n). These panning gains can be determined using vector-based amplitude panning (VBAP), for example based on the method set out in V. Pulkki, "Virtual source positioning using vector base amplitude panning", J. Audio Eng. Soc., Vol. 45, pp. 456-466, June 1997, and in EP Application No. 18161580.8. In such an embodiment, each path j is directed to a (time-varying) target direction θ target (j,n),φ target For each channel i, we have a dedicated panning gain based on (j,n). For simplicity, in the following we consider only one time instant and one path, so in the following we denote the panning gain 404 as g(i).

[0090] The panning gain 404 g(i) is forwarded to the panning gain applier 403, which applies the panning gain 404 and the reverberant audio signal 402 s rev (n,l), where l is the reverberator path.

[0091] As the panning gains 404g(i) are created in VBAP, in some embodiments only 1 to 3 of them are non-zero. In the following example, it is assumed that there are exactly 3 channels (channels i1, i2, i3) that have non-zero gain at the first time instant, and the remaining channels have zero gain. In the following example, the non-zero channels are 3, 4, and 10.

[0092] For the first time, these can be assigned in any order (e.g., i1=3, i2=4, i3=10). Then, the reverberant audio signal 402 s rev (n,l) are assigned to each of these channels and processed with their respective gains. For example,

number

[0093] All other channels are set to zero.

number

[0094] And the panned reverberation signal 208 s pr,1 It is possible to output (n,i).

[0095] In this example, at the next time, θ target (j,n),φ target (j,n) change, and so do the panning gains 404g(i). However, the non-zero gains are still in the same channels, i.e., 3, 4, 10. In this example, we cannot freely choose the allocation of reverberation signals to the non-zero channels. Instead, the order of allocation remains the same, i.e., i1=3, i2=4, i3=10. This results in an output signal s pr,1 There is no discontinuity in (n,i) and good audio quality is maintained. If the allocation is changed, discontinuities in the audio signal may occur, resulting in clicks or snaps in the audio signal.

[0096] And at the next time θ target (j,n),φ target(j,n) changes again, and so does the panning gain 404 g(i). This time, assume that the non-zero gains are different channels, e.g. 3, 4, 14. Also, in this example, the assignment of reverberation signals to the non-zero channels cannot be freely chosen. Channels 3 and 4 must keep their respective reverberation signals to avoid discontinuities (and subsequent clicks and snaps). However, the third reverberation signal can be changed to a new channel. Thus, the new assignment is i1=3, i2=4, i3=14. Thus, the output is:

number

[0097] In this way, the selection of loudspeaker channels for each reverberation signal is performed such that the channels are changed only through "zero gain". In other words, for channels with gain greater than zero, the same reverberation is maintained. Furthermore, when one panning gain 404 value becomes zero and other channels are assigned gain values ​​greater than zero, a change in the channel mapping of the reverberation signal is performed. When using VBAP as the panning tool, this change is made even more smoothly.

[0098] With reference to FIG. 5, a flow diagram of the operation of the panner shown in FIG. 4 is shown in accordance with some embodiments.

[0099] For example, the method may include obtaining an audio signal, reverberation parameters, speaker settings, and a target direction, as shown in FIG. 5 by step 501 .

[0100] Then, as indicated by step 503 in FIG. 5, a reverberant audio signal is generated based on application of the reverberation parameters to the audio signal.

[0101] Further, in FIG. 5, as indicated by step 504, panning gain parameters may be determined based on the speaker setup and the target direction.

[0102] Thereafter, as indicated by step 505 in FIG. 5, a gain parameter may be applied to the reverberant audio signal to generate a panned reverberant signal.

[0103] The reverberant audio signal may then be output, as indicated by step 507 in FIG.

[0104] Figure 6 shows the target direction is θ target =0,φ target = 10, θ target =0,φ target 1 is an example graph showing an implementation of some embodiments in which g(10) smoothly changes from g(10) to g(10)=-10. The corresponding panning gains also change smoothly, with the panning gain for channel 10 smoothly going to zero and the panning gain for channel 14 smoothly increasing from zero (after the gain for channel 10 goes to zero). In this way, the channel mapping can be performed without any discontinuity and without extra processing at the instant in time when g(10) goes to zero. For other panning tools (or for abrupt changes in the "target direction"), smoothing over time can be used to slowly fade the old panning gains out to zero, and only then change the channel mapping, and then fade in the new panning gains (e.g., using a Hann windowed slope about 10 ms long, fading in half the first Hann window and fading out the second half).

[0105] With reference to Figure 7, an exemplary FDN reverberator is shown that may be employed as a reverberator 401 and used to generate D uncorrelated outputs. In the example shown in Figure 4, there are three such FDN reverberators 401, each configured to generate 15 uncorrelated outputs (D=15) for a total of 45 outputs. Thus, in this example, there are 15 reverberation panner paths j.

[0106] The exemplary FDN-reverberator implementation processes the reverberation parameters to obtain the coefficients GEQ d (GEQ1, GEQ2, ... GEQ D ), coefficients A and D of the feedback matrix 757, length m of the delay line 759 d (m1, m2, ...m D ) and the coefficients of the direct-to-reverberation filter 753 GEQ DDR The apparatus is configured to generate

[0107] In some embodiments, each attenuation filter GEQ d is implemented as a graphic EQ filter using M biquad IIR band filters. Thus, for octave bands M=10, the parameters of each graphic EQ include feedforward and feedback coefficients of 10 biquad IIR filters, gains of the biquad band filters, and an overall gain. In some embodiments, any suitable method can be implemented to determine the FDN reverberator parameters, for example, the method described in patent application GB2101657.1 can be implemented to derive FDN reverberator parameters that can reproduce a desired RT60 time of a virtual / physical scene.

[0108] The reverberator generates a very dense impulse response in the latter part using a network of delays 759, feedback elements (shown as gains 761, 757 combiners 755 and output gains 763). Input samples 751 are input to the reverberator, which generates reverberated audio signal components, which can be output.

[0109] The FDN reverberator includes multiple recirculation delay lines. A unitary matrix A757 is used to control the recirculation within the network. The attenuation filter 761, which in some embodiments may be implemented as a graphic EQ filter implemented as a cascade of second order cross-sectional IIR filters, can facilitate control of the rate of energy attenuation at different frequencies. The filter 761 is designed to attenuate the desired amount in decibels with each pulse passing through the delay line, resulting in the desired RT60 time.

[0110] The illustrated FDN reverberator exhibits D channel outputs by providing the outputs from each FDN delay line as independent outputs.

[0111] FIG. 8 is a flow diagram showing the adjustment of the parameters of one FDN reverberator. The parameters of this reverberator include each attenuation filter GEQ d The coefficients of the feedback matrix A and the length of the D delay lines m d In addition, the spread DC ratio filter GEQ DDR In these embodiments, each attenuation filter GEQ d is a graphic EQ filter using M biquad IIR band filters. Therefore, when the octave band M=10, the parameters of each graphic EQ consist of the feedforward and feedback coefficients of the 10 biquad IIR filters, the gains of the biquad band filters, and the overall gain.

[0112] Thus, in FIG. 8, as indicated by step 801, the method involves obtaining dimensions from the geometry of the virtual scene.

[0113] Next, as indicated in FIG. 8 by step 803, the method may further include determining a length of at least one delay line length based on the dimensions.

[0114] Next, as indicated by step 805 in FIG. 8, coefficients of at least one attenuation filter are determined based on the desired reverberation characteristics of the virtual scene.

[0115] Further, in FIG. 8, as indicated by step 807, the method is configured to determine coefficients of at least one diffuseness-to-directionality ratio control filter based on desired diffuseness-to-directionality ratio characteristics of the virtual scene.

[0116] The number of delay lines D can be adjusted depending on the quality requirements and the desired tradeoff between reverberation quality and computational complexity. In some embodiments, an efficient implementation with D=15 delay lines is used. This allows the feedback matrix coefficients A to be defined in terms of Galois sequences that facilitate an efficient implementation, as shown in the method described by Rocchesso: Maximally Diffusive Yet Efficient Feedback Delay Networks for Artificial Reverberation, IEEE Signal Processing Letters, Vol. 4, No. 9, Sep 1997.

[0117] The length m of the delay line d d can be determined based on the dimensions of the virtual room. The virtual room can be any suitable cubic shape. Furthermore, in acoustics, these cubes are referred to as "shoebox rooms". For example, a shoebox room can be defined with dimensions xDim, yDim, zDim. If the room shape is not a "shoebox", a "shoebox" can be fitted into the room and the dimensions of the fitted shoebox can be used for the delay line length. Alternatively, the dimensions can be taken as the three longest dimensions of the non-shoebox shaped room, or in any other suitable manner.

[0118] In some embodiments, the delay is set proportional to the standing wave resonant frequency in the virtual or real room. dcan furthermore be mutually prime.

[0119] In some embodiments, the attenuation filter coefficients of the delay line are adjusted to produce a desired amount of attenuation in decibels with each signal recirculation through the delay line such that a desired RT60 time is obtained. This is performed in a frequency-dependent manner to ensure an appropriate rate of attenuation of signal energy at specific frequencies.

[0120] The input to the encoder may, in some embodiments, provide a desired RT60 time per specified frequency f, denoted as RT60(f). For frequency f, the desired attenuation per signal sample is calculated as attenuationPerSample(f)=-60 / (samplingRate*rt60(f)). d The attenuation in decibels for a delay line of f is attenuationDb(f)=m d *attenuationPerSample(f).

[0121] In some embodiments, the RT60 time can be different for different spatial directions, in which case the absorption filter of the delay line is adjusted based on the RT60 time of the target direction to which the delay line is panned.

[0122] The attenuation filters in some embodiments are designed as cascade graphic equalizer filters as described in V. Valimaki and J. Liski, “Accurate cascade graphic equalizer”, IEEE Signal Process.Lett., Vol. 24, No. 2, pp. 176-180, February 2017 for each delay line. The outlined design procedure takes as input a set of command gains in octave bands. There are also methods for similar graphic EQ structures that can support a third octave band, increasing the number of biquad filters to 31 and providing better fit to detailed target responses as described in Third-Octave and Bark Graphic-Equalizer Design with Symmetric Band Filters, https: / / www.mdpi.com / 2076-3417 / 10 / 4 / 1222 / pdf.

[0123] With reference to FIG. 9, a flow diagram is shown illustrating a method for tuning parameters of three FDN reverberators that produce uncorrelated outputs. In these embodiments, this involves tuning the parameters of one reverberator based on the unmodified virtual room geometry, and tuning the parameters of the second and third FDN reverberators with the modified virtual room geometry. For example, reverberator 1 is parameterized using the method shown in FIG. 8 using virtual room dimensions xDim, yDim, zDim. The second FDN reverberator is tuned using modified virtual room dimensions of 1.2*xDim, 1.2*yDim, 1.2*zDim. The third FDN reverberator is tuned using modified virtual room dimensions of 0.8*xDim, 0.8*yDim, 0.8*zDim.

[0124] Thus, for example, in FIG. 9, as indicated by step 901, the method may obtain the dimensions, RT60, and optionally the spread-to-rectangular ratio characteristics of the environment.

[0125] Next, in FIG. 9, as indicated by step 903, the method includes configuring a first reverberator for generating reverberation according to the environmental characteristics.

[0126] Next, at least one dimension of the environment is modified, as indicated by step 905 in FIG.

[0127] After modifying the environment, a second reverberator is configured to generate reverberation according to the modified environment characteristics, as indicated by step 907 in FIG.

[0128] Next, at least a second dimension of the environment is altered, as indicated by step 909 in FIG.

[0129] Next, as indicated by step 911 in FIG. 9, a third reverberator is configured to generate reverberation further according to the modified environmental characteristics.

[0130] Since the FDN delay line lengths m1 to mD are adjusted based on the shape of the scene, changing the shape of the scene results in different lengths of the delay lines in each reverberator, making the outputs uncorrelated.

[0131] In some embodiments, all delay lines across all FDN reverberators are adjusted to have mutually prime lengths to ensure mutually uncorrelated outputs. This can be done, for example, by having the first created FDN report the delay line length it is using, and creating a second FDN such that it does not use any of the delay line lengths used by the first FDN. A third FDN is created in such a way that it does not use any of the delay line lengths used by the first or second FDNs.

[0132] 10 is a diagram illustrating an example implementation scenario according to some embodiments, which corresponds to an envisioned use case of the MPEG-I Audio Phase 2 standard that is expected to support audio rendering in six degree of freedom (6DoF) scenarios for virtual reality (VR) and augmented reality (AR).

[0133] The inputs to the encoder are one or more audio signals 200 and a virtual scene description 282. The virtual scene description parameters 282 in some embodiments include virtual scene geometry, which may be defined as a triangle mesh format, (mesh) acoustic material properties, (mesh) reverberation properties, audio object positions (which may be defined as Cartesian coordinates in some embodiments). In other words, the virtual scene description 282 includes a description of an acoustic environment with desired reverberation parameters such as RT60 time, diffuse to total energy ratio, and scene geometry. These parameters are obtained by the encoder 1001.

[0134] The encoder 1001 comprises a reverberation parameter obtainer 1003 configured to derive reverberation parameters which are passed to a reverberation panner parameter obtainer 1005 and a reverberation parameter obtainer configured to determine (using the method described above) the reverberation panner parameters. The method derives the reverberator parameters based on the scene geometry and reverberation characteristics. If the reverberation characteristics are not provided, they can be obtained via acoustic simulation using the geometry and material properties of a virtual scene. Geometry or wave-based virtual acoustic simulation methods or a combination thereof can be used. For example, wave-based virtual acoustic simulation can be used for low frequencies and geometric acoustic methods for high frequencies. The method described in UK patent application GB2101657.1 can be used to derive the reverberator parameters.

[0135] The parameters of the reverberation panner (delay line length, delay line attenuation filter coefficient, diffusion pair filter coefficient, and target direction) can then be passed to a reverberation panner parameter encoder 1007 configured to encode the parameters. The encoded reverberation panner parameters can then be passed to a bitstream encoder 1009, which is then configured to generate a bitstream 220 together with the audio signal 200. In other words, other contents of the virtual scene description can also be encoded into the bitstream. The audio signal is encoded with MPEG-H 3D audio and multiplexed into the bitstream.

[0136] The decoder / renderer 1011 is configured to receive a bitstream 220 description of the virtual scene content, rendering parameters such as reverberation panner parameters, and an audio signal.

[0137] In some embodiments, the decoder / renderer 1011 includes a bitstream decoder 1031 configured to decode / separate and output the "encoded" description of the virtual scene content, rendering parameters such as reverberation panner parameters, and audio signals from the bitstream.

[0138] In some embodiments, the decoder / renderer 1011 includes a reverberation panner parameter decoder 1033 configured to obtain the encoded reverberation panner parameters from the bitstream decoder 1031, create reverberation panner parameters and output these to a reverberation panner creator 1035.

[0139] The decoder / renderer 1011 further comprises a reverberation panner creator 1035 configured to receive the decoded reverberation panner parameters and initialize the reverberation panner 201. In this example, only one reverberation panner 201 is shown, but as mentioned above, multiple reverberation panners may be employed, each with its own reverberation parameters and target direction.

[0140] The reverberation panner 201, loudspeaker signal combiner 203 and HRTF processor 205 may then be implemented as described above based on the output of the head direction determiner 1099 and the loudspeaker setting or configuration information from the bitstream decoder 1031. In other words, the reverberation panner 201, loudspeaker signal combiner 203 and HRTF processor 205 may be used to render an audio signal having desired reverberation characteristics. Note that in this example, the rotation of the target direction based on the head tracking information is performed inside the reverberation panner 201, as opposed to outside the panner in the exemplary embodiment described with respect to Figures 2-5.

[0141] Further, the decoder / renderer 1011 comprises a direct sound processor 1039 configured to receive the decoded audio signal from the bitstream decoder 1031 and configured to perform any direct sound processing such as air absorption and distance vs. gain attenuation, which generates a direct sound component together with a head orientation determination to an HRTF processor 1041 which may be passed together with the reverberant component from the HRTF processor 205 to a binaural signal combiner 207. The binaural signal combiner 207 is configured to combine the direct sound and reverberant sound parts to generate a suitable output (e.g. for headphone playback).

[0142] Also, although not shown, various other audio processing methods can be applied, such as early reflection rendering in combination with the proposed method.

[0143] In some embodiments, the reverberation panner parameters may be partially or fully derived by the renderer, as may be the case, for example, in AR audio rendering, where the renderer receives a description of the listening space along with the desired reverberation parameters.

[0144] Approaches such as those described in the above embodiments can also be configured to address the problem of reverberators and reverberant spatialization solutions in rendering reverberation from a large number of channels in a computationally efficient manner. A simple way to obtain high-quality reverberation that really engulfs the user is to tune a large reverberator with, for example, 45 output channels. However, if such a reverberator is implemented as an FDN reverberator, the feedback matrix needs to implement feedback over 45 delay lines for each sample, which makes the feedback matrix computations huge.

[0145] In the embodiments described herein, it is possible to employ three FDN reverberators, each with only 15 channels, which can be run in parallel on modern processor architectures, performing fast feedback matrix calculations separately without actually performing the matrix calculations. Furthermore, whereas spatialization of 45 reverberator output channels currently requires 45 virtual loudspeakers and 45 HRTF filters, the embodiments described herein only require computing gains for 15 virtual loudspeakers and performing spatialization with 15 HRTF filters.

[0146] In some embodiments, the apparatus and methods described herein can be employed without significant creative insight to generate other incoherent content (besides reverberation). For example, ambience sounds can be rendered circumstantial and enveloping using the above-described embodiments. In this example, the reverberator can be replaced by a decorrelator. Also, in some embodiments, the reverberation parameters can be omitted. Instead, different microphone signals can be routed to different reverberation panner paths j. For example, the microphones can be mounted on the surface of an acoustically shaded device. As a result, the different microphones can capture the ambience (and / or reverberation) in a direction-dependent manner. Thus, in effect, the same effect is achieved as providing direction-dependent reverberation parameters.

[0147] Figure 11 is a schematic diagram of an example embodiment, and Figure 12 is a flow diagram showing the operation of the example embodiment, which is similar to the example shown in Figure 2, with only the differences being detailed.

[0148] The input to the device shown in FIG. 11 is, instead of a single audio signal 200, multiple microphone signals 1100 (these are microphone signal 1 11001, microphone signal 2 11002, microphone signal N 1100, N These input microphone signals are forwarded to the associated decorrelator panner 1101. Thus, microphone signal 1 11001 is forwarded to decorrelator panner 11011, microphone signal 2 11002 is forwarded to decorrelator panner 11012, microphone signal N 1100 N is the decorrelator Panner 1101 N will be forwarded to.

[0149] Each decorrelator panner 1101 (instead of the reverberation panner of FIG. 2) is configured to receive parameters of a loudspeaker configuration 1102 and a target direction 1104, but does not receive reverberation parameters. Thus, for example, as shown in FIG. 11 , there is a first decorrelator panner 11011 configured to receive a loudspeaker configuration 1102 and a first target direction (target direction 1) 11041, a second decorrelator panner 11012 configured to receive a loudspeaker configuration 1102 and a second target direction (target direction 2) 11042, a third decorrelator panner 11013 configured to receive a loudspeaker configuration 1102 and an Nth target direction (target direction N) 11043, and a fourth decorrelator panner 11014 configured to receive a loudspeaker configuration 1102 and an Nth target direction (target direction N) 11044. N 11. An Nth decorrelator panner 1101 configured to receive N There is.

[0150] The target direction 1104 in some embodiments may be derived from the head orientation and the respective directions of the microphones in the array. N Each of the above-mentioned reverberation panners 2011, 2012, and 201 N1103, but instead of reverberating the input microphone signal, it is configured to decorrelate the microphone audio signal to generate a panning ambience signal (multi-channel) 1108. For example, a first panning ambience signal (panned ambience signal 1) 11081 from a first decorrelator panner 11011, a second panning ambience signal (panned ambience signal 2) 11082 from a second decorrelator panner 11012, and an Nth decorrelator panner 11013, N Nth panned ambience signal (panned ambience signal N) 1108 from N can be passed to the speaker signal combiner 1103.

[0151] The loudspeaker signal combiner 1103 includes decorrelator panners 11011, 11012 and 1101 N The output of each of the panned ambience signals 11081, 11082, and 1108 N and further generating panned ambience signals 1110 for selected channels 1 to N (11101, 11102, 1110 in FIG. 11). N ) and pass it to the HRTF processor 1105.

[0152] The HRTF processors are configured to obtain the HRTFs 212 of each HRTF processor 1105 and are configured to generate a binaural panned ambience signal 1114 from the processed panned ambience signals, which are passed to the binaural signal combiner 1107.

[0153] The binaural signal combiner 1107 receives the binaural panned ambience signals 1114 and generates the ambience binaural signal 1150 based on them. As a result, the ambience binaural signal 1150 creates the perception of a surround, enveloping ambience. Furthermore, the directional characteristics of the different microphones are preserved and reproduced from the correct direction, so that the directional characteristics of the ambience are generated in the correct direction.

[0154] With reference to FIG. 12, a flow diagram illustrating an exemplary operation of the apparatus 1199 of FIG. 11 is shown.

[0155] Thus, in FIG. 12, as indicated by step 1201, the method includes obtaining a microphone audio signal, a speaker setting, and a target direction.

[0156] Then, after obtaining the microphone audio signal, the speaker settings and the target direction, generate a panned ambience signal (multi-channel), as shown by step 1203 in FIG.

[0157] Next, as indicated by step 1205 in FIG. 12, the panned ambience signals may be combined to generate panned ambience signals for the loudspeaker channels.

[0158] Next, HRTF processing is performed on the channel panned ambience signal, as indicated by step 1207 in FIG.

[0159] The processed signals can then be combined to generate an ambience binaural signal, as shown by step 1209 in FIG.

[0160] Then, as indicated by step 1211 in FIG. 12, the ambience binaural signal can be output.

[0161] Figure 13 illustrates diagrammatically an exemplary decorrelator panner (e.g., decorrelator panner 11011) as shown in Figure 11. It is otherwise configured to operate similarly to the reverberation panner illustrated in Figure 4, but with the reverberator 401 replaced by a decorrelator 1301 configured to produce mutually incoherent decorrelated signals. In these embodiments, there are no reverberation parameter inputs, instead output from each of the decorrelators is a decorrelated audio signal 1302 that is passed to a panning gain applier 1303. 13 shows a first decorrelator 13011 receiving the microphone signal 11001 and outputting a first de-correlated audio signal (decorrelated audio signal 1) 13021, a second decorrelator 13012 receiving the microphone signal 11001 and outputting a second de-correlated audio signal (decorrelated audio signal 2) 13022, a third decorrelator 13013 receiving the microphone signal 11001 and outputting an N-th de-correlated audio signal (decorrelated audio signal N) 1302 N An N-th decorrelator 1301 that outputs N Shows.

[0162] Further shown is a panning gain determiner 1305 configured to receive the loudspeaker settings 1102 and the target directions 1104 , generate and pass panning gains 1304 to the panning gain applier 1303 .

[0163] The panning gain applier 1303 is a filter that includes the decorrelators 13011, 13012, and 1301. N and applying a panning gain to combine them to produce a panned and de-correlated signal 11081.

[0164] With reference to FIG. 14, a flow diagram of the operation of the panner shown in FIG. 13 is shown in accordance with some embodiments.

[0165] For example, in FIG. 14, as indicated by step 1401, the method may include obtaining a microphone audio signal, a speaker setting, and a target direction.

[0166] Next, in FIG. 14, a decorrelated audio signal is generated from the microphone audio signal 1100 as indicated by step 1403 .

[0167] Additionally, in FIG. 14, as indicated by step 1404, panning gain parameters may be determined based on the speaker setup and the target direction.

[0168] Thereafter, as indicated by step 1405 in FIG. 14, a gain parameter may be applied to the decorrelated audio signal to generate a panned ambience signal.

[0169] Then, the ambience audio signal can be output, as indicated by step 1407 in FIG.

[0170] Note that in the examples described herein, several reverberation panners or reverberators are shown, but they can be implemented inside a single reverberation panner or reverberator. For example, the FDN reverberator feedback matrix can be configured to have a block structure whose blocks correspond to the desired feedback matrices of the smaller FDN instances. The actual implementation can then be a single FDN that jointly implements the smaller FDN with block feedback matrices and appropriate delay lines.

[0171] Furthermore, in some embodiments, the delay line length of the FDN reverberator can be set in a different way than described herein. For example, one further option is to make the delay length proportional to the mean free path length in the virtual room. In some embodiments, the dimensions of the virtual room are mapped to the dimensions of another room. For example, one of the rooms can have dimensions with a ratio of [1, 1.6, 2.56]. In these embodiments, the shortest dimension of the input virtual room is used as is, corresponding to a ratio of 1, and the other two dimensions are calculated based on a ratio of 1.6 and 2.56 times the shortest input room dimension. The delay line length is then adjusted based on these calculated dimensions of the other room.

[0172] In some embodiments, other dimensional ratios may be present. For example, the following dimensional ratios may be used:

[0111] [1 1.14 1.39] [1 1.26 1.59] [1 1.28 1.54] [1 1.3 1.9] [1 1.4 1.9] [1 1.5 2.5] [1 1.6 2.33] From this, one set of dimension ratios can be selected.

[0173] Additionally, in some embodiments, the different dimension ratios may be stored in the renderer and an index indicating which one to use may be sent from the encoder to the renderer.

[0174] The delay line damping filters of the FDN reverberator may further have different implementations, such as parallel second order section filters, any other combination of IIR filters, or FIR filters, in some embodiments.

[0175] The reverberator may be implemented in any suitable manner. For example, in some embodiments, the reverberator may be implemented using convolution with a decaying noise sequence. In this approach, a multi-channel reverberator may be created by initializing N uncorrelated band-pass noise sequences that are multiplied with a desired decay envelope based on the desired RT60 time of each band. The output signal may be created by convolving the input signal with each band-pass noise sequence. Since such a reverberator does not depend on the geometry of the virtual scene, three reverberators may be initialized by using different uncorrelated noise sequences in all bands of all reverberators.

[0176] In the above exemplary embodiment, the target direction θ target (j,n),φ target (j,n), and all subsequent processing, was performed with audio sample temporal accuracy. In some embodiments, the target direction and / or any other variables (such as panning gain) can be determined with any other time resolution (e.g., every 10 ms), and the required variables can then be suitably interpolated.

[0177] In the illustrated embodiment, VBAP was used to determine the panning gains. Since VBAP generates up to three non-zero gains, up to three reverberators are required for each reverberation panner. In some embodiments, a different method for panning gain determination may be used. Thus, in some embodiments, the number of reverberators may be any suitable number accordingly. For example, if the panning tool generates four non-zero gains, four reverberators per panner may be employed.

[0178] In some embodiments, the tuning of the reverberator parameters can be split between the encoder / renderer, where the parameters of the first reverberator are tuned in the encoder and encoded into the bitstream. In the renderer, the parameters of the first reverberator are decoded and then modified to create the second and third reverberators. Examples of such modifications include adjusting the delay line length mD and damping filter coefficients GEQ of the first reverberator to obtain the parameters of the second and third reverberators. d The delay line length can be changed to be shorter or longer to produce the desired RT60 time, and the damping filter coefficients changed accordingly. A first reverberator is then initialized with parameters derived by the encoder and received from the bitstream, and the second and third reverberators are initialized with parameters modified from those of the first reverberator.

[0179] In some embodiments, the bitstream from the encoder to the renderer may include signaling of whether head tracking is applied (e.g., employing a "headTrackingEnabled" signal). In instances where headTrackingEnabled is true (or other suitable signaling indicating that head tracking is applied), reverberation may be rendered using the methods presented herein. In instances where headTrackingEnabled is false (or any other suitable signaling indicating that head tracking is not used), reverberation may be rendered without panning simply by using a single reverberator for each channel of a multi-channel setup. This headTrackingEnabled may be signaled for the entire scene with a single value, or may be signaled separately for different parts of the scene (e.g., having individual values ​​for different acoustic environments). Furthermore, this information may also be signaled indirectly in some embodiments (e.g., if there are parameters to initialize the three reverberators of each reverberation panner, head tracking is enabled, and if they are not available, head tracking is disabled).

[0180] 15 illustrates an exemplary electronic device that can be used as any of the apparatus parts of the system as described above. The device may be any suitable electronic device or device. For example, in some embodiments, the device 2000 is a mobile terminal, a user equipment, a tablet computer, a computer, an audio playback device, etc. The device may be configured to implement, for example, an encoder or a renderer, or any of the functional blocks as described above.

[0181] In some embodiments, the device 2000 includes at least one processor or central processing unit 2007. The processor 2007 may be configured to execute various program code, such as the methods as described herein.

[0182] In some embodiments, the device 2000 comprises a memory 2011. In some embodiments, at least one processor 2007 is connected to the memory 2011. The memory 2011 may be any suitable storage means. In some embodiments, the memory 2011 comprises a program code section for storing program code implementable by the processor 2007. Furthermore, in some embodiments, the memory 2011 may further comprise a stored data section for storing data, e.g., data that has been processed or is to be processed according to the embodiments described herein. The implemented program code stored in the program code section and the data stored in the stored data section may be retrieved by the processor 2007 when needed via the memory-processor connection.

[0183] In some embodiments, the device 2000 comprises a user interface 2005. The user interface 2005 may be coupled to the processor 2007 in some embodiments. In some embodiments, the processor 2007 may control the operation of the user interface 2005 and receive input from the user interface 2005. In some embodiments, the user interface 2005 may allow a user to input commands to the device 2000, for example, via a keypad. In some embodiments, the user interface 2005 may allow a user to obtain information from the device 2000. For example, the user interface 2005 may include a display configured to display information from the device 2000 to the user. The user interface 2005 may include a touch screen or touch interface in some embodiments that may both allow information to be input into the device 2000 and also display information to the user of the device 2000. In some embodiments, the user interface 2005 may be a user interface for communication.

[0184] In some embodiments, device 2000 comprises an input / output port 2009. The input / output port 2009 in some embodiments includes a transceiver. The transceiver in such embodiments may be coupled to the processor 2007 and configured to enable communication with other apparatuses or electronic devices, for example, via a wireless communication network. The transceiver, or any suitable transceiver, or transmitting means and / or receiving means, in some embodiments, may be configured to communicate with other electronic devices or apparatuses via a wired or non-wired connection.

[0185] The transceiver may communicate with the further device by any suitable known communication protocol, for example, in some embodiments, the transceiver may use a suitable Universal Mobile Telecommunications System (UMTS) protocol, a Wireless Local Area Network (WLAN) protocol such as IEEE 802.X, a suitable short-range wireless communication protocol such as Bluetooth, or an Infrared Data Path (IRDA).

[0186] The input / output port 2009 may be configured to receive a signal.

[0187] In some embodiments, the device 2000 may be used as at least a part of a renderer. The input / output port 2009 may be connected to headphones (which may be head-tracking headphones or non-tracking headphones) or the like.

[0188] In general, various embodiments of the present invention may be implemented in hardware or special purpose circuits, software, logic, or any combination thereof. For example, some aspects may be implemented in hardware, and other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device, but the present invention is not limited thereto. Although various aspects of the present invention may be illustrated and described as block diagrams, flow charts, or using some other graphical representation, it should be appreciated that these blocks, devices, systems, techniques, or methods described herein may be implemented in, by way of non-limiting examples, hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controllers or other computing devices, or any combination thereof.

[0189] The embodiments of the present invention may be implemented by computer software executable by a data processor of a mobile terminal, such as in a processor entity, or by hardware, or by a combination of software and hardware. Furthermore, it should be noted in this respect that any block of the logic flow as shown may represent program steps, or interconnected logic circuits, blocks and functions, or a combination of program steps and logic circuits, blocks and functions. The software may be stored on physical media, such as memory chips or memory blocks implemented in the processor, magnetic media, such as hard disks or floppy disks, and optical media, such as DVDs and data variations thereof, such as CDs.

[0190] The memory may be of any type suitable for the local technology environment and may be implemented using any suitable data storage technology, such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed and removable memories, etc. The data processing device may be of any type suitable for the local technology environment and may include, by way of non-limiting examples, one or more of general purpose computers, special purpose computers, microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), gate level circuits, and processors based on multi-core processor architectures.

[0191] Embodiments of the present invention can be implemented in a variety of components, such as integrated circuit modules. The design of integrated circuits is generally a highly automated process. Complex and powerful software tools are available to convert logic level designs into semiconductor circuit designs that can be etched onto semiconductor substrates.

[0192] Programs offered by Synopsys, Inc. of Mountain View, California, and Cadence Design, Inc. of San Jose, California, use established design rules and libraries of prestored design modules to automatically route conductors and place components on a semiconductor chip. Once the design of a semiconductor circuit is complete, the resulting design in a standardized electronic format (Opus, GDSII, etc.) may be sent to a semiconductor manufacturing facility or "fab" for fabrication.

[0193] The above description has provided a full and informative description of the exemplary embodiments of the present invention, by way of illustrative and non-limiting examples. However, various modifications and adaptations will become apparent to those skilled in the art in view of the above description, when read in conjunction with the accompanying drawings and the appended claims. However, all such and similar modifications of the teachings of this invention will still fall within the scope of the present invention, as defined in the appended claims.

Claims

1. 1. An apparatus for positioning at least a portion of a sound field based on a target direction, the apparatus comprising at least one processor and at least one memory including computer program code, the at least one memory and the computer program code being configured by the at least one processor to provide the apparatus with at least: Obtaining at least one audio signal; Obtaining speaker configuration information; obtaining at least one processing path parameter for at least two processing paths, the at least one processing path parameter including a target direction associated with each of the at least two processing paths; and for each of the at least two processing paths, processing the at least one audio signal based on the at least one processing path parameter to generate a multi-channel audio signal, wherein for each processing path, means comprises: generating at least two at least partially mutually incoherent audio signals from said at least one audio signal; determining at least two panning gains based on the target directions and the speaker setup information associated with the processing paths; applying each of the at least two panning gains to an associated one of the at least partially mutually incoherent audio signals to generate an at least partially mutually incoherent audio signal with at least two panning gains applied; combining the at least two panning-gain applied at least partially mutually incoherent audio signals to generate the multi-channel audio signal. and generating the combining the multi-channel audio signals from each processing path to generate a composite panning gain applied multi-channel audio signal; An apparatus configured to cause

2. 2. The apparatus of claim 1, wherein the at least one processing path parameter further comprises at least one reverberation parameter associated with each of the at least two processing paths, and the apparatus is configured to generate at least two at least partially mutually incoherent audio signals from the at least one audio signal, and to reverberate the at least one audio signal based on the at least one reverberation parameter to generate each of the at least two at least partially mutually incoherent audio signals.

3. The device comprises: generating at least two at least partially mutually incoherent audio signals from said at least one audio signal; decorrelating the at least one audio signal to generate each of the at least two at least partially mutually incoherent audio signals. The apparatus according to claim 1 .

4. The apparatus of claim 1, wherein the apparatus is configured to determine at least two panning gains based on the target direction associated with the processing path and the speaker setting information, and to apply vector-based amplitude panning based on the target direction associated with the processing path and a direction associated with the speaker setting information.

5. The apparatus of claim 1 , wherein the apparatus is adapted to generate an immersive audio signal based on processing the composite panning gain applied multi-channel audio signal.

6. The apparatus comprises: generating the immersive audio signal based on processing the composite panning gain applied multi-channel audio signal; for each channel of the composite panning gain applied multi-channel audio signal, processing the composite panning gain applied multi-channel audio signal based on a head-related transfer function related to a direction with respect to a loudspeaker associated with said channel to generate a channel binaural panned processed audio signal; combining the binaural panned audio signals for all channels to generate the immersive audio signal; The apparatus according to claim 5 .

7. The apparatus comprises: Get speaker configuration information, receiving speaker configuration information; Determining speaker configuration information; and Obtaining predefined or default speaker configuration information; 2. The apparatus of claim 1, adapted to perform any one of the following:

8. The apparatus of claim 1 , wherein the at least two at least partially mutually incoherent audio signals are mutually incoherent audio signals.

9. The device described in claim 1, wherein the at least one acquired audio signal is at least one microphone audio signal.

10. The device of claim 1, wherein the speaker setting information represents a loudspeaker setting.

11. 1. A method for an apparatus for positioning at least a portion of a sound field based on a target direction, the method comprising: Obtaining at least one audio signal; Obtaining speaker configuration information; obtaining at least one processing path parameter for at least two processing paths, the at least one processing path parameter including a target direction associated with each of the at least two processing paths; for each of the at least two processing paths, processing the at least one audio signal based on the at least one processing path parameter to generate a multi-channel audio signal, generating at least two at least partially mutually incoherent audio signals from the at least one audio signal; determining at least two panning gains based on the target directions and the speaker setup information associated with the processing paths; applying each of the at least two panning gains to an associated one of the at least partially mutually incoherent audio signals to generate an at least partially mutually incoherent audio signal with at least two panning gains applied; combining the at least two panning-gain applied at least partially mutually incoherent audio signals to generate the multi-channel audio signal; generating combining the multi-channel audio signals from each processing path to generate a composite panning gain applied multi-channel audio signal; A method comprising:

12. 12. The method of claim 11 , wherein the at least one processing path parameter further comprises at least one reverberation parameter associated with each of the at least two processing paths, and wherein generating at least two at least partially mutually incoherent audio signals from the at least one audio signal comprises reverberating the at least one audio signal based on the at least one reverberation parameter to generate each of the at least two at least partially mutually incoherent audio signals.

13. 12. The method of claim 11 , wherein generating at least two at least partially mutually incoherent audio signals from the at least one audio signal comprises decorrelating the at least one audio signal to generate each of the at least two at least partially mutually incoherent audio signals.

14. 12. The method of claim 11 , wherein determining at least two panning gains based on the target direction associated with the processing path and the speaker setup information comprises applying vector-based amplitude panning based on a direction associated with the target direction associated with the processing path and the speaker setup information.

15. The method of claim 11 , wherein the method includes generating an immersive audio signal based on processing the composite panning gain applied multi-channel audio signal.

16. generating the immersive audio signal based on processing the composite panning gain applied multi-channel audio signal, - for each channel of the composite panning gain applied multi-channel audio signal, processing the composite panning gain applied multi-channel audio signal based on a head-related transfer function associated with a direction relative to a loudspeaker associated with said channel to generate a channel binaural panned processed audio signal; combining the binaural panned audio signals for all channels to generate the immersive audio signal; 16. The method of claim 15, comprising:

17. To get speaker configuration information, receiving speaker configuration information; Determining speaker configuration information; and Obtaining predefined or default speaker configuration information; The method of claim 11 , comprising any one of the following:

18. The method of claim 11 , wherein the at least two at least partially mutually incoherent audio signals are mutually incoherent audio signals.

19. The method of claim 11, wherein the at least one audio signal is at least one microphone audio signal.

20. The method of claim 11, wherein the speaker setting information represents a loudspeaker setting.

Images