Apparatus, method and computer program for spatial rendering of reverberation

Abstract

An apparatus for applying reverberation to at least one audio signal, the apparatus comprising means configured to: obtain at least one audio signal; obtaining a plurality of sound reproduction positions for playing back audio in relation to the listening position; determining a reverberator, the reverberator comprising a plurality of channels, where the plurality of channels are configured with at least two respective delay lines having different delays; assigning reverberator channels to sound reproduction positions based on delays of the plurality of channels; and based on the allocated channel, processing the at least one audio signal with a reverberator to generate a reverberated playback audio signal for the listening position.

Description

Technical Field

[0001] The present application relates to an apparatus and method for spatial rendering of reverberation, but not exclusively to an apparatus and method for spatial rendering of reverberation in an augmented reality and / or virtual reality device. Background Art

[0002] Reverberation is the persistence of sound in a space after the actual sound source has ceased. Different spaces are characterized by different reverberation properties. In order to convey the spatial impression of an environment, it is important to reproduce the reverberation perceptually accurately. Room acoustics are usually modeled with a separately synthesized early reflection component and a statistical model for the diffuse late reverberation. Figure 1 An example of a synthesized room impulse response is depicted, where direct sound 101 is followed by discrete early reflections 103 with directions of arrival (DOA) and diffuse late reverberation 105 that may be synthesized without any specific direction of arrival. Figure 1 The delay d1(t) 102 in FIG. 5 can be viewed as representing the direct sound arrival delay from the source to the listener, and the delay d2(t) 104 can represent the delay from the source to the listener for one of the early reflections (in this case the first to arrive).

[0003] One approach to reproducing reverberation is to use a set of N loudspeakers (or virtual loudspeakers using a set of head-related transfer functions (HRTF) binaural reproduction). These loudspeakers are positioned somewhat uniformly around the listener. Mutually incoherent reverberant signals are reproduced from these loudspeakers, resulting in the perception of ambient diffuse reverberation.

[0004] The reverberations produced by different loudspeakers must be mutually incoherent. In a simple case, reverberations can be produced using different channels of the same reverberator, where the output channels are uncorrelated but otherwise share the same acoustic properties, such as RT60 time and level (specifically, diffuse-to-direct ratio or reverberant-to-direct ratio or diffuse-to-total ratio or diffuse-to-source ratio or any other suitable parameter for representing the reverberation energy or level). Such uncorrelated outputs sharing the same acoustic properties can be obtained, for example, by tapping the output of a feedback delay network (FDN) reverberator (with appropriately tuned delay line lengths), or from a reverberator based on the use of decaying uncorrelated noise (by using a different uncorrelated noise sequence in each channel). In this case, the different reverberation signals have virtually the same characteristics, and the reverberation is generally perceived to be similar for all directions. Summary of the invention

[0005] According to a first aspect, there is provided an apparatus for applying reverberation to at least one audio signal, the apparatus comprising components configured to: obtain at least one audio signal; obtain a plurality of sound reproduction positions for playing back audio associated with a listening position; determine a reverberator, the reverberator comprising a plurality of channels, wherein the plurality of channels are configured with at least two corresponding delay lines having different delays; assign reverberator channels to the sound reproduction positions based on the delays of the plurality of channels; and process the at least one audio signal with the reverberator to generate a reverberated playback audio signal for the listening position based on the assigned channels.

[0006] The component may be further configured to: determine a directional filter pair associated with each of the sound reproduction positions for playback audio; and filter the reverberant playback audio signal using the associated directional filter pair to generate a reverberant playback binaural audio signal.

[0007] The component configured to assign channels to sound reproduction positions based on delays of the channels may be configured to: order the channels based on the delays; and assign the channels to the sound reproduction positions based on the ordering of the channels.

[0008] The component configured to assign channels to sound reproduction positions based on delays may be configured to assign channels having similar delay lengths to sound reproduction positions that are distributed far from each other.

[0009] A component configured to assign channels having similar delay lengths to sound reproduction positions that are distributed far apart from each other may be configured to: assign a first sorted channel of the sorted channels to a first sound reproduction position; and assign subsequent sorted channels to sound reproduction positions that are more than a threshold distance away from the previously assigned sound reproduction position.

[0010] The component configured to assign subsequent sorted channels to sound reproduction positions that are more than a threshold distance away from a previously assigned sound reproduction position may be configured to assign subsequent sorted channels to the currently unassigned sound reproduction position that is farthest from an immediately previous assigned sound reproduction position.

[0011] A component configured to assign channels to sound reproduction positions based on delays may be configured to: determine that the number of sound reproduction positions is less than the number of channels; and assign more than one channel to at least one of the sound reproduction positions such that each channel is assigned to one of the sound reproduction positions.

[0012] The component configured to assign channels to sound reproduction positions based on delays may be configured to: determine that the number of sound reproduction positions is greater than the number of reverberator channels; generate at least one other reverberator channel; and assign the at least one other reverberator channel to at least one sound reproduction position among the sound reproduction positions that have not been previously assigned.

[0013] A component configured to determine a reverberator comprising a plurality of channels, wherein the plurality of channels are configured with respective delay lines having different delays may be configured to: obtain at least one dimension of a scene associated with the reverberation; and determine at least one time delay of a reflection path based on the at least one dimension of the scene, wherein the delay line delay is based on the at least one time delay determined for the reflection path.

[0014] The component configured to assign channels to sound reproduction positions based on the delays may be configured to assign channels to sound reproduction positions further based on at least one dimension of the scene.

[0015] A component configured to assign channels to sound reproduction positions further based on at least one dimension of a scene may be configured to: select two initial directions from the sound reproduction position based on a longest dimension of the scene so that the two initial directions are separated by the longest dimension; assign a first channel to a first direction of the two initial directions from the sound reproduction position; and assign a second channel to a second direction of the two initial directions from the sound reproduction position, wherein a corresponding delay line delay of the second channel is similar to or equal to a corresponding delay line delay of the first channel, and the corresponding delay line delay of the second channel and the corresponding delay line delay of the first channel are the two shortest delays of the delay line.

[0016] The component configured to assign channels to sound reproduction positions based on the delay may be configured to: assign a first sorted channel to a first sound reproduction position; and assign subsequent sorted channels to unassigned sound reproduction positions that are the smallest distance from a previously assigned sound reproduction position.

[0017] The reverberator may include a plurality of reverberators, each of the plurality of reverberators including a single channel, and wherein the component configured to sort the channels based on the delay line delay may be configured to arrange the plurality of reverberators in order of increasing delay line delay.

[0018] The reverberator may include a feedback delay network reverberator.

[0019] Channels can be configured with corresponding delay lines having different delays.

[0020] The channels may be configured with corresponding delay lines having different delays that make them mutually prime.

[0021] According to a second aspect, a method for an apparatus for applying reverberation to at least one audio signal is provided, the method comprising: obtaining at least one audio signal; obtaining a plurality of sound reproduction positions for playing back audio in relation to a listening position; determining a reverberator, the reverberator comprising a plurality of channels, wherein the plurality of channels are configured with at least two corresponding delay lines having different delays; assigning reverberator channels to the sound reproduction positions based on the delays of the plurality of channels; and processing the at least one audio signal with the reverberator to generate a reverberated playback audio signal for the listening position based on the assigned channels.

[0022] The method may further include determining a directional filter pair associated with each of the sound reproduction positions for playing back the audio; and filtering the reverberant playback audio signal using the associated directional filter pair to generate a reverberant playback binaural audio signal.

[0023] Assigning the channels to the sound reproduction positions based on delays of the channels may include: ordering the channels based on the delays; and assigning the channels to the sound reproduction positions based on the ordering of the channels.

[0024] Based on the delays, assigning the channels to the sound reproduction positions may include assigning channels having similar delay lengths to the sound reproduction positions that are distributed far from each other.

[0025] Assigning channels having similar delay lengths to sound reproduction positions that are distributed far from each other may include assigning a first ordered channel of the ordered channels to a first sound reproduction position; and assigning subsequent ordered channels to sound reproduction positions that are more than a threshold distance away from the previously assigned sound reproduction position.

[0026] Assigning subsequent ordered channels to sound reproduction positions that are more than a threshold distance from a previously assigned sound reproduction position may include assigning subsequent ordered channels to currently unassigned sound reproduction positions that are farthest from an immediately previous assigned sound reproduction position.

[0027] Based on the delay, assigning the channels to the sound reproduction positions may include: determining that the number of sound reproduction positions is less than the number of channels; and assigning more than one channel to at least one of the sound reproduction positions so that each channel is assigned to one of the sound reproduction positions.

[0028] Based on the delay, assigning channels to the sound reproduction positions may include: determining that the number of sound reproduction positions is greater than the number of reverberator channels; generating at least one other reverberator channel; and assigning the at least one other reverberator channel to at least one of the sound reproduction positions that have not been previously assigned.

[0029] Determining a reverberator comprising a plurality of channels, wherein the plurality of channels are configured with corresponding delay lines having different delays may include: obtaining at least one dimension of a scene associated with the reverberation; and determining at least one time delay of a reflection path based on the at least one dimension of the scene, wherein the delay line delay is based on the at least one time delay determined for the reflection path.

[0030] Assigning the channels to the sound reproduction positions based on the delays may include assigning the channels to the sound reproduction positions further based on at least one dimension of the scene.

[0031] Further based on at least one dimension of the scene, assigning channels to the sound reproduction position may include: based on the longest dimension of the scene, selecting two initial directions from the sound reproduction position so that the two initial directions are separated by the longest dimension; assigning a first channel to a first direction of the two initial directions from the sound reproduction position; and assigning a second channel to a second direction of the two initial directions from the sound reproduction position, wherein a corresponding delay line delay of the second channel is similar to or equal to a corresponding delay line delay of the first channel, and the corresponding delay line delay of the second channel and the corresponding delay line delay of the first channel are the two shortest delays of the delay line.

[0032] Assigning channels to sound reproduction positions based on the delays may include assigning a first ordered channel to a first sound reproduction position; and assigning subsequent ordered channels to unassigned sound reproduction positions having a minimum distance from a previously assigned sound reproduction position.

[0033] The reverberator may include a plurality of reverberators, each reverberator of the plurality of reverberators including a single channel, and wherein ordering the channels based on the delay line delay may include arranging the plurality of reverberators in order of increasing delay line delay.

[0034] The reverberator may include a feedback delay network reverberator.

[0035] Channels can be configured with corresponding delay lines having different delays.

[0036] The channels may be configured with corresponding delay lines having different delays that make them mutually prime.

[0037] According to a third aspect, there is provided an apparatus for applying reverberation to at least one audio signal, the apparatus comprising at least one processor and at least one memory comprising computer program code, the at least one memory and the computer program code being configured to, with the at least one processor, cause the apparatus to at least: obtain at least one audio signal; obtain a plurality of sound reproduction positions for playing back audio associated with a listening position; determine a reverberator, the reverberator comprising a plurality of channels, wherein the plurality of channels are configured with at least two corresponding delay lines having different delays; assign reverberator channels to the sound reproduction positions based on the delays of the plurality of channels; and process the at least one audio signal with the reverberator to generate a reverberated playback audio signal for the listening position based on the assigned channels.

[0038] The apparatus may be further caused to: determine a directional filter pair associated with each of the sound reproduction positions for playing back audio; and filter the reverberant playback audio signal using the associated directional filter pair to generate a reverberant playback binaural audio signal.

[0039] The apparatus caused to assign channels to sound reproduction positions based on delays of the channels may be caused to: order the channels based on the delays; and assign the channels to the sound reproduction positions based on the ordering of the channels.

[0040] The apparatus, being caused to assign channels to sound reproduction positions based on delays, may be caused to assign channels having similar delay lengths to sound reproduction positions distributed far from each other.

[0041] The device, which is caused to assign channels having similar delay lengths to sound reproduction positions that are distributed far away from each other, may be caused to: assign a first sorted channel of the sorted channels to a first sound reproduction position; and assign subsequent sorted channels to sound reproduction positions that are more than a threshold distance away from the previously assigned sound reproduction position.

[0042] The device that is caused to assign subsequent sorted channels to sound reproduction positions that are more than a threshold distance away from a previously assigned sound reproduction position may be caused to assign subsequent sorted channels to the currently unassigned sound reproduction position that is farthest from the immediately previous assigned sound reproduction position.

[0043] The device that is caused to assign channels to sound reproduction positions based on delays can be caused to: determine that the number of sound reproduction positions is less than the number of channels; and assign more than one channel to at least one of the sound reproduction positions so that each channel is assigned to one of the sound reproduction positions.

[0044] The apparatus caused to assign channels to sound reproduction positions based on delays may be caused to: determine that the number of sound reproduction positions is greater than the number of reverberator channels; generate at least one other reverberator channel; and assign the at least one other reverberator channel to at least one of the sound reproduction positions that have not been previously assigned.

[0045] The apparatus is caused to determine a reverberator, the reverberator comprising a plurality of channels, wherein the plurality of channels are configured with respective delay lines having different delays, the apparatus being caused to: obtain at least one dimension of a scene associated with the reverberation; and determine at least one time delay of a reflection path based on the at least one dimension of the scene, wherein the delay line delay is based on the at least one time delay determined for the reflection path.

[0046] The apparatus being caused to assign channels to sound reproduction positions based on the delays may be caused to assign channels to sound reproduction positions further based on at least one dimension of the scene.

[0047] The device, which is caused to assign channels to sound reproduction positions further based on at least one dimension of the scene, can be caused to: select two initial directions from the sound reproduction position based on the longest dimension of the scene so that the two initial directions are separated by the longest dimension; assign a first channel to a first direction of the two initial directions from the sound reproduction position; and assign a second channel to a second direction of the two initial directions from the sound reproduction position, wherein a corresponding delay line delay of the second channel is similar to or equal to a corresponding delay line delay of the first channel, and the corresponding delay line delay of the second channel and the corresponding delay line delay of the first channel are the two delay lines with the shortest delays.

[0048] The apparatus caused to assign channels to sound reproduction positions based on delays may be caused to: assign a first sorted channel to a first sound reproduction position; and assign subsequent sorted channels to unassigned sound reproduction positions having a minimum distance from a previously assigned sound reproduction position.

[0049] The reverberator may comprise a plurality of reverberators, each reverberator of the plurality of reverberators comprising a single channel, and wherein the means caused to order the channels based on the delay line delay may be caused to arrange the plurality of reverberators in order of increasing delay line delay.

[0050] The reverberator may include a feedback delay network reverberator.

[0051] Channels can be configured with corresponding delay lines having different delays.

[0052] The channels may be configured with corresponding delay lines having different delays that make them mutually prime.

[0053] According to a fourth aspect, there is provided an apparatus for applying reverberation to at least one audio signal, the apparatus comprising: an obtaining circuit configured to obtain at least one audio signal; an obtaining circuit configured to obtain a plurality of sound reproduction positions for playing back audio in relation to a listening position; a determining circuit configured to determine a reverberator, the reverberator comprising a plurality of channels, wherein the plurality of channels are configured with at least two corresponding delay lines having different delays; an allocating circuit configured to allocate reverberator channels to the sound reproduction positions based on the delays of the plurality of channels; and a processing circuit configured to process the at least one audio signal with the reverberator based on the allocated channels to generate a reverberated playback audio signal for the listening position.

[0054] According to a fifth aspect, there is provided a computer program [or a computer readable medium comprising program instructions] comprising instructions for causing an apparatus for applying reverberation to at least one audio signal to at least perform the following operations: obtain at least one audio signal; obtain a plurality of sound reproduction positions for playing back audio associated with a listening position; determine a reverberator, the reverberator comprising a plurality of channels, wherein the plurality of channels are configured with at least two corresponding delay lines having different delays; assign reverberator channels to the sound reproduction positions based on the delays of the plurality of channels; and process the at least one audio signal with the reverberator based on the assigned channels to generate a reverberated playback audio signal for the listening position.

[0055] According to a sixth aspect, there is provided a non-transitory computer-readable medium comprising program instructions for causing an apparatus for applying reverberation to at least one audio signal to at least perform the following operations: obtain at least one audio signal; obtain a plurality of sound reproduction positions for playing back audio associated with a listening position; determine a reverberator, the reverberator comprising a plurality of channels, wherein the plurality of channels are configured with at least two corresponding delay lines having different delays; assign reverberator channels to the sound reproduction positions based on the delays of the plurality of channels; and process the at least one audio signal with the reverberator based on the assigned channels to generate a reverberated playback audio signal for the listening position.

[0056] According to a seventh aspect, there is provided an apparatus for applying reverberation to at least one audio signal, comprising: means for obtaining at least one audio signal; means for obtaining a plurality of sound reproduction positions for playing back audio associated with a listening position; means for determining a reverberator, wherein the reverberator comprises a plurality of channels, wherein the plurality of channels are configured with at least two corresponding delay lines having different delays; means for assigning reverberator channels to the sound reproduction positions based on the delays of the plurality of channels; and means for processing the at least one audio signal with the reverberator to generate a reverberated playback audio signal for the listening position based on the assigned channels.

[0057] According to an eighth aspect, there is provided a computer-readable medium comprising program instructions for causing an apparatus for applying reverberation to at least one audio signal to at least perform the following operations: obtain at least one audio signal; obtain a plurality of sound reproduction positions for playing back audio associated with a listening position; determine a reverberator, the reverberator comprising a plurality of channels, wherein the plurality of channels are configured with at least two corresponding delay lines having different delays; assign reverberator channels to the sound reproduction positions based on the delays of the plurality of channels; and process the at least one audio signal with the reverberator to generate a reverberated playback audio signal for the listening position based on the assigned channels.

[0058] An apparatus comprises means for performing the actions of the method as described above.

[0059] A device is configured to perform the actions of the method described above.

[0060] A computer program includes program instructions for causing a computer to execute the method as described above.

[0061] A computer program product stored on a medium may cause an apparatus to perform the method described herein.

[0062] An electronic device may include an apparatus as described herein.

[0063] A chipset may include the apparatus as described herein.

[0064] The embodiments of the present application are intended to solve the problems associated with the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] For a better understanding of the present application, reference will now be made by way of example to the accompanying drawings, in which:

[0066] Figure 1 Shows the model of room acoustics and room impulse response;

[0067] Figure 2schematically illustrates example apparatus in which some embodiments may be implemented;

[0068] Figure 3 Shown as Figure 2 A flowchart of the operation of the example apparatus shown in;

[0069] Figure 4 Further detailed schematically illustrates how according to some embodiments Figure 2 The example reverb shown in ;

[0070] Figure 5 Shown as Figure 4 A flowchart of the operation of the example reverberator shown in;

[0071] Figure 6 Further detailed schematically illustrates how according to some embodiments Figure 2 The example reverb signal router shown in ;

[0072] Figure 7 Shown as Figure 6 A flowchart of the operation of an example reverberation signal router as shown in;

[0073] Figures 8 to 11 shows an example reverb to speaker channel routing sequence;

[0074] Fig.12 Further detailed schematically illustrates how according to some embodiments Figure 2 The example binaural renderer shown in ;

[0075] Fig.13 Shown as Fig.12 A flowchart of the operation of an example binaural renderer shown in ;

[0076] Fig.14 schematically illustrates an example feedback delay network (FDN) reverberator according to some embodiments;

[0077] Fig.15 According to some embodiments, Figure 2 Implementation of the apparatus shown in within an example application; and

[0078] Fig.16 An example apparatus suitable for implementing the means shown in the preceding Figures is shown. DETAILED DESCRIPTION

[0079] Suitable apparatus and possible mechanisms for parameterizing and rendering audio scenes with diffuse reverberation are described in further detail below.

[0080] As discussed above, reproducing reverberation from N incoherent speakers (virtual or real) around a listener will typically reproduce the perception of diffuse reverberation. However, depending on the reverberator, the perception of fully surrounding and enveloping reverberation may not always be produced.

[0081] For example, when using a feedback delay network (FDN) reverberator, the spatial image may "collapse" to some directions or a single direction. This may be perceived as if the reverberation mainly originates from a certain direction, such as left, right, front, or back, rather than surrounding and enveloping.

[0082] The FDN reverberator comprises delay lines with different lengths for different output channels. In fact, at the beginning of the reverberation, before the pulse density is high enough, the outputs of the different channels are relatively coherent. As a result, the timing of the first pulse affects the perceived direction of the reverberation. For example, in case the reverberation signal obtained by the virtual loudspeaker on the left side of the listener has a relatively earlier pulse than the other virtual loudspeakers, the sound image is perceived as collapsing to the left side of the listener. Since the late reverberation is intended to be perceived as surrounding and enveloping, this is obviously not desired and makes the perception of the reverberation unnatural.

[0083] Although it is possible to create a reverberation channel using a decaying uncorrelated noise sequence as a finite impulse response (FIR) filter, this may result in an implementation with very high computational complexity because the FIR filter will be very long. For example, in the case of a reverberation greater than 2 seconds, the FIR filter may have more than 96,000 taps per channel.

[0084] Thus, the concepts further described with respect to the following embodiments relate to the reproduction of diffuse reverberation, wherein apparatus and methods are presented that enable computationally efficient rendering of surround and envelop diffuse reverberation. In some embodiments, this is achieved by determining N surround spatial locations, determining an N-channel FDN reverberator, ordering the reverberator channels based on delay line length, assigning the ordered reverberator channels to the spatial locations in a manner that reverberator channels having similar delay line lengths are distributed separately (away) from each other, reverberating an input audio signal with the reverberator (having N channels), and spatializing the reverberant audio signal (e.g., by filtering the reverberant audio signal with an associated directional filter pair (such as HRTF) for binaural reproduction (headphone reproduction) or by panning with Vector Basis Amplitude Panning (VBAP) for loudspeaker reproduction (or reproducing each reverberant channel only from the loudspeaker closest to the desired spatial location)).

[0085] Thus, in such embodiments, computational efficiency is improved by utilizing a computationally efficient reverberator (such as an FDN reverberator having only a relatively small number of filter taps) rather than using, for example, an attenuated noise burst having a large number of filter taps (and therefore being computationally intensive).

[0086] Although the following examples describe the use of an FDN reverberator, some embodiments may include another reverberator with different output channels having different delays. For example, various digital reverberators other than an FDN (where there are delay lines with different delays from which output channels are derived) may benefit from implementing embodiments as described herein without requiring extensive creative input.

[0087] Embodiments as described herein can achieve the perception of surround and enveloping reverberation through appropriate spatial distribution of reverberator channels based on delay line lengths. Since reverberator channels with similar delay line lengths are reproduced from spatial locations far away from each other, the reverberation is not perceived as originating from any particular direction. In addition, if the reverberator channel with the shortest delay line length is positioned to the most sideways spatial location, the width of the reverberation is maximized. As a result, human listeners perceive the reverberation reproduced using the present invention as surround and enveloping. In some embodiments, for example when head tracking is used, the reverberator channels can be evenly positioned in the left, right, front and rear directions to ensure that the width of the reverberation is similar for all directions regardless of head rotation.

[0088] In some embodiments, the apparatus and method use 15 virtual loudspeakers that can be (pseudo) evenly distributed around the listener according to a suitable spatial distribution. Furthermore, in such embodiments, a 15-channel FDN reverberator is used to generate a reverberation signal for the virtual loudspeakers. In these embodiments, the channels of the FDN reverberator can be sorted based on delay line length (from shortest to longest).

[0089] In some embodiments, the FDN channel with the shortest delay line length is assigned to the leftmost or rightmost virtual speaker (it does not matter which one is chosen). In turn, the FDN channel with the second shortest delay line length is assigned to the virtual speaker farthest from the previously assigned virtual speaker. This method can then be repeated for all remaining FDN channels: the channel with the next shortest delay line length is assigned to the virtual speaker farthest from the last assigned FDN channel (to which no FDN channel has yet been assigned).

[0090] Furthermore, according to these embodiments, an input audio signal (eg, a mono audio signal) may be reverberated using an FDN reverberator, and the resulting reverberated audio signal may be routed to the corresponding virtual speakers determined as above.

[0091] Finally, in some embodiments, the virtual speaker signals are filtered with a head-related transfer function (HRTF) corresponding to the direction of the virtual speakers. The resulting binaural audio signals are added, and the added binaural audio signal is provided as the output of the reverberator. In some other embodiments, the virtual speaker signals are reproduced from a physical speaker setup instead of HRTF filtering for headphones.

[0092] Therefore, in practice, FDN channels can be allocated in such a way that FDN channels with similar delay line lengths are rendered with approximately equal virtual speakers on the left and right sides. As a result, the sound image does not collapse to any direction. Instead, the sound image is perceived as surrounding and enveloping. Furthermore, since the FDN reverberator is computationally efficient, the apparatus and method as shown in the following embodiments are also computationally efficient.

[0093] about Figure 2 , an example apparatus 299 embodiment is shown. The input to the system is an audio signal 200 to be reverberated, reverberation parameters 202, and speaker setup parameters 208. Furthermore, the apparatus 299 is configured to generate a reverberated binaural audio signal 212 as output, which is perceived as surround and enveloping.

[0094] The audio signal s(n) (where n is time) is forwarded to the reverberator 201. In some embodiments, the reverberator 201 is implemented as a feedback delay network (FDN) reverberator with N channels. The FDN reverberator with N channels is configured to generate N (almost) mutually incoherent reverberation signals as output.

[0095] Furthermore, in some embodiments, the reverberator is configured to receive reverberation parameters 202. The reverberation parameters 202 include parameters that control the generation of reverberation (e.g., a desired reverberation time RT60(f), a reverberation to direct ratio RDR(f) (or other equivalent representations, such as diffuse to total emitted energy ratio or diffuse to source energy ratio), and / or dimensions of the virtual environment and / or one or more materials). Thus, for example, the reverberation parameters 202 are used to configure the FDN reverberator 201. In some embodiments, the input 202 of the reverberator 201 may be FDN reverberator parameters, such as attenuation filter parameters, parameters for an RDR control filter, and the like.

[0096] In some embodiments, the reverberator 201 is configured to generate a reverberant audio signal s rev (n, i) 204 (where i is the channel index) as output. These are mutually incoherent reverberation signals. In addition, the reverberator is configured to determine and output a delay line length ξ(n) 206, which corresponds to the delay m1 to m2 in the FDN reverberator. D .

[0097] Reverberant audio signals rev (n, i) 204 and delay line length ξ(i) 206 are forwarded to the reverberation signal router 207, which also receives the loudspeaker setting parameters θ ls (i) φ ls (i) 208 as input.

[0098] The speaker setup parameters include information that can be used to create the perception of an enveloping diffuse reverberation. Such information has been discussed above. Illustrative examples of speaker setup parameter values ​​are:

[0099] Azimuth θ ls (i): 0, 45, 90, 135, 180, -135, -90, -45, 45, 135, -135, -45, 45, 135, -135, -45 degrees.

[0100] Elevation angle φ ls (i): 0, 0, 0, 0, 0, 0, 0, 0, 30, 30, 30, 30, -30, -30, -30, -30 degrees.

[0101] Therefore, there are N channel positions defined by the speaker setup parameters. In the above example, there are 16 channels, but in other embodiments, N may be less or greater than 16.

[0102] In some embodiments, the apparatus 299 further comprises a reverberation signal router 207 configured to route the reverberation audio signal s based on the delay line length ξ(i) 206. rev (n, i) 204 so that the reverberation is perceived as surrounding and enveloping when reproduced using a (real or virtual) speaker setup corresponding to the speaker setup parameters 208. Thus, in some embodiments, the reverberation signal 207 is configured to determine and output a rearranged reverberant audio signal s rev-rearr (n, i)210.

[0103] In some embodiments, the apparatus 299 may include a binaural renderer 209 (or more generally an output renderer) configured to receive the rearranged reverberant audio signal s rev-rearr (n, i) 210 and speaker setting parameters 209. The binaural renderer 209 is further configured to render the reverberant audio signal into a reverberant binaural signal s bin (n, j) 212 (where j is a binaural channel), and the binaural signals can be reproduced using headphones, for example. These signals are perceived as surround and enveloping. As indicated above, the binaural renderer 209 can be replaced with a suitable renderer (or spatializer) configured to generate a suitable number of channel outputs.

[0104] about Figure 3 , showing Figure 2 Flowchart of the operation of the example device 299 shown in .

[0105] In some embodiments, the method comprises the operations of obtaining an audio signal, speaker setup parameters defining (virtual) speaker positions, and reverberation parameters for controlling a reverberation process, such as Figure 3 As shown in step 301.

[0106] After the audio signal and the reverberation parameters have been obtained, the method may then be configured to determine or generate a reverberant audio signal based on the input audio signal and the reverberation parameters. Furthermore, the method comprises determining an associated reverberation delay line length. Figure 3 The operation of generating the reverberant audio signal and the delay line length is shown by step 303 .

[0107] Furthermore, the method may be configured to rearrange the reverberant audio signal based on the speaker setup parameters and the delay line length, such as Figure 3 As shown in step 305.

[0108] After the reverberant audio signal has been rearranged, the method may further comprise rendering a suitable reverberant binaural signal based on the speaker setup parameters, such as Figure 3 As shown in step 307.

[0109] Furthermore, the method may include outputting a reverberant binaural signal, such as Figure 3 As shown in step 309.

[0110] Figure 4 In more detail, the schematic diagram shows Figure 2 As described above, in some embodiments, the reverberator 201 includes a feedback delay network (FDN) reverberator 403 and a reverberator configurator / controller 401 .

[0111] In some embodiments, the reverberator configurator / controller 401 is configured to obtain the reverberation parameters 202 and generate appropriate configuration parameters to control the FDN reverberator 403. Additionally, in some embodiments, the reverberator configurator / controller 401 is configured to determine or otherwise obtain the delay line length 206. Additionally, the delay line length 206 may be output from the reverberator 201.

[0112] After being configured by control parameters from the reverberator configurator / controller 401 , the FDN reverberator 403 may generate a reverberant audio signal 204 having desired reverberant characteristics based on applying reverberation to the audio signal 200 .

[0113] Fig.14An example FDN reverberator that can be used to generate D uncorrelated output audio signals is shown in more detail. In this example, each output signal can be rendered at a certain spatial position around the listener for surround reverberation perception.

[0114] In some embodiments, the FDN reverberator 403 includes an input gain amplifier 1452 configured to receive the input 1451 and apply a gain g, and output the input to which the gain is applied to the DDR energy ratio control filter (GEQ DDR ) 1453. The energy ratio control filter may also be referred to as a reverberation to direct ratio (RDR) control filter. The goal of this filter is to produce a desired level of reverberation (energy) for a given unit of energy input.

[0115] The example FDN reverberator 403 further includes a DDR energy ratio control filter (GEQ DDR )1453, which applies the filter coefficient GEQ controlled by the DDR energy ratio ddr Furthermore, the reverberator is configured such that the reverberation parameters are processed to generate the coefficients GEQd (GEQ1, GEQ2, ... GEQ D ), the coefficient A of the feedback matrix 1457, the length md (m1, m2, ...m D ). Delay line 1459 is sometimes referred to as a feedback loop. The length is sometimes referred to as the loop filter delay. Thus, the example FDN reverberator 403 shows a D channel output by providing the output from each FDN delay line as a separate output.

[0116] In some embodiments, each attenuation filter GEQ d 1461 is implemented as a graphic EQ filter using M biquad IIR band filters. Therefore, in the case of octave band M=10, the parameters of each graphic EQ include feedforward and feedback coefficients for the biquad IIR filter, gain for the biquad band filter, and total gain.

[0117] The reverberator uses a network of delays 1459, feedback elements (shown as attenuation filters 1461, feedback matrices 1457 and combiners 1455, and output gains 1463).

[0118] The FDN reverberator includes multiple recirculating delay lines. Unitary matrix A 1457 is used to control the recirculation in the network. Attenuation filters 1461 (which in some embodiments may be implemented as a graphic EQ filter implemented as a cascade of second-order-section IIR filters) can facilitate controlling the rate of energy decay at different frequencies. The filters 1461 are designed so that they attenuate the desired amount (in decibels) as each pulse passes through the delay line and so that the desired RT60 time is obtained.

[0119] For the FDN Reverberator 403, these parameters contain the decay filter GEQ for each d The coefficients of 1461, the feedback matrix coefficients A 1457, and the lengths m of the D delay lines 1459 d In some embodiments, each attenuation filter GEQ d 1461 is a graphic EQ filter using M biquad IIR band filters.

[0120] Thus, in the case of octave bands M=10, the parameters of each graphic EQ include the feedforward b and feedback a coefficients of 10 biquad IIR filters, the gain for the biquad band filter, and the total gain.

[0121] The number of delay lines D can be adjusted according to the quality requirements and the desired trade-off between reverberation quality and computational complexity. In an embodiment, an efficient implementation with D=15 delay lines is used. This makes it possible to define the feedback matrix coefficients A as proposed by Rocchesso in "Maximally Diffusive Yet Efficient Feedback Delay Networks for Artificial Reverberation" (IEEE Signal Processing Letters, Vol. 4, No. 9, September 1997) in terms of Galois sequences that facilitate efficient implementation.

[0122] The length of the delay line d is m dIt can be determined based on the size of the virtual room. Here, we use the size of the enclosed space. For example, a shoebox-shaped room can be defined with dimensions xDim, yDim, and zDim. If the room is not shoebox-shaped (rectangular), the shoebox can be placed in the room, and the size of the placed shoebox can be used as the delay line length. Alternatively, these dimensions can be obtained as the three longest dimensions in the non-shoebox-shaped room, or obtained using other suitable methods. If the bounding box is provided as a grid, such dimensions can also be obtained from the grid. These dimensions can be further converted to modified dimensions of a virtual room or enclosed space having the same volume as the input room or enclosed space. For example, ratios 1, 1.3, and 1.9 can be used for the converted virtual room dimensions. When this method is executed in the renderer, the enclosed space vertices will be obtained from the bitstream, and the size can be calculated along each axis x, y, and z by the difference between the maximum and minimum values ​​of the vertices.

[0123] In some embodiments, in some embodiments, the delay can be set proportionally to the resonant frequency of a standing wave in a virtual room or a physical room.Thus, the delay can be set proportionally to the resonant frequency of a standing wave in a virtual room or a physical room (acoustic environment).

[0124] Room modes can be defined as:

[0125] roomModes{

[0126] {1,0,0},

[0127] {0,2,1},

[0128] {1,0,1},

[0129] {2,1,0},

[0130] {0,1,1},

[0131] {1,1,1},

[0132] {1,1,0},

[0133] {0,1,2},

[0134] {1,2,1},

[0135] {1,2,0},

[0136] {0,0,1},

[0137] {2,1,1},

[0138] {0,1,0},

[0139] {1,0,2},

[0140] {2,0,1},

[0141] };

[0142] The calculation of the i-th delay can be as follows:

[0143] xMode=roomModes[i][0] / xDim;

[0144] yMode=roomModes[i][1] / yDim;

[0145] zMode=roomModes[i][2] / zDim;

[0146] xMode*=xMode;

[0147] yMode*=yMode;

[0148] zMode*=zMode;

[0149] resonanceFreq=

[0150] 0.5*343*sqrt(xMode+yMode+zMode);

[0151] delays[i]=sampleRateHz / resonanceFreq;

[0152] Latencies can be sorted in ascending order.

[0153] Delay line length m d Can be further set to integers that are relatively prime. This selection maximizes the number of samples in the FDN output impulse response before it repeats. The sieve method of Sundaram's algorithm can be used to find the prime number that reaches the maximum delay line length. Then, each delay line length can be mapped to the closest prime number in the obtained prime number set.

[0154] In addition, the attenuation filter coefficients in the delay line can be adjusted so that the desired amount of attenuation (in decibels) occurs each time the signal recycles through the delay line, thereby achieving the desired RT60(k) time. This is done in a frequency-specific manner to ensure the appropriate rate of attenuation of the signal energy at a specified frequency k.

[0155] For frequency k, the expected attenuation per signal sample is calculated as attenuationPerSample(k) = -60 / (samplingRate*RT60(k)). d The attenuation (in decibels) of the delay line is attenuationDb(k)=md *attenuationPerSample(k).

[0156] Furthermore, the attenuation filters are designed as cascaded graphic equalizer filters for each delay line, as in V. and J. Liski, “Accurate cascade graphic equalizer” (IEEE Signal Processing Letters, Vol. 24, No. 2, pp. 176-180, February 2017). The design process outlined takes as input a set of command gains in octave bands. There are also methods for similar graphic EQ structures that can support three octave bands, thereby increasing the number of biquad filters to 31 and providing a better match for the detailed target response, such as in “Third-Octave and Bark Graphic-Equalizer Design with Symmetric Band Filters” ( https: / / www.mdpi.com / 2076-3417 / 10 / 4 / 1222 / pdf ) indicated in .

[0157] The reverberation ratio parameter may refer to the diffuse-to-total energy ratio (DDR) or the reverberation-to-direct ratio (RDR) or other equivalent representations. These ratio parameters may be equivalently represented on a linear scale or a logarithmic scale.

[0158] In this step, the filter is designed so that when the filter is applied to the input data of the FDN reverberator, the output reverberation is configured to have a desired energy ratio defined by DDR(k). In some embodiments, the input to the design process may be a DDR value DDR(k). When input values ​​are provided at arbitrary frequencies k, these values ​​may be converted to a desired frequency band center frequency b, such as an octave band center frequency, via interpolation.

[0159] When receiving linear DDR values ​​DDR(b), these values ​​can be converted to linear RDR values, such as:

[0160] RDR(b)=DDR(b)*10 (41 / 10)

[0161] When receiving logarithmic RDR values ​​logRDR(b), these values ​​can be converted to linear RDR values ​​as follows:

[0162] RDR(b)=10 (logRDR(b) / 10)

[0163] GEQDDR The reverberator spectral energy is matched to a target spectral energy. To this end, an estimate of the RDR of the reverberator output and a target RDR are obtained. The RDR of the reverberator output may be obtained by rendering a unit pulse through the reverberator using a first reverberator parameter, measuring an energy of the reverberator output and an energy of the unit pulse, and calculating a ratio of these energies.

[0164] In some embodiments, a unit pulse input is generated, where the first sample value is 1 and the length of the zero tail is long enough. In fact, we have adjusted the length of the zero tail to be equal to max(RT60(b)) plus t in the sample. predelay , where t predelay is the pre-delay time in seconds. The mono output of the reverberator is of interest, so the filter is configured to sum the delay line j to obtain the reverberator output s as a function of time t rev (t).

[0165] In s rev The long FFT (length is NFFT) is calculated on (t), and its absolute value is obtained as:

[0166] FFA(kk)=abs(FFT(s rev (t))

[0167] Where kk is the FFT bin index. The positive half-spectrum energy density is obtained as:

[0168] S(kk)=1 / NFFT*FFA(kk) 2

[0169] Here, the energy from the negative frequency index kk is added to the corresponding positive frequency index kk.

[0170] The energy per unit pulse can be calculated or obtained analytically and may be denoted as Su(kk).

[0171] Calculate the band energy of the reverberator's positive half-spectrum energy density S(kk) and the unit pulse's positive half-spectrum energy density Su(kk). The band energy can be calculated as:

[0172]

[0173] Among them, b low and b high are the lowest and highest bin indices belonging to frequency band b, respectively. The frequency band bin index can be obtained by comparing the frequency of the bin with the lower and upper bound frequencies of each frequency band.

[0174] Reproduction RDR of the reverberator output at frequency band b rev(b) can be obtained as: RDR rev (b) = S(b) / Su(b)

[0175] GEQ DDR The target linear amplitude response can be obtained as: ddrFilterTargetResponse(b)=sqrt(RDR(b)) / sqrt(RDR rev (b))

[0176] Wherein, RDR(b) is the linear target RDR value mapped to frequency band b.

[0177] GontrolGain(b)=20*log10(ddrFilterTargetResponse(b)) is the input, used as The target response of the graphic equalizer design routine in “Neurally Controlled Graphic Equalizer” (IEEE / ACM Transactions on Audio, Speech, and Language Processing, Vol. 27, No. 12, December 2019) or the target response of another suitable routine (controlling gain).

[0178] The DDR filter target response (control gain used in the graphic EQ design routine) can also be obtained directly in the logarithmic domain as:

[0179] ControlGain(b)=10log 10 (RDR(k))-10log 10 (RDR rev (b))

[0180] The audio processing in each audio block of length B audio input samples in the reverberator may involve the following steps, as described in m59696 "First version of Text of Working Draft of RM0":

[0181] Write the B input samples to the pre-delay line (if one exists). Fig.14 The pre-delay line is not depicted in FIG, but in order to achieve the desired pre-delay of the FDN reverberator, an additional pre-delay line may be implemented. In an embodiment, the length of the additional pre-delay line may be equal to the desired pre-delay (in samples) minus the length of the shortest FDN delay line (in samples).

[0182] Read B pre-delayed input samples from the pre-delay line.

[0183] The pre-delayed input samples may be further filtered with an RDR controlled filter to obtain ratio controlled input samples to the FDN reverberator.

[0184] In some embodiments, the RDR controls the filter GEQ RDR The delay line decay filter can be a cascade of second order partial filters to implement a GEQ filter bank. Such a second order partial filter can be implemented using the following process. The input sample is sample, and the output is filteredSample.

[0185]

[0186] where numBands is the number of band filters in the SOS cascade.

[0187] levelAndFactor can be calculated as levelAndFactor = G0 * db2mag (-levelDb). levelDb is the offset (in decibels) that is subtracted from the target response of the GEQ filter bank before design. This makes GEQ design easier in some embodiments because the GEQ does not need to model the average gain of the response, but rather treats it individually. G0 is the global gain for the GEQ filter bank.

[0188] The coefficients a1[i], a2[i], b1[i], b2[i] are the feedback and feedforward coefficients of the i-th IIR filter in the SOS cascade. The past output values ​​wPast0 and wPast1 can be initialized to zero vectors of length numBands and stored between successive calls to the filterSOSCascade and filterSOS methods so that the filter state is preserved.

[0189] The procedure sosFilt can be used to implement an SOS filter in direct form 2 as follows:

[0190]

[0191] The FDN reverberator can process the rate-controlled input samples at m_inputBuffer to produce NUM_OUTPUTS reverberation output signals s rev (n,i), where i is the output channel index. The following code shows an example implementation:

[0192]

[0193] In the code example above, copy_n(sourcePointer, count, targetPointer) copies count samples starting from the memory address pointed to by sourcePointer to the memory address pointed to by targetPointer.

[0194] array(value, count) initializes an array of length count using the value value.

[0195] The &variable operator returns the memory address of variable.

[0196] The *pointer operator returns the value of the variable located at the address specified by pointer.

[0197] The operator variable++ increases the value of variable by one.

[0198] Therefore, the output signal is at m_outputBuffer[j][n]. These correspond to the reverberation output signal s rev (n,i) (index in pseudocode, corresponding to index i in the equation).

[0199] The resulting reverberant audio signal s rev (n,i) (where i is the output audio channel index) is the output of the reverberator 201. In addition, the reverberator also outputs the delay line lengths m1 to m2 used to configure the FDN reverberator. D .

[0200] about Figure 5 , showing that Figure 4 Flowchart of the operation of the example reverberator shown in . For example, in some embodiments, the method includes obtaining an audio signal and a reverberation parameter, such as Figure 5 As shown in step 501.

[0201] The method may further include generating reverberator control / configuration parameters such as Figure 5 As shown in step 503.

[0202] Additionally, the method may include determining a delay line length, such as Figure 5 As shown in step 504.

[0203] Furthermore, in some embodiments, the method includes configuring the FDN reverberator using the determined control / configuration parameters, such as Figure 5 As shown in step 505.

[0204] Then, after the FDN reverberator has been configured, the FDN reverberator is applied to the audio signal to generate a reverberated audio signal, such as Figure 5 As shown in step 507.

[0205] Finally, the reverberation audio signal and delay line length can be output, such as Figure 5 As shown in step 509.

[0206] Figure 6 A schematic diagram of a reverberation signal router 207 according to some embodiments is shown in more detail. The delay line lengths ξ(i) 206 are forwarded to a delay line length sorter 601, which is configured to arrange the delay line lengths from shortest to longest. As a result, the index of the reverberator channel is output as sorted reverberation channel index ξ(i) 602 according to the sorted delay line lengths.

[0207] In some embodiments, the reverberation signal router 207 includes a reverb to speaker setup distributor 603. The reverb to speaker setup distributor 603 is configured to receive the sorted reverberation channel index ξ(i) 602 and the speaker setup parameter θ ls (i) φ ls (i) 208, which is configured to determine which (real or virtual) speaker should reproduce which reverberation channel. In some embodiments, the reverb to speaker setup assignor is configured to select the initial speaker position from the left or right side. In this example, the reverb to speaker setup assignor 603 is configured to select the speaker closest to the left, located at 90 degrees in azimuth and 0 degrees in elevation. This selection is Figure 8 , where an initial selection 801 is shown. In turn, a reverb to speaker setup assignor 603 may create or otherwise begin generating a reverb to speaker setup mapping μ(i) 604 comprising a mapping from the sorted reverb channel index ζ(i) to the speaker setup parameters θ ls (i) φ ls For example, the mapping μ(i) may include for each loudspeaker θ ls (i) φ ls (i) The assigned sorted reverberation channel index. In this example, μ(3) = 1. Figure 8 In FIG. 8 , black circles (eg, 800) correspond to speaker locations, and black asterisks 801 with azimuth and elevation next to them indicate the assignment of reverberation channels to the speakers. Vector 851 indicates the forward direction (azimuth is zero degrees, elevation is zero degrees).

[0208] The reverb to speaker setup allocator 603 may further be configured to calculate the distance from the initial speaker position to the remaining speaker positions. The distance may be calculated in a Cartesian coordinate domain. The azimuth and elevation of the speaker setup (with a set unit radius value) may be converted to Cartesian coordinates, and the Euclidean distance between the initial speaker position and the remaining speaker positions may be calculated. Further, the reverb to speaker setup allocator 603 may be configured to select the next speaker as the speaker having the longest (or farthest) distance from the initial speaker position. The next selection is a speaker position at distance 2, at an azimuth of -90 and an elevation of 0, such as Fig. 9 901. In some embodiments, the distances between the speaker positions may be calculated at the beginning of the process and stored in a data structure (such as a distance matrix with the speaker positions at rows and columns and the entries corresponding to the distances), and during iterations, the distance values ​​may be obtained from the distance matrix (rather than calculating them).

[0209] The allocation is μ(7)=2.

[0210] This process is then continued until all reverberation channels have been assigned to speaker positions. At each iteration, the distance from the just selected speaker position to the remaining speaker positions is calculated, and the new position is selected as the one that maximizes this distance. For example, Fig.10 The third step of the assignment is depicted, namely selecting the next loudspeaker position t001 and extending the mapping, μ(2)=4.

[0211] After the process is complete, all 16 reverb channels have been assigned, and the result is as follows: Fig.11 as shown in .

[0212] The resulting allocation is μ(i), i=1,...,16: 7, 3, 1, 10, 8, 4, 2, 9, 5, 14, 16, 11, 15, 12, 6, 13

[0214] In this assignment, the value at μ(i)=j means that the ordered reverberation output at index j is routed to the loudspeaker at index i. That is, the reverberation output corresponding to the shortest delay line (at index 1) is routed to the loudspeaker at index 3 (90, 0). The next shortest delay line (at index 2) is routed to the loudspeaker at index 7 (-90, 0), and so on.

[0215] In some embodiments where there are more reverberator outputs than speaker channels, the method may start again from the beginning after all speaker channels have been assigned a reverberator output. This means that some speaker channels may have more than one reverberator channel as input, and reverberator channels may be grouped into speaker channels.

[0216] In some embodiments where the reverberator outputs fewer than speaker channels, the method stops after all speaker channels have been assigned. This means that not all speaker channels may have been assigned a reverberator output channel and therefore no reverberation signal will be rendered.

[0217] Alternatively, in some embodiments, additional decorrelation may be applied to some of the reverberator output channels to create additional reverberation channels for the remaining (some) speakers, if this is beneficial in some situations (such as not having a sufficient number of speakers to reproduce the reverberation signal).

[0218] The resulting reverberation-to-speaker-setting mapping μ(i) is related to the reverberant audio signal s rev (n, i) 204 are forwarded together to the signal router 605. In turn, by using the mapping, the signal router 605 is configured to determine the rearranged reverberant audio signal 210, for example, using the following formula:

[0219] s rev-rearr (n,i)=s rev (n, μ(i))

[0220] Then, the rearranged reverberation audio signal s can be outputted. rev-rearr (n, i)210.

[0221] Note that in some embodiments, the rearranged reverberation signal is not implemented as a separate module after the reverberator. An example is an implementation where the reverberated audio signal output from the FDN reverberator is arranged in order from the shortest delay line to the longest delay line (by arranging the delay lines in the FDN accordingly) to provide a reverberated output signal, spatially position matched reverberation to speaker setup distributor 603. Such a spatially positional reverberated output signal can then be rendered from the corresponding spatial position with HRTF filtering or from the speakers.

[0222] about Figure 7 , which shows a flow chart of the operation of the reverberation signal router 207. The method may include the operation of obtaining a reverberation audio signal, a speaker setting parameter, and a delay line length, such as Figure 7 As shown in step 701.

[0223] Furthermore, the method may further include sorting the delay line lengths and generating sorted reverberation channel indices, such as Figure 7 As shown in step 703.

[0224] The next operation may include assigning or generating a reverb channel to speaker setup mapping, such as Figure 7 As shown in step 705.

[0225] Furthermore, the method may include routing the reverberant audio signal based on the mapping to generate a rearranged reverberant audio signal, such as Figure 7 As shown in step 707.

[0226] In addition, the method also includes outputting the rearranged reverberation signal, such as Figure 7 As shown in step 709.

[0227] also, Fig.12 In more detail, the schematic diagram shows Figure 2 The binaural renderer 209 shown in .

[0228] The input of the binaural renderer 209 is the rearranged reverberant audio signal s rev-rearr (n, i) 210 and speaker setting parameters θ ls (i) φ ls (i)208.In Fig.12 In the example shown in , the binaural renderer 209 is organized channel by channel and Fig.12 In the example shown, there are multiple HRTF processors (one per channel) 1201. For example, for the first channel, the HRTF processor 12011 is configured to receive the rearranged reverberant audio signal (channel 1) 2101 and the speaker setup 2081 associated with channel 1. For the second channel, the HRTF processor 12012 is configured to receive the rearranged reverberant audio signal (channel 2) 2102 and the speaker setup 2082 associated with channel 2. The Nth channel HRTF processor 1201 is also shown. N , which is configured to receive the rearranged reverberant audio signal (channel N) 210 N and the speaker setup 208 associated with channel N N Each HRTF processor may include a pair of HRTF filters h selected for each loudspeaker channel i bin (m, j, i) (where m is the time index of the filter coefficient and j is the index of the binaural channel).

[0229] Use HRTF filter to h bin(m, j, i), the reverberant binaural audio signal s can be determined for each channel of the rearranged reverberant audio signal 210 by the following formula: bin (n, j, i) 1202:

[0230]

[0231] in, denotes convolution (in some implementations, filtering can also be performed in the frequency domain instead of time domain convolution). Thus, the reverberant binaural audio signal s bin (n,j,i) 1202 may be passed to a binaural signal combiner 1203 .

[0232] Reverberant binaural audio signal bin (n,j,i) 1202 can be combined in the binaural signal combiner 1203 for different speaker channels i by the following formula:

[0233]

[0234] Get the reverberant binaural signal s bin (n,j)212, which is the output.

[0235] The binaural renderer operates in Fig.13 The flow chart is shown in FIG.

[0236] The first operation is to obtain a re-arranged reverberant audio signal and speaker settings, such as Fig.13 As shown in step 1301.

[0237] Furthermore, the method may include generating a reverberant binaural audio signal (multi-channel) channel by channel by applying HRTF processing, such as Fig.13 As shown in step 1303.

[0238] The method may further comprise combining the reverberant binaural audio signals to generate a reverberant binaural audio signal such as Fig.13 As shown in step 1305.

[0239] Furthermore, a reverberant binaural audio signal can be output, such as Fig.13 As shown in step 1307.

[0240] Fig.15 An example system is schematically shown in which an embodiment is implemented in an encoder device 1901 that performs part of the functionality; data is written into a bitstream 1921 and sent to a renderer device 1941, which decodes the bitstream, performs reverberator processing according to the embodiment and outputs audio for headphone listening.

[0241] Fig.15 The encoder side 1901 of can be executed on the content creator computer and / or the network server computer. The output of the encoder is a bitstream 1921 that can be used for download or streaming. The decoder / renderer 1941 function runs on the end-user device, which can be a mobile device, personal computer, sound bar, tablet computer, car media system, home HiFi or theater system, head mounted display for AR or VR, smart watch, or any suitable system for audio consumption.

[0242] The encoder 1901 is configured to receive a virtual scene description 1900 and an audio signal 1904. The virtual scene description 1900 may be provided in an MPEG-I encoder input format (EIF) or in another suitable format. Generally, the virtual scene description contains an acoustically relevant description of the content of the virtual scene, and for example contains scene geometry as a grid or as voxels, acoustic materials, an acoustic environment with reverberation parameters, sound source locations, and other audio element related parameters (such as whether reverberation is to be rendered for the audio element). In some embodiments, the encoder 1901 includes a scene and reverberation payload encoder 1913, which is configured to generate reverberation parameters.

[0243] The encoder 1901 further comprises an MPEG-H 3D audio encoder 1914 configured to obtain the audio signal 1904 , MPEG-H encode it and pass it to a bitstream encoder 1915 .

[0244] In some embodiments, encoder 1901 further includes a bitstream encoder 1915 configured to receive the output of scene and reverberation payload encoder 1913 and the encoded audio signal from MPEG-H encoder 1914 and generate a bitstream 1921 that can be passed to a bitstream decoder 1941. In some embodiments, bitstream 1921 can be streamed to an end-user device or made available for download or storage.

[0245] In some embodiments, decoder 1941 includes a bitstream decoder 1951 configured to decode a bitstream.

[0246] The decoder 1941 may also include a scene and reverberation payload decoder 1953 configured to obtain encoded reverberation parameters and decode these parameters in a relative or inverse operation to the reverberation payload encoder 1913 .

[0247] The speaker setup determiner 1955 is configured to generate and pass speaker configuration parameters to the reverberant signal router 207 / 1962 and the binaural renderer 209 / 1959 .

[0248] In addition, the head posture generator 1957 receives information from a head mounted device or similar device and generates head posture information or parameters, which can be passed to the binaural renderer 209 / 1959 and the HRTF processor 1963.

[0249] The decoder 1941 includes an MPEG-H 3D audio decoder 1954 configured to decode the audio signal and pass it to a (FDN) reverberator 1961 and a direct sound processing 1965 .

[0250] The decoder 1941 further comprises a (FDN) reverberator 1961 which is configured to achieve a suitable reverberation of the audio signal.

[0251] The output of the (FDN) reverberator 1961 is configured to be output to the reverberation signal router 207 / 1962.

[0252] In some embodiments, the decoder 1941 includes a reverberation signal router 207 / 1962 configured to apply signal routing, as described above.

[0253] The decoder further comprises a binaural renderer 209 / 1959 configured to generate a binaural reverberation audio signal.

[0254] In addition, the decoder / renderer 1941 includes a direct sound processor 1965, which is configured to receive the decoded audio signal and is configured to implement any direct sound processing (such as air absorption and distance gain reduction), which can be passed to the HRTF processor 1963, which, using the head orientation determination (from a suitable sensor), can generate a direct sound component, which is passed to the binaural signal combiner 1967 along with the reverberant component from the HRTF processor 1963. The binaural signal combiner 1967 is configured to combine the direct portion and the reverberant portion to generate a suitable output (e.g., for headphone reproduction).

[0255] Furthermore, in some embodiments, the decoder also includes a head orientation determiner 1991 which passes the head orientation information to the head posture generator 1957 .

[0256] Although in Fig.15 The reverberator output allocation is depicted as occurring in the renderer device, but it may also occur in the encoder device. For example, the encoder device may calculate the appropriate location of the reverberator output and then signal it in the bitstream for different speaker configurations. Furthermore, instead of sending the reverberation parameters from the encoder to the renderer, in some embodiments, the reverberator parameters may be sent in the bitstream. The reverberator parameters refer to FDN parameters such as delay line length, decay filter, reverberation ratio control filter, etc.

[0257] In some embodiments, the assignment of the reverberator outputs to the speaker channels occurs during configuration of the reverberator. The assignment may be stored during configuration and provided to the reverberation signal router.

[0258] In some embodiments, the output is a multi-channel speaker setup (such as a 5.1 or 7.1+4 multi-channel speaker setup). In this case, the output can be modified by the following operations: Figure 2 The proposed process uses the speaker positions of the actual speakers as the speaker settings θ ls (i) φ ls (i) and omitting the binaural renderer, and reproducing the rearranged reverberant audio signal s from the corresponding loudspeakers of the loudspeaker arrangement rev-rearr (n,i). Reference Fig.15 In the case of speaker output, instead of the binaural renderer 209 / 1959, there will be a speaker renderer (or panner) which, in the simplest case, will just pass the speaker signal to a speaker signal combiner which will replace the binaural signal combiner 1967. Accordingly, the direct sound part is spatialized using a panner (such as the implemented VBAP) instead of the HRTF processor 1963.

[0259] In some embodiments, rather than arranging the delay lines to provide the maximum width of the reproduced reverberant sound image as suggested above, the delay lines are arranged so that the maximum width is created along the longest dimension of the (physical or virtual) room. This may be particularly advantageous if head tracking is applied to the reverberant signal so that the reverberant signal will remain fixed to world coordinates when the listener's head rotates. In this embodiment, the method is as described above, but the two initial directions are chosen to be opposite directions in the longest dimension of the virtual or physical space. For example, assume that the space is longest along the x-dimension. Then, the shortest delay line can be assigned to the direction towards the positive x-axis, and the next shortest delay line can be assigned to the direction towards the negative x-axis.

[0260] Another example is when using head tracking, it is desirable for the reverberation to sound equally wide in all directions, rather than maximizing the width of the reverberation. In this case, the algorithm for assigning the reverberator output to the speakers will not select the farthest speaker channel as the next channel, but rather the next speaker channel that is free and far enough away from the previously selected channel. Far enough may be determined by a threshold, which may be expressed, for example, in degrees. A suitable threshold may be, for example, 90 degrees.

[0261] In some embodiments, there are N independent reverberators, rather than a single multi-channel reverberator. In this case, the reverberators may be arranged in order of increasing delay they cause to the signal. Furthermore, the method may operate as described above.

[0262] In some embodiments, the speaker positions are almost evenly distributed on the sphere. The sequenced reverberator outputs are routed to the following output speaker positions:

[0263] OutputDirection{90,0},

[0264] OutputDirection{-90,0},

[0265] OutputDirection{114,20},

[0266] OutputDirection{-60,-6},

[0267] OutputDirection{85,-44},

[0268] OutputDirection{-130,-21},

[0269] OutputDirection{49,41},

[0270] OutputDirection{-67,52},

[0271] OutputDirection{154,-9},

[0272] OutputDirection{-48,-55},

[0273] OutputDirection{19,-31},

[0274] OutputDirection{-162,21},

[0275] OutputDirection{151,73},

[0276] OutputDirection{-9,20},

[0277] OutputDirection{180,-63},

[0278] The underlying Matlab code illustrating the maximum width algorithm for an example speaker configuration is described below.

[0279]

[0280]

[0281] In some embodiments, in addition to finding a "virtual speaker" channel position whose distance to a previously selected "virtual speaker" channel position is greater than a threshold, the assignment of the "virtual speaker" channel also checks whether the selected "virtual speaker" positions are not too close to each other. In other words, the method is configured to check whether the potential next "virtual speaker" position is not closer to the already selected "virtual speaker" position than a threshold (e.g., 10 degrees azimuth / elevation). Alternatively, the method can first pre-process the speaker settings and disable "virtual speakers" that are too close to other speakers (so that they cannot be selected during the process).

[0282] In the above example, the allocation of "virtual speaker" channels is based on the order of channels defined by the delay length of each channel. However, in some embodiments, the allocation is implemented without sorting. For example, in some embodiments, rather than sorting each channel, channel groups are selected based on a delay length range and then allocated group by group so that channels with similar delay lengths are allocated as groups to be spaced apart from each other.

[0283] about Fig.16 , shows an example electronic device that can be used as any device part of the system described above. The device can be any suitable electronic device or device. For example, in some embodiments, the device 2000 is a mobile device, a user device, a tablet computer, a computer, an audio playback device, etc. The device can be configured to implement an encoder or a renderer or any functional block as described above.

[0284] In some embodiments, the device 2000 includes at least one processor or central processing unit 2007. The processor 2007 may be configured to execute various program codes, such as the methods described herein.

[0285] In some embodiments, the device 2000 includes a memory 2011. In some embodiments, at least one processor 2007 is coupled to the memory 2011. The memory 2011 can be any suitable storage component. In some embodiments, the memory 2011 includes a program code portion for storing program codes that can be implemented on the processor 2007. In addition, in some embodiments, the memory 2011 can also include a storage data portion for storing data (e.g., data that has been processed or will be processed according to the embodiments described herein). When needed, the implemented program code stored in the program code portion and the data stored in the storage data portion can be retrieved by the processor 2007 via the memory-processor coupling.

[0286] In some embodiments, the device 2000 includes a user interface 2005. In some embodiments, the user interface 2005 may be coupled to the processor 2007. In some embodiments, the processor 2007 may control the operation of the user interface 2005 and receive input from the user interface 2005. In some embodiments, the user interface 2005 may enable a user to enter commands to the device 2000, for example, via a keypad. In some embodiments, the user interface 2005 may enable a user to obtain information from the device 2000. For example, the user interface 2005 may include a display configured to display information from the device 2000 to the user. In some embodiments, the user interface 2005 may include a touch screen or touch interface that enables information to be input into the device 2000 and to display information to a user of the device 2000. In some embodiments, the user interface 2005 may be a user interface for communication.

[0287] In some embodiments, the device 2000 includes an input / output port 2009. In some embodiments, the input / output port 2009 includes a transceiver. In such embodiments, the transceiver can be coupled to the processor 2007 and configured to communicate with other devices or electronic devices, for example, via a wireless communication network. In some embodiments, the transceiver or any suitable transceiver or transmitter and / or receiver components can be configured to communicate with other electronic devices or devices via wired or wired coupling.

[0288] The transceiver can communicate with other devices via any suitable known communication protocol. For example, in some embodiments, the transceiver can use a suitable Universal Mobile Telecommunications System (UMTS) protocol, a wireless local area network (WLAN) protocol such as IEEE 802.X, a suitable short-range radio frequency communication protocol such as Bluetooth, or an infrared data communication path (IRDA).

[0289] Input / output port 2009 may be configured to receive signals.

[0290] In some embodiments, device 2000 may be used as at least part of a renderer.Input / output port 2009 may be coupled to a headset (which may be a head tracking or non-tracking headset) or the like.

[0291] In general, various embodiments of the present invention may be implemented in hardware or dedicated circuits, software, logic, or any combination thereof. For example, some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device, but the present invention is not limited thereto. Although various aspects of the present invention may be illustrated and described as block diagrams, flow charts, or using some other graphical representation, it is well known that the boxes, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general hardware or controllers or other computing devices, or some combination thereof as non-limiting examples.

[0292] Embodiments of the present invention may be implemented by computer software executable by a data processor of a mobile device (such as in a processor entity), or by hardware, or by a combination of software and hardware. In addition, in this regard, it should be noted that any block of the logic flow as in the accompanying drawings may represent program steps, or interconnected logic circuits, blocks and functions, or a combination of program steps and logic circuits, blocks and functions. The software may be stored on a physical medium such as a memory chip or a memory block implemented within a processor, on a magnetic medium such as a hard disk or a floppy disk, and on an optical medium such as a DVD and its data variants, a CD.

[0293] The memory may be of any type suitable for the local technical environment and may be implemented using any suitable data storage technology, such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory, and removable memory. The data processor may be of any type suitable for the local technical environment and may include, as non-limiting examples, one or more of a general purpose computer, a special purpose computer, a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a gate level circuit based on a multi-core processor architecture, and a processor.

[0294] Embodiments of the present invention may be practiced in various components such as integrated circuit modules. The design of integrated circuits is generally a highly automated process. Complex and powerful software tools are available to convert a logic level design into a semiconductor circuit design ready to be etched and formed on a semiconductor substrate.

[0295] Programs, such as those offered by Synopsys, Inc. of Mountain View, Calif., and Cadence Design, of San Jose, Calif., use well-established design rules and a library of pre-stored design modules to automatically route conductors and position components on a semiconductor chip. Once the design of a semiconductor circuit is complete, the resulting design in a standardized electronic format (e.g., Opus, GDSII, etc.) can be transmitted to a semiconductor manufacturing facility or "fab" for fabrication.

[0296] The foregoing description has provided a complete and useful description of exemplary embodiments of the present invention by way of exemplary and non-limiting examples. However, various modifications and adaptations will become apparent to those skilled in the relevant art in view of the above description when read in conjunction with the accompanying drawings and the appended claims. However, all such and similar modifications of the teachings of the present invention will still fall within the scope of the present invention as defined by the appended claims.

Claims

1. An apparatus for applying reverberation to at least one audio signal, the apparatus comprising components configured to: obtaining the at least one audio signal; obtaining a plurality of sound reproduction positions for playing back audio associated with a listening position; determining a reverberator, the reverberator comprising a plurality of channels, wherein the plurality of channels are configured with at least two corresponding delay lines having different delays; assigning the reverberator channels to the sound reproduction positions based on delays of the plurality of channels; and Based on the assigned channels, the at least one audio signal is processed with the reverberator to generate a reverberant playback audio signal for the listening position.

2. The device according to claim 1, wherein: The component is further configured to: determining a directional filter pair associated with each of the sound reproduction locations for playing back audio; and The reverberant playback audio signal is filtered using the associated pair of directional filters to generate a reverberant playback binaural audio signal.

3. The device according to any one of claims 1 or 2, wherein: The component configured to assign the channels to the sound reproduction positions based on delays of the channels is configured to: sorting the channels based on the delays; as well as Based on the ranking of the channels, the channels are assigned to the sound reproduction positions.

4. The device according to any one of claims 1 to 3, wherein: The means configured to assign the channels to the sound reproduction positions based on the delays is configured to assign channels having similar delay lengths to the sound reproduction positions distributed away from each other.

5. Apparatus according to claim 4 when dependent on claim 3, wherein: The component configured to assign channels having similar delay lengths to sound reproduction positions distributed far from each other is configured to: assigning a first ordered channel of the ordered channels to a first sound reproduction position; and Subsequent ordered channels are assigned to sound reproduction positions that are more than a threshold distance from previously assigned sound reproduction positions.

6. The device according to claim 5, wherein: The component configured to assign subsequent sorted channels to sound reproduction positions that are more than a threshold distance away from the previously assigned sound reproduction position is configured to assign subsequent sorted channels to the currently unassigned sound reproduction position that is farthest from the immediately previous assigned sound reproduction position.

7. The device according to any one of claims 1 to 6, wherein: The component configured to assign the channel to the sound reproduction position based on the delay is configured to: determining that the number of the sound reproduction positions is less than the number of the channels; as well as More than one channel is assigned to at least one of the sound reproduction positions such that each channel is assigned to one of the sound reproduction positions.

8. The device according to any one of claims 1 to 7, wherein: The component configured to assign the channel to the sound reproduction position based on the delay is configured to: determining that the number of the sound reproduction positions is greater than the number of the reverberator channels; generating at least one other reverberator channel; and The at least one other reverberator channel is assigned to at least one of the sound reproduction positions that have not been previously assigned.

9. The device according to claim 4, wherein: The component configured to determine the reverberator, the reverberator comprising the plurality of channels, wherein the plurality of channels are configured with respective delay lines having different delays is configured to: obtaining at least one dimension of a scene associated with the reverberation; and At least one time delay of a reflection path is determined based on the at least one dimension of the scene, wherein the delay line delay is based on the at least one time delay determined for the reflection path.

10. The device according to claim 9, wherein: The component configured to assign the channels to the sound reproduction positions based on the delays is configured to assign the channels to the sound reproduction positions further based on the at least one dimension of the scene.

11. The device according to claim 10, wherein: The component configured to assign the channels to the sound reproduction positions further based on the at least one dimension of the scene is configured to: based on a longest dimension of the scene, selecting two initial directions from the sound reproduction position such that the two initial directions are separated by the longest dimension; assigning a first channel to a first of the two initial directions from the sound reproduction position; as well as A second channel is assigned to a second direction of the two initial directions from the sound reproduction position, wherein a corresponding delay line delay of the second channel is similar to or equal to a corresponding delay line delay of the first channel, and the corresponding delay line delay of the second channel and the corresponding delay line delay of the first channel are delay lines of two shortest delays.

12. An apparatus according to claim 3 or any claim dependent thereon, wherein: The component configured to assign the channel to the sound reproduction position based on the delay is configured to: assigning a first ordered channel to a first sound reproduction position; and Subsequent ordered channels are assigned to unassigned sound reproduction positions that are the smallest distances from previously assigned sound reproduction positions.

13. An apparatus according to claim 3 or any claim dependent thereon, wherein: The reverberator comprises the plurality of reverberators, each of the plurality of reverberators comprising a single channel, and wherein the component configured to sort the channels based on the delay line delay is configured to arrange the plurality of reverberators in order of increasing delay line delay.

14. The device according to any one of claims 1 to 13, wherein: The reverberator comprises a feedback delay network reverberator.

15. The device according to any one of claims 1 to 14, wherein: The channels are configured with corresponding delay lines having different delays.

16. The device according to claim 15, wherein: The channels are configured with corresponding delay lines having different delays that make them mutually prime.

17. A method for applying a reverberation device to at least one audio signal, the method comprising: obtaining the at least one audio signal; obtaining a plurality of sound reproduction positions for playing back audio associated with a listening position; determining a reverberator, the reverberator comprising a plurality of channels, wherein the plurality of channels are configured with at least two corresponding delay lines having different delays; assigning the reverberator channels to the sound reproduction positions based on delays of the plurality of channels; and Based on the assigned channels, the at least one audio signal is processed with the reverberator to generate a reverberant playback audio signal for the listening position.

18. The method according to claim 17, further comprising: determining a directional filter pair associated with each of the sound reproduction locations for playing back audio; as well as The reverberant playback audio signal is filtered using the associated pair of directional filters to generate a reverberant playback binaural audio signal.

19. The method according to any one of claims 17 or 18, wherein: Assigning the channels to the sound reproduction positions based on the delays of the channels comprises: Based on the delays, ordering the channels; and Based on the ranking of the channels, the channels are assigned to the sound reproduction positions.

20. The method according to any one of claims 17 to 19, wherein: Based on the delays, assigning the channels to the sound reproduction positions includes assigning channels having similar delay lengths to the sound reproduction positions that are distributed far from each other.

21. An apparatus for applying reverberation to at least one audio signal, the apparatus comprising at least one processor and at least one memory including computer program code, the at least one memory and the computer program code being configured to, with the at least one processor, cause the apparatus to at least: obtaining the at least one audio signal; obtaining a plurality of sound reproduction positions for playing back audio associated with a listening position; determining a reverberator, the reverberator comprising a plurality of channels, wherein the plurality of channels are configured with at least two corresponding delay lines having different delays; assigning the reverberator channels to the sound reproduction positions based on delays of the plurality of channels; and Based on the assigned channels, the at least one audio signal is processed with the reverberator to generate a reverberant playback audio signal for the listening position.

Images