Rendering reverberation with start-up control

By using reverb parameters to control the digital reverb and delay line gain filters in virtual acoustic rendering, the problem of early reflection and late reverb interference is solved, improving the audio quality and the realism of the virtual space.

CN118921589APending Publication Date: 2024-11-08NOKIA TECHNOLOGIES OY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410553058.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-05-08
Filing Date
2024-05-07
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

In the prior art, in virtual acoustic rendering, it is difficult to effectively control the start behavior of the reverb, resulting in greater interference between early reflection and late reverb, affecting the reality and audio quality of the virtual space.

Method used

By obtaining reverb parameters, configuring digital reverbs and control lines, using delay lines and gain filters to generate reverb audio signals, significantly attenuating early reflection interference in late reverbs, achieving control of the reverb's startup behavior and reducing early reflections Interference to late reverb.

Benefits of technology

Improve the audio quality in virtual acoustic rendering, enhance the realism of the virtual space, and effectively control the start behavior of the reverb, reducing the interference between early reflections and later reverbs, achieving a more natural reverb effect.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118921589A_ABST
    Figure CN118921589A_ABST
Patent Text Reader

Abstract

A method for applying reverberation to at least one audio signal, the method comprising: obtaining at least one audio signal; obtaining at least one reverberation parameter; controlling the digital reverberator based on the at least one reverberation parameter; providing at least one control line, the at least one control line comprising a delay line and a gain filter, the at least one control line using at least one reverberation parameter to generate an output from the at least one control line; and generating at least one reverberated audio signal based on at least one output of the at least one digital reverberator and the generated output from the at least one control line, where at least a portion of a later reverberation interfering with the at least one reflected echo in the generated at least one reverberated audio signal is significantly attenuated.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to an apparatus and method for rendering reverberation using activation control, and not only to an apparatus and method for rendering reverberation using activation control in an augmented reality and / or virtual reality device. Background Art

[0002] Reverberation is the persistence of sound in a space after the actual sound source has ceased. Different spaces are characterized by different reverberation properties. In order to convey the spatial impression of an environment, it is important to reproduce the reverberation perceptually accurately. Room acoustics are usually modeled with a separately synthesized early reflection component and a statistical model for the diffuse late reverberation. Figure 1 An example of a synthesized room impulse response is depicted, where direct sound 101 is followed by discrete early reflections 103 (or reflection echoes) with directions of arrival (DOA) and diffuse late reverberation 105 that may be synthesized without any specific direction of arrival. Figure 1 The delay d1(t) 102 in FIG. 5 can be viewed as representing the direct sound arrival delay from the source to the listener, and the delay d2(t) 104 can represent the delay from the source to the listener for one of the early reflections (in this case the first to arrive).

[0003] One way to reproduce reverberation is to use a set of D loudspeakers (or virtual loudspeakers using a set of head-related transfer functions (HRTF) binaural reproduction). These loudspeakers are positioned somewhat uniformly around the listener. Mutually incoherent reverberant signals are reproduced from these loudspeakers, resulting in the perception of ambient diffuse reverberation.

[0004] The reverberations produced by different loudspeakers must be mutually incoherent. In a simple case, reverberations can be produced using different channels of the same reverberator, where the output channels are uncorrelated but otherwise share the same acoustic properties, such as reverberation time and level (specifically, diffuse-to-direct ratio or reverberant-to-direct ratio or diffuse-to-total ratio or diffuse-to-source ratio or any other suitable parameter for representing reverberation energy or level). Such uncorrelated outputs sharing the same acoustic properties can be obtained, for example, by tapping the output of a feedback delay network (FDN) reverberator (with appropriately tuned delay line lengths and mixing matrices), or from a reverberator based on the use of decaying uncorrelated noise (by using a different uncorrelated noise sequence in each channel). In this case, the different reverberation signals actually have the same characteristics, and the reverberation is generally perceived to be similar in all directions. Summary of the invention

[0005] According to a first aspect, there is provided a method for applying reverberation to at least one audio signal, the method comprising: obtaining at least one audio signal; obtaining at least one reverberation parameter; controlling a digital reverberator based on the at least one reverberation parameter; providing at least one control line, the at least one control line comprising a delay line and a gain filter, the at least one control line using the at least one reverberation parameter to generate an output from the at least one control line; and generating at least one reverberated audio signal based on at least one output of the at least one digital reverberator and the generated output from the at least one control line, wherein at least a portion of late reverberation interfering with at least one reflected echo in the generated at least one reverberated audio signal is significantly attenuated.

[0006] Obtaining at least one reverberation parameter may include obtaining at least one reverberation time.

[0007] Based on the at least one reverberation parameter, controlling the digital reverberator may include applying at least one delay line to the at least one audio signal to generate at least two echoes.

[0008] Providing at least one control line may include providing at least one control echo.

[0009] Generating at least one reverberant audio signal based on at least one output of the at least one digital reverberator and the generated output from the at least one control line may include generating at least one reverberant audio signal based on a combination of at least one audio signal comprising at least two echoes from the at least one output of the digital reverberator and at least one audio signal comprising at least one control echo from the at least one output of the at least one control line.

[0010] Based on the at least one output of the at least one digital reverberator and the generated output from the at least one control line, generating the at least one reverberant audio signal may include providing at least one control echo to at least partially suppress a first echo of the at least two echoes so that the at least one reverberant audio signal includes reverberation that does not interfere with the at least one reflected echo.

[0011] At least one reflection echo is an early reflection that is rendered separately.

[0012] Obtaining at least one reverberation parameter may include obtaining the at least one reverberation parameter based on at least one size parameter associated with a virtual acoustic space in which the audio signal is rendered.

[0013] Based on the at least one reverberation parameter, controlling the digital reverberator may include controlling at least one feedback attenuation filter.

[0014] Based on the at least one reverberation parameter, controlling the digital reverberator may include controlling a feedback matrix configured to feed back at least one output of the digital reverberator to at least one input of the digital reverberator.

[0015] The at least one control line may be a first number of control lines, and the digital reverberator may include a second number of delay lines.

[0016] The first number may be less than or equal to the second number.

[0017] At least a portion of the late reverberation may be at least an early portion of the late reverberation.

[0018] According to a second aspect, there is provided an apparatus for applying reverberation to at least one audio signal, the apparatus comprising at least one processor and at least one memory storing instructions, which instructions, when executed by the at least one processor, cause the system to at least perform: obtaining at least one audio signal; obtaining at least one reverberation parameter; controlling a digital reverberator based on the at least one reverberation parameter; providing at least one control line, the at least one control line comprising a delay line and a gain filter, the at least one control line using the at least one reverberation parameter to generate an output from the at least one control line; and generating at least one reverberant audio signal based on at least one output of the at least one digital reverberator and the generated output from the at least one control line, wherein at least a portion of late reverberation interfering with at least one reflected echo in the generated at least one reverberant audio signal is significantly attenuated.

[0019] The apparatus caused to perform the obtaining of at least one reverberation parameter may further be configured to obtain at least one reverberation time.

[0020] The apparatus caused to perform controlling the digital reverberator based on at least one reverberation parameter may be configured to apply at least one delay line to at least one audio signal to generate at least two echoes.

[0021] The apparatus caused to perform providing at least one control line may further be configured to provide at least one control echo.

[0022] The apparatus caused to perform generating at least one reverberant audio signal based on at least one output of the at least one digital reverberator and the generated output from the at least one control line may be further configured to generate the at least one reverberant audio signal based on a combination of at least one audio signal comprising at least two echoes from the at least one output of the digital reverberator and at least one audio signal comprising at least one control echo from the at least one output of the at least one control line.

[0023] The apparatus caused to perform generating at least one reverberant audio signal based on at least one output of the at least one digital reverberator and an output generated from the at least one control line may be further configured to provide at least one control echo to at least partially suppress a first echo of the at least two echoes so that the at least one reverberant audio signal includes reverberation that does not interfere with the at least one reflected echo.

[0024] The apparatus caused to perform the obtaining of at least one reverberation parameter may be configured to obtain the at least one reverberation parameter based on at least one size parameter associated with a virtual acoustic space in which the audio signal is rendered.

[0025] The apparatus caused to perform controlling the digital reverberator based on at least one reverberation parameter may be configured to control at least one feedback attenuation filter.

[0026] The apparatus caused to perform controlling the digital reverberator based on at least one reverberation parameter may be configured to control a feedback matrix configured to feed back at least one output of the digital reverberator to at least one input of the digital reverberator.

[0027] The at least one control line may be a first number of control lines, and the digital reverberator may be configured as a second number of delay lines.

[0028] The first number may be less than or equal to the second number.

[0029] At least a portion of the late reverberation may be at least an early portion of the late reverberation.

[0030] According to a third aspect, there is provided an apparatus for applying reverberation to at least one audio signal, the apparatus comprising components configured to: obtain at least one audio signal; obtain at least one reverberation parameter; control a digital reverberator based on the at least one reverberation parameter; provide at least one control line, the at least one control line comprising a delay line and a gain filter, the at least one control line using the at least one reverberation parameter to generate an output from the at least one control line; and generate at least one reverberated audio signal based on at least one output of the at least one digital reverberator and the generated output from the at least one control line, wherein at least a portion of late reverberation interfering with at least one reflected echo in the generated at least one reverberated audio signal is significantly attenuated.

[0031] The component configured to obtain at least one reverberation parameter may be configured to obtain at least one reverberation time.

[0032] The component configured to control the digital reverberator based on the at least one reverberation parameter may be configured to apply at least one delay line to the at least one audio signal to generate at least two echoes.

[0033] The component configured to provide at least one control line may be configured to provide at least one control echo.

[0034] The component configured to generate at least one reverberant audio signal based on at least one output of the at least one digital reverberator and the generated output from the at least one control line may be configured to generate the at least one reverberant audio signal based on a combination of at least one audio signal comprising at least two echoes from the at least one output of the digital reverberator and at least one audio signal comprising at least one control echo from the at least one output of the at least one control line.

[0035] The component configured to generate the at least one reverberant audio signal based on the at least one output of the at least one digital reverberator and the generated output from the at least one control line may be configured to provide at least one control echo to at least partially suppress a first echo of the at least two echoes so that the at least one reverberant audio signal includes reverberation that does not interfere with the at least one reflected echo.

[0036] At least one reflection echo is an early reflection that is rendered separately.

[0037] The component configured to obtain the at least one reverberation parameter may be configured to obtain the at least one reverberation parameter based on at least one size parameter associated with a virtual acoustic space in which the audio signal is rendered.

[0038] The component configured to control the digital reverberator based on the at least one reverberation parameter may be configured to control at least one feedback attenuation filter.

[0039] The component configured to control the digital reverberator based on the at least one reverberation parameter may be configured to control a feedback matrix configured to feed back at least one output of the digital reverberator to at least one input of the digital reverberator.

[0040] The at least one control line may be a first number of control lines, and the digital reverberator may be configured as a second number of delay lines.

[0041] The first number may be less than or equal to the second number.

[0042] At least a portion of the late reverberation may be at least an early portion of the late reverberation.

[0043] According to a fourth aspect, there is provided an apparatus for applying reverberation to at least one audio signal, the apparatus comprising: an acquisition circuit configured to obtain at least one audio signal; an acquisition circuit configured to obtain at least one reverberation parameter; a control circuit configured to control a digital reverberator based on the at least one reverberation parameter; providing at least one control line, the at least one control line comprising a delay line and a gain filter, the at least one control line using the at least one reverberation parameter to generate an output from the at least one control line; and generating at least one reverberated audio signal based on at least one output of the at least one digital reverberator and the generated output from the at least one control line, wherein at least a portion of a late reverberation interfering with at least one reflected echo in the generated at least one reverberated audio signal is significantly attenuated.

[0044] According to a fifth aspect, there is provided a computer program [or a computer-readable medium comprising instructions], the instructions being for causing an apparatus to apply reverberation to at least one audio signal, the apparatus being caused to perform at least the following operations: obtain at least one audio signal; obtain at least one reverberation parameter; control a digital reverberator based on the at least one reverberation parameter; provide at least one control line, the at least one control line comprising a delay line and a gain filter, the at least one control line using the at least one reverberation parameter to generate an output from the at least one control line; and generate at least one reverberant audio signal based on at least one output of the at least one digital reverberator and the generated output from the at least one control line, wherein at least a portion of late reverberation interfering with at least one reflected echo in the generated at least one reverberant audio signal is significantly attenuated.

[0045] According to a sixth aspect, there is provided a non-transitory computer-readable medium comprising program instructions for causing an apparatus for applying reverberation to at least one audio signal to at least perform the following operations: obtain at least one audio signal; obtain at least one reverberation parameter; control a digital reverberator based on the at least one reverberation parameter; provide at least one control line, the at least one control line comprising a delay line and a gain filter, the at least one control line using the at least one reverberation parameter to generate an output from the at least one control line; and generate at least one reverberant audio signal based on at least one output of the at least one digital reverberator and the generated output from the at least one control line, wherein at least a portion of late reverberation interfering with at least one reflected echo in the generated at least one reverberant audio signal is significantly attenuated.

[0046] According to a seventh aspect, there is provided an apparatus for applying reverberation to at least one audio signal, comprising: means for obtaining at least one audio signal; means for obtaining at least one reverberation parameter; means for controlling a digital reverberator based on the at least one reverberation parameter; means for providing at least one control line, wherein the at least one control line comprises a delay line and a gain filter, the at least one control line using the at least one reverberation parameter to generate an output from the at least one control line; and means for generating at least one reverberated audio signal based on at least one output of the at least one digital reverberator and the generated output from the at least one control line, wherein at least a portion of a late reverberation interfering with at least one reflected echo in the generated at least one reverberated audio signal is significantly attenuated.

[0047] According to an eighth aspect, there is provided a computer-readable medium comprising instructions for causing an apparatus for applying reverberation to at least one audio signal to at least perform the following operations: obtain at least one audio signal; obtain at least one reverberation parameter; control a digital reverberator based on the at least one reverberation parameter; provide at least one control line, the at least one control line comprising a delay line and a gain filter, the at least one control line using the at least one reverberation parameter to generate an output from the at least one control line; and generate at least one reverberated audio signal based on at least one output of the at least one digital reverberator and the generated output from the at least one control line, wherein at least a portion of late reverberation interfering with at least one reflected echo in the generated at least one reverberated audio signal is significantly attenuated.

[0048] An apparatus comprises means for performing the actions of the method as described above.

[0049] A device is configured to perform the actions of the method described above.

[0050] A computer program includes program instructions for causing a computer to execute the method as described above.

[0051] A computer program product stored on a medium may cause an apparatus to perform the method described herein.

[0052] An electronic device may include an apparatus as described herein.

[0053] A chipset may include the apparatus as described herein.

[0054] The embodiments of the present application are intended to solve the problems associated with the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] For a better understanding of the present application, reference will now be made by way of example to the accompanying drawings, in which:

[0056] Figure 1 showing a room acoustic model with respect to the room impulse response;

[0057] Figure 2 schematically illustrates example apparatus in which some embodiments may be implemented;

[0058] Figure 3 Shown as Figure 2 A flowchart of the operation of the illustrated example apparatus with respect to rendering of a reverberant audio signal;

[0059] Figure 4 In more detail, the following schematically illustrates how Figure 2 A reverberator including an example feedback delay network (FDN) with attack control is shown in FIG.

[0060] Figure 5 In more detail, the following schematically illustrates how Figure 2 The example reverberator parameter determiner shown;

[0061] Figure 6 Shown as Figure 5 A flow chart of the operation of an example reverberator parameter determiner is shown;

[0062] Figure 7 In more detail, the following schematically illustrates how Figure 2 Example binaural renderer shown;

[0063] Figure 8 Shown as Figure 7 A flowchart of the operation of an example binaural renderer is shown;

[0064] Fig. 9 Schematically illustrating how Figure 2 Example early reflections processor and renderer shown;

[0065] Fig.10 Shows example early reflection processing;

[0066] Fig.11 illustrates an example system in which some embodiments may be implemented; and

[0067] Fig.12 An example apparatus suitable for implementing the means shown in the preceding Figures is shown. DETAILED DESCRIPTION

[0068] Suitable means and possible mechanisms for controlling the startup behavior parameterization and rendering of an audio scene with diffuse reverberation are described in more detail below.

[0069] In a virtual acoustic rendering system, reverberation is usually rendered as a combination of a certain number of different early reflections (or reflection echoes) and a random model for late reverberation. Therefore, early reflection synthesis is usually position-dependent because it varies with the source and listener position, while late reverberation synthesis is not. Together, these two can create a reasonable reverberation rendering for a physical or virtual space. Reverberation rendering is combined (added) with direct sound rendering, which involves distance gain attenuation, air absorption filtering, and directional reproduction (binaural or loudspeaker) of direct sound components (propagating directly to the listener's ears without reflections or reverberation in space).

[0070] In order to produce a good quality reverberation output, the late reverberation is configured so that the transition from early reflections to late reverberation is perceived as smooth and continuous, without noticeable gaps or energy fluctuations. Rendering is achieved by constructing the impulse response offline (or as a background process), which in turn can process the part of the impulse response corresponding to the late part to ensure that it fits well with the early reflections.

[0071] However, when rendering is performed without creating the impulse responses offline using, for example, a digital reverberator, such offline processing of the impulse responses is not possible because the impulse responses may not be available in the system. An example of such a system generates early reflections using a geometric model and generates late reverberation using a feedback delay network (FDN) digital reverberator.

[0072] It has been suggested to filter the input signal using a second FDN with a shorter reverberation time, thereby generating a second reverberation signal, which in turn can be inverted relative to the main FDN and added to the output signal of the main FDN to achieve suppression (or control) of the early echoes (or impulses, if impulse responses are involved) generated by the main FDN and reduce their interference with the separately rendered early reflections. However, this can be computationally complex, as it requires running two complete FDNs in parallel (and therefore, requiring double the computations compared to a single FDN). Furthermore, such suggested approaches can be difficult to configure, as they require the use of two complete sets of reverberator parameters for configuration.

[0073] The concept, as discussed in more detail below with respect to examples and embodiments, is to control the startup behavior of a feedback network (FN) of a reverberator, but with lower computational complexity than two concurrent FNs, as in the previous dual FDN example, and with the aim of being simple to configure and achieving an improved interaction between late reverberation and synthesized early reflections.

[0074] This example selectively suppresses the first (early) echo from the FN, thereby producing a fixed delay early echo of the FN that is less disruptive to the synthetic (variable delay, position-dependent) early reflections. This suppression of the first echo may be considered to achieve a desired fade-in behavior or start-up control of a reverberator.

[0075] With regard to the reverberator and the reflection processor, they are configured to generate audio signals associated with echoes within the system. For example, the (early) reflection processor is configured to generate a discrete number of echoes that are specular with respect to the features and geometry of the modeled room and are accordingly precise and varied in their direction of arrival, intensity and coloration. Therefore, the echoes generated by the (early) reflection processor are referred to as reflection echoes in the following. The reflection processor is located outside the (late) reverberator and runs in parallel with the (late) reverberator, which is configured to generate the late reverberation.

[0076] A (late) reverberator is configured to generate, by means of a feedback network, a decaying stream of many echoes which increase in number (density) and decrease in intensity (loudness) over time. This is achieved via a feedback aspect, where the input audio signal is passed through the network, split into many "echoes", all of which are then recirculated through the same network again, each of which is split into more echoes, and so on. These echoes correspond only loosely to the geometry of the room and therefore do not represent geometrically exact (specular) reflections.

[0077] Although the preceding and following discussions refer to the FN being configured to generate "early" echoes, the term "early" describes the perceptual distinction between the sparse echoes in the early response of the reverberator and the denser, diffuse echoes that arise from the feedback process and define the late response of the reverberator. Thus, the FN is configured to generate a series (e.g., at least two consecutive) of echoes that are perceptually distinguishable as separate / individual echoes and that are early in the response of the reverberator.

[0078] In general, early echoes may refer to echoes generated, for example, from one to five passes through the feedback loop of the FN and up to a particular transition time. However, with respect to the following embodiments, the suppressed early echoes are the early echoes generated from the first pass through the delay line and feedback attenuation filter of the FN before recirculation, and are specifically referred to as the first echo or the first (early) echo. In contrast, echoes generated from successive passes through the FN are referred to as early echoes, up to the transition time, after which further passes through the FN produce perceptually indistinguishable echoes (referred to as late echoes), diffuse tails, late reverberation, or the like. In addition to the various early echoes generated by the reverberator, so-called reflection echoes are generated by an (early) reflection processor that is external to the reverberator and runs in parallel with the reverberator.

[0079] Thus, in summary, the concepts further defined with respect to the following embodiments relate to the reproduction of (late) reverberation, wherein apparatus and methods are proposed that enable control of the startup behavior of a digital reverberator so that the early stages of the rendered late reverberation are less interfering with the separately rendered reflection echoes. The result is an improved virtual acoustic rendering for a physical or virtual room, wherein the early reflections blend well with the diffuse late reverberation. In some embodiments, this may be achieved by:

[0080] obtaining reverberation parameters including at least reverberation time;

[0081] According to these reverberation parameters, configure the digital reverberator;

[0082] configuring a first echo suppressor using the reverberator parameters, the first echo suppressor comprising at least one controlled delay line and at least one controlled line gain filter;

[0083] generating an output from a first echo suppressor using at least one input signal; and

[0084] A reverberated signal is rendered using a reverberator and at least one input signal, while an output of a first echo suppressor is used to control an attack behavior of the reverberator to achieve late reverberation, thereby minimizing interference with separately rendered early reflections.

[0085] In some embodiments, the feedback network (FN) of the reverberator is a feedback delay network (FDN) that produces 15 output channels. These output channels may be associated with 15 spatial directions around the listener. The output of the reverberator may be reproduced with (virtual or real) speakers in the corresponding directions.

[0086] In some embodiments, a desired reverberation time (RT60) parameter is estimated. In turn, the delay line decay filter of the FN is adjusted using the RT60 parameter and at least one dimensional parameter associated with the room.

[0087] Furthermore, in some embodiments, the control delay line is adjusted to have the same length as the FN delay line. The number of control delay lines need not be equal to the number of FN delay lines. If it is not necessary to suppress or control all echoes corresponding to all first (early) echoes of all FN delay lines, there may be fewer control delay lines. Furthermore, in some embodiments, a control line gain filter may be added to each control delay line to achieve the desired start-up control behavior. The functional pairing of a control delay line with a subsequent control line gain filter will be designated as a control line. The first echo suppressor includes one or more control lines.

[0088] In addition, in some embodiments, the FN can be used to reverberate the input audio signal. The input audio signal can also be fed to the control delay line of the first echo suppressor. In addition, the output from the first echo suppressor can also be combined (added) with the FN output. Doing so can effectively suppress the first echo of the FN.

[0089] In addition, in some embodiments, the (early) reflection echoes are rendered separately using, for example, an image source model. The reflection echoes are synthesized using tapped delay lines and absorptive filtering. Furthermore, the synthesized reflection echo signal is combined with a synthesized late reverberation signal in which the first echo has been suppressed.

[0090] Thus, early reflections are primarily perceived by the early reflection echo rendering path, while late diffuse reverberation is perceived by the reverberator rendering path. Since there is less interference between the reflection echo and the onset of the late reverberation, the resulting reverberation is perceived as varying depending on the listener position. Interference refers to two signals interfering or overlapping (or nearly overlapping) in time. Interference results in a reduction in the perceived significance of the more important signal (in this case, the reflection echo). As discussed in these embodiments, the perception of the reflection echo is increased by a reduction in interference.

[0091] In some embodiments, as output there are 15 mutually (almost) incoherent reverberation signals. The output signals can then be reproduced using loudspeakers located in directions corresponding to the spatial directions used above and whose levels are controlled using channel gain coefficients (or alternatively using virtual loudspeakers that are convolved with the HRTFs or alternatively encoded as ambisonics which in turn are decoded into binaural or loudspeaker formats).

[0092] Figure 2 An example system or apparatus 299 is shown which is suitable for implementing an embodiment of the present invention within a direct early reflection and late reverberation audio signal processing system 290. The system includes inputs such as an audio signal 200, reverberation parameters 202, and directional configuration parameters 212. The system or apparatus 299 is further configured to render a reverberant binaural signal 214 as an output, containing late reverberation perceived according to the acoustic parameters carried in the reverberation parameters 202.

[0093] In some embodiments, the reverberation parameters 202 include suitable (non-directional) parameters for configuring the reverberator 201. Suitable reverberation parameters 202 include, for example, the reverberation time T in a frequency band. 60 (k) (where k is the frequency band index), the reverberation-to-direct ratio RDR(k) in the frequency band, the pre-delay t pre , and the room size. As an alternative to RDR, the diffusion to source energy ratio (DSR) may also be used. The reverberation parameters 202 may be obtained, for example, from the bitstream or from the Listening Space Description Format (LSDF) input to the renderer.

[0094] The directional configuration parameters 212 may indicate the encoding direction for rendering the reverb by a suitable rendering scheme that creates the perception of an envelope diffuse reverb, such as ambisonic or amplitude panning rendering, or simply rendering directly to a surround (real or virtual) speaker setup. As an example, the following directional configuration may be used:

[0095] Azimuth θ ls(d) : 90, -90, 114, -60, 85, -130, 49, -67, 154, -48, 19, -162, 151, -9, 180 degrees.

[0096] Elevation angle φ ls (d) : 0, 0, 20, -6, -44, -21, 41, 52, -9, -55, -31, 21, 73, 20, 63 degrees.

[0097] The index d corresponds to the output channel of the reverberator 201 .

[0098] In some embodiments, the system 299 includes a reverberator parameter determiner 203 configured to obtain a reverberation parameter 202 (such as T 60 (k), ROR(k), and t pre ), and convert these parameters into appropriate reverberator parameters 204 for the reverberator 201 and transmit them to the first echo suppressor parameter determiner 205.

[0099] Furthermore, the system 299 includes a first echo suppressor parameter determiner 205 configured to receive the reverberator parameters 206 and generate first echo suppressor parameters 208, which are used to configure the first echo suppressor 499 of the reverberator to attenuate the acoustic sound from the first echo suppressor. Figure 4 The first (early) echo is the output of the reverberator's feedback network 450.

[0100] In some embodiments, the system 299 includes a reverberator 201 configured to receive an audio signal 200s in (t), where t is a sample (time) index. In addition, the reverberator 201 is configured to receive reverberator parameters 204 and first echo suppressor parameters 208. In some embodiments, the reverberator 201 is further configured to receive a directional configuration (and room dimensions) 212 that can be used to configure the reverberation.

[0101] The reverberator 201 is shown herein as using a feedback network (FN) implemented as a feedback delay network (FDN), but in other embodiments, the FN may be implemented using other suitable feedback structures.

[0102] In this example embodiment, the reverberator 201 has D = 15 output channels indexed by d = 1, 2, ..., d, ..., D. The resulting reverberant audio signal 210s rev (t, d) are mutually incoherent and they have acoustic characteristics according to the reverberator parameters 204 , and the first (early) echo is suppressed (eliminated or otherwise attenuated) according to the first echo suppressor parameters 208 .

[0103] In some embodiments, the system 299 includes a binaural renderer 209. The reverberant audio signal 210s rev (t, d) is forwarded to the binaural renderer 209, which also receives as further input the directional configuration parameters 212. In some embodiments, the binaural renderer 209 is configured to render the reverberated audio signal as a reverberated binaural signal 214s bin (t,i) (where i=1,2 are binaural channel indices), which can be reproduced using headphones, for example. According to the reverberation parameters 202, these signals are perceived as being ambient and surrounded by acoustic properties.

[0104] Furthermore, the direct, reflections and late reverberation audio signal processing system 290 may further include a reflections R (or early reflections ER) processor 251 configured to receive the audio signal 200 and generate a reflection audio signal 250 that is passed to an R binaural renderer 259 .

[0105] Furthermore, the direct, reflections and late reverberation audio signal processing system 290 may further include a reflections R (or early reflections ER) binaural renderer 259 configured to receive the reflections audio signal 250 and generate an early reflections binaural audio signal 254 which is output.

[0106] In addition, the direct, reflected and late reverberation audio signal processing system 290 may include a direct audio (D) processor 261, which is configured to receive the audio signal 200 and generate a direct audio signal 260 that is passed to the direct audio binaural renderer 269. The direct audio (D) processor 261 renders the portion of the sound that reaches the listener directly without reflections or reverberations. It applies distance gain attenuation (e.g., attenuation is proportional to 1 / distance, where distance is the distance from the sound source to the listener) and air absorption filtering (which is a distance-dependent low-pass filter that attenuates high frequencies).

[0107] Furthermore, the direct, reflected and late reverberation audio signal processing system 290 may also include a direct audio binaural renderer 269 configured to receive the direct (D) audio signal 260 and generate a direct audio binaural audio signal 264 that is output.

[0108] Figure 3 Shows Figure 2 An example flow chart of the system shown in FIG. 1 regarding the operation of a reverberator and a binaural renderer associated with the reverberator.

[0109] First, as indicated by 301 , an audio signal, directional configuration parameters, and reverberation parameters are obtained.

[0110] Furthermore, as indicated by 303, there is an operation of determining a reverberator configuration based on the reverberation parameter input and an optional directional configuration.

[0111] Additionally, as shown by 305, there is an operation of a first echo suppressor that determines control delay line and control line gain filter parameters to configure a reverberator based on reverberator parameter inputs.

[0112] Then, as shown by 307, a reverberated audio signal is generated based on the audio signal and the configured reverberator.

[0113] Furthermore, as shown by 309, the reverberated audio signal and the directional configuration parameters are used to render a reverberated binaural signal.

[0114] Then, as shown by 311, the reverberated binaural signal is output.

[0115] Figure 4 More detailed illustration of the reverberator 201 introduced in Figure 2 As described above, the reverberator 201 can be used to produce D uncorrelated outputs. In some embodiments, the reverberator 201 is configured to produce 15 outputs, which are subsequently encoded for rendering from different spatial directions defined by the directional configuration.

[0116] In some embodiments, the reverberator 201 includes a pre-delay line 401, which is configured to receive and delay an input audio signal. The reverberator 201 also includes a reverberation ratio control filter GEQ ratio 403, which is configured to receive the pre-delay line output. The reverberator 201 further includes a number D of feedback delay lines 451 and corresponding feedback delay line attenuation filters GEQ d 453. The signal output from GEQ d 453 is sent to the input of a feedback matrix A457. A D-signal combiner 454 (adder) adds the output of the feedback matrix A457 to the output of GEQ ratio 403 to be used as the input to each feedback delay line 451. The reverberator further includes a C-signal combiner 459, which combines the output of the feedback delay line attenuation filter GEQ d 453 with the output of a first echo suppressor 499 (discussed later). The output of the signal combiner 459 (or, in the case where C < D, the output of the delay line attenuation filter GEQ d 453 that is not routed to the signal combiner 459) is routed to a D-signal multiplier 461, which in turn outputs a reverberated audio signal 210.

[0117] Thus, in some embodiments, the reverberator 201 is configured to receive reverberator parameters, which include a pre-delay line The delay length m in the sample of 401 pre , Reverb Ratio Control Filter GEQ ratio 403 coefficients, D feedback delay lines The delay length m of each feedback delay line in 451 d , D feedback delay line attenuation filter GEQ d The coefficients of each feedback delay line attenuation filter in 453, and the coefficients of the feedback matrix A457. These reverberator parameters also include the output channel gain g d , which is used to configure the D signal multiplier 461.

[0118] In some embodiments, the attenuation filter GEQ d is a graphic equalizer (EQ) filter using M biquad IIR band filters. In the case of octave band filtering, M=10. Therefore, the reverberator parameters corresponding to each graphic EQ filter include the feedforward and feedback coefficients for the 10 biquad IIR filters, the gain for the biquad band filter, and the total gain. Feedback delay line 451 can also be called a loop delay line or recirculating delay line, and the feedback delay line attenuation filter GEQ d 453 may be referred to as a loop filter or a recirculating filter. In some embodiments, the coefficients of the feedback matrix A 457 are hard-coded in the software code rather than being provided as parameters.

[0119] Thus, the reverberator includes a plurality of recirculating delay lines associated with a feedback network (FN) 450. Feedback matrix A 457 is used to control the recirculating gain and routing within the network. In some embodiments, the feedback delay line attenuation filter GEQ d 453 can be implemented as a graphic EQ filter (which is implemented as a cascade of second-order partial IIR filters) and can help control the energy decay rate at different frequencies. In addition, the feedback delay line decay filter GEQ d The 453 are designed so that they attenuate the signal by the desired amount each time it passes through the FN, thereby achieving the desired reverberation time (RT60).

[0120] In addition, the reverberator further comprises a first echo suppressor 499. The first echo suppressor 499 comprises C control lines, each of which comprises a control line attenuation filter GEQ ctrl,c 455 series controlled delay line 452, such as Figure 4Depicted in medium gray. The echo generated by the control line of the first echo suppressor is designated as the control echo, and each control echo suppresses the first echo from the FN. The first echo is a signal that has only passed through the feedback delay line 451 and the feedback delay line attenuation filter 453 once and has not been recycled via the feedback matrix A457.

[0121] Control delay line length m c (where c = 1, 2,..., c,..., C) has the same length as the feedback delay line in the FN, and the corresponding output channels of the feedback delay line in the FN carry the first echo to be suppressed. The control line attenuation filter GEQ ctrl,c 455 is designed and configured such that when their output signals are combined with the output signals from the feedback delay line attenuation filter GEQ d 453, the first echo from the FN is suppressed to the desired level. Due to the control delay line 452 and the control line attenuation filter GEQ ctrl,c 455, the control echo from the first echo suppressor is temporally aligned with the corresponding first echo of the FN to be suppressed. The output signals from the control line are routed to the signal combiner 459, which is configured to combine them with the corresponding outputs from the FN.

[0122] In this example, the number C of control delay lines 452 is the same as the number D of the recycle delay lines in the FN. However, in some embodiments, C < D, or C does not need to be equal to D. For example, the first three control delay lines c = 1, 2, 3 can be used to suppress the first echo associated with the FN delay lines identified by the indices d = 1, 4, 6.

[0123] The control delay line 452 is a non - cyclic delay line (so there is no feedback loop), while the feedback delay line 451 is recycled through a feedback loop and its output passes through the feedback matrix A457.

[0124] The number of delay lines D (and control delay lines C) can be adjusted according to the quality requirements and the desired trade-off between reverberation quality and computational complexity. In an embodiment, an efficient implementation with D=15 delay lines is used. This makes it possible to define the coefficients of the feedback matrix A457 as proposed by Rocchesso in "Maximally Diffusive Yet Efficient Feedback Delay Networks for Artificial Reverberation" (IEEE Signal Processing Letters, Vol. 4, No. 9, September 1997) in terms of Galois sequences that facilitate efficient implementation.

[0125] Figure 5 An example reverberator parameter determiner 203 is shown. In some embodiments, the reverberator parameter determiner 203 is configured to obtain reverberation parameters (and room dimensions) 202 and generate appropriate reverberator parameters 204, such as: delay line length; delay line decay filter; reverberation ratio control filter; pre-delay line length; and feedback matrix coefficients.

[0126] For example, the reverberator parameter determiner 203 includes a delay line length determiner 501, which may be configured to determine a delay line length m of each of the D delay lines based on the virtual room size. d . Here, the dimensions of the shell are used, i.e., the bounding box that surrounds or is aligned with the walls of the physical or virtual room. For example, a shoebox-shaped room can be defined with dimensions xDim, yDim, zDim. If the room is not shoebox-shaped (or cuboid), a shoebox can be placed inside the room, and the dimensions of the placed shoebox can be used for the delay line length. Alternatively, the dimensions can be obtained as the three longest orthogonal dimensions in the non-shoebox-shaped room, or by other suitable methods. If the bounding box is provided as a grid, such dimensions can also be obtained from the grid. These dimensions can be further converted into modified dimensions of a virtual room or shell having the same volume as the input room or shell. For example, ratios 1, 1.3 and 1.9 can be used for the converted virtual room dimensions. When this method is executed in the renderer, the shell vertices are obtained from the bitstream, and the dimensions along each axis x, y, z can be calculated by the difference between the maximum and minimum values ​​of the vertices.

[0127] In some embodiments, the delay may be set proportional to the resonant frequency of a standing wave in a virtual or physical room.Thus, the delay may be set proportional to the resonant frequency of a standing wave in a virtual or physical room (acoustic environment).

[0128] Delay line length md Can be further set to coprime integers. This selection minimizes the coherent repetition in the impulse response of FN. The sieve of Sundaram algorithm can be used to find the prime number that reaches the maximum delay line length. Then, each delay line length can be mapped to the closest prime number in the obtained prime number set.

[0129] In some embodiments, the reverberator parameter determiner 203 further comprises a delay line attenuation filter parameter determiner 503 configured to determine an attenuation filter coefficient. The attenuation filter coefficients in the delay line may be configured so that a certain amount of attenuation occurs each time the signal recycles through the delay line, thereby achieving a desired RT60(k) time. This is done in a frequency-dependent manner to ensure an appropriate rate of attenuation of the signal energy at a specified frequency.

[0130] For frequency bin k, the expected attenuation per signal sample is calculated as attenuationPerSample(k)=-60 / (f s *RT60(k)), where f s is the sampling rate. The length is m d The attenuation (in decibels) of the delay line is attenuationDb(k)=m d *attenuationPerSample(k).

[0131] Furthermore, the attenuation filters are designed as cascaded graphic equalizer filters for each delay line, as in V. and J. Liski, “Accurate cascade graphic equalizer”, (IEEE Signal Processing Letters, Vol. 24, No. 2, pp. 176–180, February 2017). The design process outlined takes as input a set of command gains in octave bands. There are also methods for similar graphic EQ structures that can support three octave bands, thereby increasing the number of biquad filters to 31 and providing a better match for the detailed target response, such as in J. J. Liski and V. As indicated in “Third-Octave and Bark Graphic-Equalizer Designwith Symmetric Band Filters” (Applied Sciences, Vol. 10, No. 4, p. 1222, February 2020).

[0132] In some embodiments, the reverberator parameter determiner 203 includes a reverberation ratio control filter parameter determiner 505, which may be configured to determine reverberation ratio control filter parameters. The control filter parameters may be designed so that when the filter is applied to the input signal of the reverberator, the output reverberation will have a desired energy ratio defined by RDR(k). In some embodiments, the input to the design procedure may be a vector of RDR values ​​RDR(k) obtained by the reverberator parameter determiner 203.

[0133] GEQ ratio is configured to match the reverberator spectral energy to a target spectral energy. To this end, an estimate of the RDR of the reverberator output and a target RDR are determined. The RDR of the reverberator output may be obtained by rendering a unit pulse through the reverberator using the first reverberator parameters, measuring the energy of the reverberator output and the energy of the unit pulse, and calculating the ratio of these energies. Note that the first echo suppressor 499 is not used during this determination.

[0134] In some embodiments, the reverberation ratio control filter parameter determiner 505 is configured to create an input of a unit pulse followed by zeros of sufficient length to capture the reverberation tail. In practice, the length of the zero tail is adjusted to be equal to max(RT60(k)) plus t in samples. pre . Mono output signal s rev (t), which is a function of time t, can be obtained by summing the outputs of the feedback network 450 .

[0135] In s rev The long FFT (length is NFFT) is calculated on (t), and its absolute value can be obtained as:

[0136] FFA(kk)=abs(FFT(s rev (t))

[0137] Here, kk is the FFT bin index. In addition, the positive half-spectrum energy density is obtained as:

[0138] S(kk)=1 / NFFT*FFA(kk) 2

[0139] Here, the energy from the negative frequency index kk is added to the corresponding positive frequency index kk.

[0140] The energy per unit pulse can be obtained by calculation or analysis. It is denoted as S u (kk).

[0141] In some embodiments, the frequency band energy is determined by the positive half-spectrum energy density of the reverberator S(kk) and the unit pulse S uThe energy density of the positive half spectrum of (kk) is calculated by both. The band energy can be calculated as:

[0142]

[0143] Among them, b low and b high are the lowest and highest bin indices belonging to frequency band k, respectively. The band bin index can be obtained by comparing the frequency of the bin with the lower and upper bound frequencies of each frequency band.

[0144] Reproduction RDR of the reverberator output in frequency band k rev (k) is obtained as:

[0145] RDR rev (k) = S(k) / S u (k)

[0146] GEQ ratio The target linear magnitude response can be obtained as:

[0147] rdrFilterTargetResponse(k)=sqrt(RDR(k)) / sqrt(RDR rev (k))

[0148] Where RDR(k) is the linear target RDR value in frequency band k. Target response

[0149] controlGain(k)=20*log 10 (rdrFilterTargetResponse(k)) is the input to the graphic equalizer design routine as described in the previously cited V. and J. Liski, “Accurate cascade graphic equalizer,” (IEEE Signal Processing Letters, Vol. 24, No. 2, pp. 176–180, February 2017) and J. J. Liski and V. As discussed in “Third-Octave and Bark Graphic-Equalizer Design withSymmetric Band Filters” (Applied Sciences, Vol. 10, No. 4, p. 1222, February 2020).

[0150] The RDR filter target response (control gain for the graphic EQ design routine) can also be obtained directly in the logarithmic domain as:

[0151] controlGain(k)=10*log 10 (RDR(k))-10*log 10 (RDR rev (k))

[0152] In some embodiments, the reverberator parameter determiner 203 comprises a pre-delay line length determiner 507. The pre-delay line length determiner 507 is configured to determine the length of the pre-delay line based on the input pre-delay time t pre To determine and adjust the length m of the pre-delay line in samples pre The input pre-delay can be converted to samples and the length of the shortest FN delay line subtracted from it. This can be set to the length m pre This will result in the first echo from FN occurring at time t pre In some other embodiments, the diffusion portion of FN is set to occur at time t pre After that, start. In this case, m pre It may be lower, and may be set based on an estimate of the time at which the FDN output becomes diffuse and by setting the pre-delay line so that the desired time can be obtained.

[0153] Figure 6 An example operation of the reverberator parameter determiner 203 is shown.

[0154] As indicated by 601 , the first operation is to obtain reverberation parameters and room dimensions.

[0155] Furthermore, as shown by 603, the delay line length is determined based on the room size.

[0156] Next, as indicated by 605, based on the delay line length and RT60(k), delay line attenuation filter parameters are determined.

[0157] Then, as indicated by 607, based on the reverberation ratio, reverberation ratio control filter parameters are determined.

[0158] Thereafter, as indicated by 609, based on the pre-delay and the delay line length, the length of the pre-delay line is determined.

[0159] The first echo suppressor parameter determiner 205 is described in more detail below. In some embodiments, the first echo suppressor parameter determiner 205 may be configured to determine or design a suppression (or inversion) control line gain parameter GEQ ctrl,c ,as follows.

[0160] In a simple example of achieving complete suppression (cancellation) of the first echo from the FN, the GEQ ctrl,c =-GEQ d, where the indices c and d identify a unique pair of channels to be summed at the signal combiner 459. If the attenuation filter GEQ d is a scalar and complete suppression is desired, this control line gain setting can be used.

[0161] However, more generally, the control line gain may alternatively be set to a range of GEQ ctrl,c ∈[-GEQ d ,0], where the value -GEQ d is complete suppression (cancellation), a value of 0 is no suppression. This allows partial suppression (attenuation) of the first echo.

[0162] In some embodiments, the principle of suppression gain control may be frequency dependent. If the desired attenuation filter amplitude response of the control line at frequency bin / index k is a(k), then the corresponding control delay line attenuation filter may have a response -γa(k), where γ∈[0,1] controls the amount of suppression and where γ=1 corresponds to maximum suppression and γ=0 corresponds to no suppression. Control of the number of first echoes to be attenuated may be signaled from the early reflection rendering module (discussed later). In general, this will depend on the magnitude of the reflection echo rendering: the more reflection echoes rendered by the early reflection rendering module, the more first echoes are suppressed from the output of the FN. In general, the number of first echoes to be suppressed from the output of the FN may be equal to or less than the number of echoes to be rendered by the early reflection rendering module, which would otherwise overlap in time with the echoes produced by the FN.

[0163] In some embodiments, the number of echoes to be suppressed can be determined based on the time region containing early reflections in the impulse response. To this end, the reflected echoes can be synthesized by an early reflection synthesizer, and the maximum and minimum time delays of these early reflections can be determined. Then, the first echo output by the FN that falls within the same time delay range can be suppressed.

[0164] In some other embodiments, echoes of the FN interfering with one or more reflected echoes to be synthesized with the early reflection synthesizer may be suppressed.Interfering first echoes may for example be determined as those echoes falling within a predetermined time span around temporally adjacent reflected echoes.

[0165] Alternatively or additionally, the amount of echo to be attenuated may be signaled as a bitstream parameter.Such an embodiment may use an analysis of the echo structure of the FN and Early Reflections Renderer performed by the encoder device.

[0166] Alternatively or additionally, the time range of the echo to be attenuated in the FN output may be signaled as a bitstream parameter.

[0167] Figure 7In more detail, the schematic diagram shows Figure 2 The binaural renderer 209 shown in .

[0168] The input of the binaural renderer 209 is the directional reverberant audio signal 210 d s rev (t,d) and the directional configuration parameter θ ls (d),φ ls (d) 212. Figure 7 In the example shown in , the binaural renderer 209 is organized channel by channel, and there are several HRTF processors (one per channel) 701 d For example, for the first channel, the HRTF processor 7011 is configured to receive the directional reverberation audio signal 2101 (channel one) and the directional configuration 2121 associated with channel one. The second channel HRTF processor 7012 is configured to receive the directional reverberation audio signal 2102 (channel two) and the directional configuration 2122 associated with channel two. The D channel HRTF processor 7011 is also shown. D , which is configured to receive a directional reverberant audio signal 210 D (channel D) and directional configuration 212 associated with channel D D Each HRTF processor may include a HRTF filter pair. bin (m,i,d), where m is the time index of the filter coefficient, i=1,2 is the index of the binaural channel, and d is the reverberator output channel index. It should be understood that these reverberator output channels mentioned here and below can generally be directional audio signals.

[0169] dHRTF processor 701 d The operation is as follows. Use HRTF filter to h bin (m,i,d), can be calculated by the following formula for the directional reverberation audio signal 210 d Each channel of the reverberant binaural audio signal s bin (t,i,d)702 d :

[0170]

[0171] in, Denotes convolution (in some implementations, filtering can also be performed in the frequency domain instead of time domain convolution).

[0172] Furthermore, the reverberant binaural audio signal s bin (t,i,d)702 d The signal is transmitted to the binaural signal combiner 703 .

[0173] Then, the reverberant binaural audio signal s can be combined across channels d in the binaural signal combiner 703 by bin (t,i,d)702 d :

[0174]

[0175] Generate reverberant binaural signal s bin (t,i)214 is taken as output.

[0176] exist Figure 8 The operation of the binaural renderer is shown in the flowchart of FIG.

[0177] As indicated by 801 , the first operation is to obtain a reverberant audio signal and directional configuration parameters.

[0178] Then, as indicated by 803, the method may include generating a reverberant binaural audio signal on a per-channel basis by applying HRTF processing.

[0179] Furthermore, as indicated by 805 , the method may include combining the pair of reverberant binaural audio signals to generate a reverberant binaural audio signal.

[0180] Then, as shown by step 807 , the reverberant binaural audio signal may be output.

[0181] Fig. 9 An example early reflection renderer 990 suitable for use with embodiments as discussed herein is shown. These embodiments are intended to reduce interference between the first (early) echo output by the FN and the early reflections synthesized by the early reflection renderer. There are several methods to calculate or simulate early reflections. As an example, an image source method can be used, such as that discussed in JB Allen and D.A. Berkley, "Image method for efficiently simulating small-room acoustics" (Journal of the Acoustical Society of America, Vol. 65, pp. 943–950, April 1979) and in J. Borish, "Extension of the image model to arbitrary polyhedra" (Journal of the Acoustical Society of America, Vol. 75, No. 6, pp. 1827-1836, 1984).

[0182] exist Fig. 9In the example early reflection renderer 990 shown in , an early reflection parameter determiner 901 is configured to receive inputs of room geometry 906, listener position 900, source position 902, and absorption coefficient 904, and generate control parameters (such as delay 906, absorption 908, attenuation 910, and direction of arrival (DOA) 912) and pass them to the processor described below.

[0183] These parameters can be related to Fig.10 1006 , where a box or rectangular space with reflective surfaces 1000, 1002, 1004, 1006 is shown. Within the (virtual) acoustic space is a source 1020 and a listener 1010. The direction of the early reflections between the source 1020 and the listener 1010 is shown, where on the reflective surface between the source 1020 and the listener 1010 is a reflection and / or absorption point 1040. The mirror image of the source 1020 across the reflective surface 1006 can be used to establish the image source 1030. In turn, the line connecting the image source 1030 to the listener 1010 can be used to establish the reflection and / or absorption point 1040 and the direction of arrival (DOA) of the reflection relative to the listener. Based on the distance of the reflection path (the path from the image source to the listener, which is equal to the length of the path from the source 1020 to the listener 1010), the delay to be applied to the synthesized early reflections is obtained. Absorption α i (k) corresponds to the reflective surface 1006 from which the sound trajectory was reflected (reflection and / or absorption point 1040). The distance attenuation is set to be proportional to 1 / distance, where distance is equal to the length of the reflection path from the source to the listener. In addition, air absorption can also be included in the attenuation of the image source. The DOA of the early reflection is set based on the arrival angle from the reflection point to the listener.

[0184] In some embodiments, the input audio signal 920 is first fed to a delay line 903 , which buffers the audio signal samples and enables picking up of fragments of past samples of the audio signal 920 .

[0185] The early reflection signal obtainer 905 may receive the output of the delay line 903 and a delay 906 parameter. The early reflection signal obtainer is configured to obtain past signal samples based on the delay 906 to obtain a delayed signal.

[0186] In turn, the early reflection absorption processor 907 may filter the selected past signal samples to apply an equalizer filter to the absorption of the early reflections α i (k) The data is modeled to obtain a delayed and absorption filtered signal.

[0187] The early reflection decay processor 909 may then attenuate the delayed and absorption filtered signal by applying 1 / distance attenuation and optional air absorption to obtain a delayed and absorption filtered and attenuated signal.

[0188] Finally, the early reflection spatializer 911 can be configured to spatialize the delayed and absorption filtered and attenuated signal by HRTF filtering using left and right HRTF filters corresponding to the desired DOA of the early reflections to obtain a reverberant binaural signal 912 containing the synthesized early reflection portion. In some cases, the early reflection spatializer can be a binauralizer.

[0189] There are various methods to determine the image source parameters within the early reflection parameter determiner 901. In the image source method, the sound source position is mirrored relative to each reflecting surface of the room geometry to obtain the image source. Fig.10 In the example shown in , mirroring is performed with respect to the rightmost reflective surface 1006. Image source 1030 is located on a line perpendicular to reflective surface 1006, at the same distance from it as source 1020. The path from image source 1020 to listener 1010 indicates the distance traveled by the reflection. First order reflections reflect from a single wall, while higher order reflections reflect from more than one wall. Higher order reflections can be obtained by using a higher order image source that is mirrored by each reflective surface in turn.

[0190] In some cases, the output of the determiner is a list of image source locations, such as [r0, r1, ..., r I ,r 1,1 ,…,r 1,I ,…,r i,i,… ], where r i,i,… =[x i,i,… ,y i,i,… ,z i,i,… ] are the coordinates of the image source that has been reflected by the i-th subsequent surface in each order of reflection.

[0191] Fig.11 An example system is schematically shown in which an embodiment is implemented in an encoder device 1901 that performs part of the functionality; data is written into a bitstream 1921 and sent to a renderer device 1941, which decodes the bitstream, performs reverberator processing according to the embodiment and outputs audio for headphone listening.

[0192] Fig.11The encoder side 1901 of can be executed on the content creator computer and / or the network server computer. The output of the encoder is a bitstream 1921 that can be used for download or streaming. The decoder / renderer 1941 function runs on the end-user device, which can be a mobile device, personal computer, sound bar, tablet computer, car media system, home HiFi or theater system, head mounted display for AR or VR, smart watch, or any suitable system for audio consumption.

[0193] The encoder 1901 is configured to receive a virtual scene description 1900 and an audio signal 1904. The virtual scene description 1900 may be provided in an MPEG-I encoder input format (EIF) or in another suitable format. Generally, the virtual scene description contains an acoustically relevant description of the content of the virtual scene, and for example contains scene geometry as a grid or as voxels, acoustic materials, an acoustic environment with reverberation parameters, sound source locations, and other audio element related parameters (such as whether reverberation is to be rendered for the audio element). In some embodiments, the encoder 1901 includes a scene and reverberation payload encoder 1913 configured to generate reverberation parameters.

[0194] The encoder 1901 further comprises an MPEG-H 3D audio encoder 1914 configured to obtain the audio signal 1904 , MPEG-H encode it and pass it to a bitstream encoder 1915 .

[0195] Furthermore, in some embodiments, the encoder 1901 further includes a bitstream encoder 1915 configured to receive the output of the scene and reverberation payload encoder 1913 and the encoded audio signal from the MPEG-H encoder 1914 and generate a bitstream 1921 that can be passed to a bitstream decoder 1941. In some embodiments, the bitstream 1921 can be streamed to an end-user device or made available for download or storage.

[0196] In some embodiments, decoder 1941 includes a bitstream decoder 1951 configured to decode a bitstream.

[0197] The decoder 1941 may also include a scene payload decoder 1953 configured to obtain encoded reverberation parameters and decode these parameters in a relative or inverse operation to the reverberation payload encoder 1913 .

[0198] The reverberator parameter determiner 203 / 1952 is configured to receive the decoded reverberation parameters along with the room dimensions and spatial room impulse response (SRIR) 1950 information and generate the reverberator control parameters discussed herein. Note that in some embodiments, no SRIR is received and the reverberator parameters are obtained from the scene payload decoder 1953 instead.

[0199] The first echo suppressor parameter determiner 205 / 1955 is configured to receive the generated reverberator control parameters together with the room dimensions and optionally the SRIR 1950 and to generate control delay line and control line gain filter parameters in the manner described above.

[0200] In addition, the head pose generator 1957 receives information from the head mounted device 1970 or similar device and generates head pose information or parameters that can be passed to the binaural renderer 209 / 1959, the early reflection renderer 990 / 1962 and the direct sound binaural renderer 1963.

[0201] The decoder 1941 includes an MPEG-H 3D audio decoder 1954 configured to decode the audio signal and pass it to the reverberator 201 / 1961 and the direct sound processing 1965 .

[0202] Furthermore, the decoder 1941 also includes a reverberator 201 / 1961 configured to achieve appropriate reverberation of the audio signal from the MPEG-H 3D audio decoder 1954 .

[0203] The output of the reverberator 201 / 1961 is configured to output reverberated audio based on the reverberator parameters to the binaural renderer 209 / 1959 .

[0204] Furthermore, the decoder comprises an early reflections renderer 990 / 1962 configured to obtain the output of the MPEG-H 3D audio decoder 1954 and generate early reflections as described above and pass these early reflections to the early reflections binaural renderer 1999 .

[0205] The decoder further comprises a binaural renderer 209 / 1959 configured to generate a binaural reverberation audio signal from the output of the reverberator 201 / 1961 .

[0206] The decoder further comprises an Early Reflections (ER) binaural renderer 1999 configured to generate a binaural early reflections audio signal from the output of the Early Reflections renderer 990 / 1962.

[0207] In addition, the decoder / renderer 1941 includes a direct sound processor 1965, which is configured to receive the decoded audio signal and is configured to implement any direct sound processing (such as air absorption and distance gain reduction) and which can be passed to a direct sound binaural renderer 1963, which can generate a direct sound component using head orientation determination (from a suitable sensor), which is passed along with a reverberant component to a binaural signal combiner 1967. The binaural signal combiner 1967 is configured to combine the direct early reflections and the reverberant parts to generate a suitable output (e.g., for headphone reproduction).

[0208] Additionally, in some embodiments, the decoder includes a head orientation determiner that passes head orientation information to the head posture generator 1957 .

[0209] As an alternative to sending reverberation parameters from the encoder to the renderer, in some embodiments, reverberator parameters may be sent in the bitstream. Reverberator parameters refer to FDN parameters such as delay line length, decay filter, reverberation ratio control filter, etc.

[0210] In some embodiments, the assignment of the reverberator outputs to the speaker channels occurs during configuration of the reverberator. The assignment may be stored during configuration and provided to the reverberation signal router.

[0211] In some embodiments, the output is a multi-channel speaker setup (such as a 5.1 or 7.1+4 multi-channel speaker setup). In this case, the directional configuration θ may be obtained by using the orientation of the actual speakers as the directional configuration θ. ls (d), φ ls (d) To modify Figure 2 The process proposed in the embodiment of the present invention is performed, and the binaural renderer is omitted, and the reverberant audio signal is reproduced from the corresponding loudspeakers of the loudspeaker arrangement. Fig.11 , in case of loudspeaker output, instead of the binaural renderer 209 / 1959, there will be a loudspeaker renderer (or panner) which, in the simplest case, will just pass the loudspeaker signal to a loudspeaker signal combiner which will replace the binaural signal combiner 1967. Accordingly, the spatialized direct sound part and the early reflection part are realized with a panner (such as a VBAP) instead of a binaural processor.

[0212] about Fig.12 , shows an example electronic device that can be used as any device part of the system described above. The device can be any suitable electronic device or device. For example, in some embodiments, the device 2000 is a mobile device, a user device, a tablet computer, a computer, an audio playback device, etc. The device can be configured to implement an encoder or a renderer or any functional block as described above.

[0213] In some embodiments, the device 2000 includes at least one processor or central processing unit 2007. The processor 2007 may be configured to execute various program codes, such as the methods described herein.

[0214] In some embodiments, the device 2000 includes a memory 2011. In some embodiments, at least one processor 2007 is coupled to the memory 2011. The memory 2011 can be any suitable storage component. In some embodiments, the memory 2011 includes a program code portion for storing program codes that can be implemented on the processor 2007. In addition, in some embodiments, the memory 2011 can also include a storage data portion for storing data (e.g., data that has been processed or will be processed according to the embodiments described herein). When needed, the implemented program code stored in the program code portion and the data stored in the storage data portion can be retrieved by the processor 2007 via the memory-processor coupling.

[0215] In some embodiments, the device 2000 includes a user interface 2005. In some embodiments, the user interface 2005 may be coupled to the processor 2007. In some embodiments, the processor 2007 may control the operation of the user interface 2005 and receive input from the user interface 2005. In some embodiments, the user interface 2005 may enable a user to enter commands to the device 2000, for example, via a keypad. In some embodiments, the user interface 2005 may enable a user to obtain information from the device 2000. For example, the user interface 2005 may include a display configured to display information from the device 2000 to the user. In some embodiments, the user interface 2005 may include a touch screen or touch interface that enables information to be input into the device 2000 and to display information to a user of the device 2000. In some embodiments, the user interface 2005 may be a user interface for communication.

[0216] In some embodiments, the device 2000 includes an input / output port 2009. In some embodiments, the input / output port 2009 includes a transceiver. In such embodiments, the transceiver can be coupled to the processor 2007 and configured to communicate with other devices or electronic devices, for example, via a wireless communication network. In some embodiments, the transceiver or any suitable transceiver or transmitter and / or receiver components can be configured to communicate with other electronic devices or devices via wired or wired coupling.

[0217] The transceiver can communicate with other devices via any suitable known communication protocol. For example, in some embodiments, the transceiver can use a suitable Universal Mobile Telecommunications System (UMTS) protocol, a wireless local area network (WLAN) protocol such as IEEE 802.X, a suitable short-range radio frequency communication protocol such as Bluetooth, or an infrared data communication path (IRDA).

[0218] Input / output port 2009 may be configured to receive signals.

[0219] In some embodiments, device 2000 may be used as at least part of a renderer.Input / output port 2009 may be coupled to a headset (which may be a head tracking or non-tracking headset) or the like.

[0220] In general, various embodiments of the present invention may be implemented in hardware or dedicated circuits, software, logic, or any combination thereof. For example, some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device, but the present invention is not limited thereto. Although various aspects of the present invention may be illustrated and described as block diagrams, flow charts, or using some other graphical representation, it is well known that the boxes, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general hardware or controllers or other computing devices, or some combination thereof as non-limiting examples.

[0221] Embodiments of the present invention may be implemented by computer software executable by a data processor of a mobile device (such as in a processor entity), or by hardware, or by a combination of software and hardware. In addition, in this regard, it should be noted that any block of the logic flow as in the accompanying drawings may represent program steps, or interconnected logic circuits, blocks and functions, or a combination of program steps and logic circuits, blocks and functions. The software may be stored on a physical medium such as a memory chip or a memory block implemented within a processor, on a magnetic medium such as a hard disk or a floppy disk, and on an optical medium such as a DVD and its data variants, a CD.

[0222] The memory may be of any type suitable for the local technical environment and may be implemented using any suitable data storage technology, such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory, and removable memory. The data processor may be of any type suitable for the local technical environment and may include, as non-limiting examples, one or more of a general purpose computer, a special purpose computer, a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a gate level circuit based on a multi-core processor architecture, and a processor.

[0223] Embodiments of the present invention may be practiced in various components such as integrated circuit modules. The design of integrated circuits is generally a highly automated process. Complex and powerful software tools are available to convert a logic level design into a semiconductor circuit design ready to be etched and formed on a semiconductor substrate.

[0224] Programs, such as those offered by Synopsys, Inc. of Mountain View, Calif., and Cadence Design, of San Jose, Calif., use well-established design rules and a library of pre-stored design modules to automatically route conductors and position components on a semiconductor chip. Once the design of a semiconductor circuit is complete, the resulting design in a standardized electronic format (e.g., Opus, GDSII, etc.) can be transmitted to a semiconductor manufacturing facility or "fab" for fabrication.

[0225] The foregoing description has provided a complete and useful description of exemplary embodiments of the present invention by way of exemplary and non-limiting examples. However, various modifications and adaptations will become apparent to those skilled in the relevant art in view of the above description when read in conjunction with the accompanying drawings and the appended claims. However, all such and similar modifications of the teachings of the present invention will still fall within the scope of the present invention as defined by the appended claims.

Claims

1. A method for applying reverberation to at least one audio signal, the method include: obtaining the at least one audio signal; obtaining at least one reverberation parameter; controlling a digital reverberator based on the at least one reverberation parameter; providing at least one control line, the at least one control line comprising a delay line and a gain filter, the at least one control line using the at least one reverberation parameter to generate an output from the at least one control line; as well as At least one reverberant audio signal is generated based on at least one output of the at least one digital reverberator and the generated output from the at least one control line, wherein at least a portion of a late reverberation interfering with at least one reflected echo in the generated at least one reverberant audio signal is significantly attenuated.

2. The method according to claim 1, in, Obtaining the at least one reverberation parameter includes: obtaining at least one reverberation time.

3. The method according to claim 2, in, Controlling the digital reverberator based on the at least one reverberation parameter comprises applying at least one delay line to the at least one audio signal to generate at least two echoes.

4. The method according to claim 3, in, Providing the at least one control line comprises providing at least one control echo.

5. The method according to claim 4, in, Generating the at least one reverberant audio signal comprises at least one of the following: generating the at least one reverberant audio signal based on a combination of at least one audio signal comprising the at least two echoes from the at least one output of the digital reverberator and at least one audio signal comprising the at least one control echo from the at least one output of the at least one control line; as well as The at least one control echo is provided to at least partially suppress a first echo of the at least two echoes such that the at least one reverberant audio signal comprises reverberation that does not interfere with the at least one reflected echo.

6. The method according to claim 1, in, Obtaining the at least one reverberation parameter comprises obtaining the at least one reverberation parameter based on at least one size parameter associated with a virtual acoustic space in which the audio signal is rendered.

7. The method according to claim 1, in, Controlling the digital reverberator based on the at least one reverberation parameter comprises at least one of the following: controlling at least one feedback reduction filter; as well as A feedback matrix is ​​controlled, the feedback matrix being configured to feed back at least one output of the digital reverberator to at least one input of the digital reverberator.

8. The method according to claim 1, in, The at least one control line is a first number of control lines and the digital reverberator includes a second number of delay lines.

9. The method according to claim 8, in, The first number is less than or equal to the second number.

10. The method according to claim 1, in, At least the portion of the late reverberation is at least an early portion of the late reverberation.

11. An apparatus for applying reverberation to at least one audio signal, the apparatus comprising at least one processor and at least one memory storing instructions which, when executed by the at least one processor, cause the apparatus to at least: obtaining the at least one audio signal; obtaining at least one reverberation parameter; controlling a digital reverberator based on the at least one reverberation parameter; providing at least one control line, the at least one control line comprising a delay line and a gain filter, the at least one control line using the at least one reverberation parameter to generate an output from the at least one control line; as well as At least one reverberant audio signal is generated based on at least one output of the at least one digital reverberator and the generated output from the at least one control line, wherein at least a portion of a late reverberation interfering with at least one reflected echo in the generated at least one reverberant audio signal is significantly attenuated.

12. The device according to claim 11, in, The means caused to obtain the at least one reverberation parameter is further configured to obtain at least one reverberation time.

13. The device according to claim 12, in, The means caused to control the digital reverberator based on the at least one reverberation parameter is configured to apply the at least one delay line to the at least one audio signal to generate at least two echoes.

14. The device according to claim 13, in, The means caused to provide at least one control line is further configured to provide at least one control echo.

15. The device according to claim 14, in, The apparatus caused to generate the at least one reverberant audio signal is further configured to perform at least one of the following: generating the at least one reverberant audio signal based on a combination of at least one audio signal comprising the at least two echoes from the at least one output of the digital reverberator and at least one audio signal comprising the at least one control echo from the at least one output of the at least one control line; as well as The at least one control echo is provided to at least partially suppress a first echo of the at least two echoes such that the at least one reverberant audio signal comprises reverberation that does not interfere with the at least one reflected echo.

16. The device according to claim 11, in, The means caused to obtain the at least one reverberation parameter is based on at least one size parameter associated with a virtual acoustic space in which the audio signal is rendered.

17. The device according to claim 11, in, The means caused to control the digital reverberator based on the at least one reverberation parameter is configured to perform at least one of the following: controlling at least one feedback reduction filter; as well as A feedback matrix is ​​controlled, the feedback matrix being configured to feed back at least one output of the digital reverberator to at least one input of the digital reverberator.

18. The device according to claim 11, in, The at least one control line is a first number of control lines and the digital reverberator includes a second number of delay lines.

19. The device according to claim 18, in, The first number is less than or equal to the second number.

20. The device according to claim 11, in, At least the portion of the late reverberation is at least an early portion of the late reverberation.