Reverberation parameter modification
By providing acoustic modeling guidance parameters, the method addresses the lack of control in AR environments, enabling content creators to optimize reverberation rendering within specified tolerances, thus enhancing audio quality and listener experience.
Patent Information
- Application Number
- GB2025005080
- Authority / Receiving Office
- GB · GB
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2025-04-04
- Publication Date
- 2026-11-11
AI Technical Summary
Existing methods for controlling reverberation in augmented reality (AR) environments lack sufficient control over acoustic modeling, leading to suboptimal audio quality due to interference from varying physical room acoustics and differing content requirements.
An apparatus and method that provide acoustic modeling guidance parameters to modify reverberation parameters, allowing content creators to control the rendering of reverberation accurately within specified tolerances or reference values, ensuring optimal audio quality by aligning virtual acoustics with physical space characteristics.
Enables content creators to optimize virtual acoustics for AR environments, ensuring high-quality immersive audio experiences by aligning virtual reverberation with desired acoustic parameters, thereby improving audio quality and listener experience.
Smart Images

Figure 00000000_0000_ABST
Abstract
Description
FIELD
[0001] The present application relates to a method, apparatus, system and computer program for modifying reverberation parameters in reverberation rendering processing and in particular but not exclusively to method, apparatus, system and computer program for modifying reverberation parameters in reverberation rendering processing in virtual reality, augmented reality and extended reality applications. BACKGROUND
[0002] Reverberation refers to the persistence of sound in a space after an actual sound source has stopped. Different spaces are characterized by different reverberation characteristics. For conveying spatial impression of an environment, reproducing reverberation perceptually accurately is important. Room acoustics are often modelled with an individually synthesized early reflection portion and a statistical model for the diffuse late reverberation. Fig. 1 depicts an example of a synthesized room impulse response where the direct sound 101 is followed by discrete early reflections 103 (or reflection echoes) which have a direction of arrival (DOA) and diffuse late reverberation 107 which can be synthesized without any specific direction of arrival. The delay dl(t) 102 in Fig. 1 can be seen to denote the direct sound arrival delay from the source to the listener. Furthermore the delay d2(t) 104 can denote the delay from the source to the listener for one of the early reflections (in this case the first arriving reflection). Additionally the delay d3(t) 106 can denote the delay from the source the onset of the diffuse late reverberation.
[0003] One method of reproducing reverberation is to utilize a set of D loudspeakers (or virtual loudspeakers reproduced binaurally using a set of head-related transfer functions (HRTFs)). The loudspeakers are positioned around the listener somewhat evenly. Mutually incoherent reverberant signals are reproduced from these loudspeakers, producing a perception of surrounding diffuse reverberation.
[0004] The reverberation produced by the different loudspeakers has to be mutually incoherent. In a simple case the reverberations can be produced using the different channels of the same reverberator, where the output channels are uncorrelated but otherwise share the same acoustic characteristics such as reverberation time and level (specifically, the diffuse-to-direct ratio or reverberant-to-direct ratio or diffuse-to-total ratio or diffuse-to-source ratio or any other suitable parameter for representing reverberation energy or level). Such uncorrelated outputs sharing the same acoustic characteristics can be obtained, for example, from the output taps of a feedback delay network (FDN) reverberator with suitable tuning of the delay line lengths and mixing matrix, or from a reverberator based on using decaying uncorrelated noise sequences by using a different uncorrelated noise sequence in each channel. In this case, the different reverberant signals effectively have the same features, and the reverberation is typically perceived to be similar in all directions. SUMMARY
[0005] According to a first aspect, there is provided an apparatus for rendering at least one reverberant audio signal from at least one audio signal, the apparatus comprising at least one processor and at least one memory storing instructions that, when executed by the at least one processor, cause the system at least to perform: obtaining at least one acoustic parameter; obtaining at least one acoustic modelling parameter; modifying the at least one acoustic parameter based on the at least one acoustic modelling parameter; obtaining the at least one audio signal; configuring at least one reverberator based on the modified at least one acoustic parameter; and rendering at least one reverberant audio signal based on the configured at least one reverberator and the at least one audio signal.
[0006] The apparatus caused to perform obtaining at least one acoustic modelling parameter may be caused to perform obtaining at least one of: at least one acoustic modeling guidance parameter; at least one acoustic parameter tolerance; at least one reference value; and at least one instructional parameter.
[0007] The apparatus caused to perform modifying the at least one acoustic parameter based on the at least one acoustic modelling parameter may be caused to perform at least one of: adjusting the at least one acoustic parameter within a provided tolerance within the at least one acoustic modelling parameter; adjusting the at least one acoustic parameter closer to a reference value within the at least one acoustic modeling parameter; and adjusting the at least one acoustic parameter to substantially a value from the at least one acoustic modelling parameter.
[0008] The apparatus, caused to perform obtaining at least one acoustic modelling parameter may be further caused to perform obtaining at least two acoustic modeling parameter values, wherein each of the at least two acoustic modeling parameter values may be associated with an acoustic environment parameter.
[0009] The apparatus may be further caused to perform obtaining at least one acoustic environment parameter value, wherein the apparatus caused to perform modifying the at least one acoustic parameter based on the at least one acoustic modelling parameter may be further caused to perform: selecting one of the at least two acoustic modeling parameter values based on the obtained at least acoustic environment parameter value; and modifying the at least one acoustic parameter based on the selected one of the at least two acoustic modeling parameter values.
[0010] The apparatus caused to perform obtaining at least one acoustic modelling parameter may be further caused to perform obtaining at least two acoustic modeling parameter values, wherein each of the at least two acoustic modeling parameter values may be associated with a content or application type.
[0011] The apparatus may be further caused to perform obtaining at least one content or application type value, wherein the apparatus caused to perform modifying the at least one acoustic parameter based on the at least one acoustic modelling parameter may be further caused to perform: selecting one of the at least two acoustic modeling parameter values based on the obtained at least one content or application type value; and modifying the at least one acoustic parameter based on the selected one of the at least two acoustic modeling parameter values.
[0012] The at least one acoustic parameter may comprise at least one of: a decay parameter; a level parameter; and a delay parameter.
[0013] The at least one acoustic parameter may be configured to define an audio scene.
[0014] According to a second aspect there is provided a method for an apparatus for rendering at least one reverberant audio signal from at least one audio signal, the method comprising: obtaining at least one acoustic parameter; obtaining at least one acoustic modelling parameter; modifying the at least one acoustic parameter based on the at least one acoustic modelling parameter; obtaining the at least one audio signal; configuring at least one reverberator based on the modified at least one acoustic parameter; and rendering at least one reverberant audio signal based on the configured at least one reverberator and the at least one audio signal.
[0015] Obtaining at least one acoustic modelling parameter may comprise obtaining at least one of: at least one acoustic modeling guidance parameter; at least one acoustic parameter tolerance; at least one reference value; and at least one instructional parameter.
[0016] Modifying the at least one acoustic parameter based on the at least one acoustic modelling parameter may comprise at least one of: adjusting the at least one acoustic parameter within a provided tolerance within the at least one acoustic modelling parameter; adjusting the at least one acoustic parameter closer to a reference value within the at least one acoustic modeling parameter; and adjusting the at least one acoustic parameter to substantially a value from the at least one acoustic modelling parameter.
[0017] Obtaining at least one acoustic modelling parameter may comprise obtaining at least two acoustic modeling parameter values, wherein each of the at least two acoustic modeling parameter values may be associated with an acoustic environment parameter.
[0018] The method may further comprise obtaining at least one acoustic environment parameter value, wherein modifying the at least one acoustic parameter based on the at least one acoustic modelling parameter may further comprise: selecting one of the at least two acoustic modeling parameter values based on the obtained at least acoustic environment parameter value; and modifying the at least one acoustic parameter based on the selected one of the at least two acoustic modeling parameter values.
[0019] Obtaining at least one acoustic modelling parameter may further comprise obtaining at least two acoustic modeling parameter values, wherein each of the at least two acoustic modeling parameter values may be associated with a content or application type.
[0020] The method may further comprise obtaining at least one content or application type value, wherein modifying the at least one acoustic parameter based on the at least one acoustic modelling parameter may further comprise: selecting one of the at least two acoustic modeling parameter values based on the obtained at least one content or application type value; and modifying the at least one acoustic parameter based on the selected one of the at least two acoustic modeling parameter values.
[0021] The at least one acoustic parameter may comprise at least one of: a decay parameter; a level parameter; and a delay parameter.
[0022] The at least one acoustic parameter may be configured to define an audio scene.
[0023] According to a third aspect, there is provided an apparatus for an apparatus for rendering at least one reverberant audio signal from at least one audio signal, the apparatus comprising means configured to: obtain at least one acoustic parameter; obtain at least one acoustic modelling parameter; modify the at least one acoustic parameter based on the at least one acoustic modelling parameter; obtain the at least one audio signal; configure at least one reverberator based on the modified at least one acoustic parameter; and render at least one reverberant audio signal based on the configured at least one reverberator and the at least one audio signal.
[0024] The means configured to obtain at least one acoustic modelling parameter may be configured to obtain at least one of: at least one acoustic modeling guidance parameter; at least one acoustic parameter tolerance; at least one reference value; and at least one instructional parameter.
[0025] The means configured to modify the at least one acoustic parameter based on the at least one acoustic modelling parameter may be configured to at least one of: adjust the at least one acoustic parameter within a provided tolerance within the at least one acoustic modelling parameter; adjust the at least one acoustic parameter closer to a reference value within the at least one acoustic modeling parameter; and adjust the at least one acoustic parameter to substantially a value from the at least one acoustic modelling parameter.
[0026] The means configured to obtain at least one acoustic modelling parameter may be configured to obtain at least two acoustic modeling parameter values, wherein each of the at least two acoustic modeling parameter values may be associated with an acoustic environment parameter.
[0027] The means may further be configured to obtain at least one acoustic environment parameter value, wherein the means configured to modify the at least one acoustic parameter based on the at least one acoustic modelling parameter may further be configured to: select one of the at least two acoustic modeling parameter values based on the obtained at least acoustic environment parameter value; and modify the at least one acoustic parameter based on the selected one of the at least two acoustic modeling parameter values.
[0028] The means configured to obtain at least one acoustic modelling parameter may further be configured to obtain at least two acoustic modeling parameter values, wherein each of the at least two acoustic modeling parameter values may be associated with a content or application type.
[0029] The means may be further configured to obtain at least one content or application type value, wherein the means configured to modify the at least one acoustic parameter based on the at least one acoustic modelling parameter may further be configured to: select one of the at least two acoustic modeling parameter values based on the obtained at least one content or application type value; and modify the at least one acoustic parameter based on the selected one of the at least two acoustic modeling parameter values.
[0030] According to a fourth aspect, there is provided a computer readable medium comprising instructions which, when executed by an apparatus for rendering at least one reverberant audio signal from at least one audio signal, cause the apparatus to perform at least the following: obtaining at least one acoustic parameter; obtaining at least one acoustic modelling parameter; modifying the at least one acoustic parameter based on the at least one acoustic modelling parameter; obtaining the at least one audio signal; configuring at least one reverberator based on the modified at least one acoustic parameter; and rendering at least one reverberant audio signal based on the configured at least one reverberator and the at least one audio signal.
[0031] According to a fifth aspect, there is an apparatus for rendering at least one reverberant audio signal from at least one audio signal, the apparatus comprising: obtaining circuitry for obtaining at least one acoustic parameter; obtaining circuitry for obtaining at least one acoustic modelling parameter; modifying circuitry for modifying the at least one acoustic parameter based on the at least one acoustic modelling parameter; obtaining circuitry for obtaining the at least one audio signal; configuring circuitry for configuring at least one reverberator based on the modified at least one acoustic parameter; and rendering circuitry for rendering at least one reverberant audio signal based on the configured at least one reverberator and the at least one audio signal.
[0032] According to a sixth aspect, there is an apparatus for rendering at least one reverberant audio signal from at least one audio signal, the apparatus comprising: means for obtaining at least one acoustic parameter; obtaining at least one acoustic modelling parameter; means for modifying the at least one acoustic parameter based on the at least one acoustic modelling parameter; means for obtaining the at least one audio signal; means for configuring at least one reverberator based on the modified at least one acoustic parameter; and means for rendering at least one reverberant audio signal based on the configured at least one reverberator and the at least one audio signal.
[0033] According to a seventh aspect, there is provided a non-transitory computer readable medium comprising program instructions that, when executed by an apparatus, cause the apparatus to perform at least the method according to any of the preceding aspects.
[0034] According to an eighth aspect there is provided a computer program comprising instructions [or a computer readable medium comprising instructions] for causing an apparatus for rendering at least one reverberant audio signal from at least one audio signal to perform at least the following: obtaining at least one acoustic parameter; obtaining at least one acoustic modelling parameter; modifying the at least one acoustic parameter based on the at least one acoustic modelling parameter; obtaining the at least one audio signal; configuring at least one reverberator based on the modified at least one acoustic parameter; and rendering at least one reverberant audio signal based on the configured at least one reverberator and the at least one audio signal.
[0035] In the above, many different embodiments have been described. It should be appreciated that further embodiments may be provided by the combination of any two or more of the embodiments described above. DESCRIPTION OF FIGURES
[0036] Embodiments will now be described, by way of example only, with reference to the accompanying Figures in which:
[0037] Fig. 1 shows a model of room acoustics with regard to the room impulse response;
[0038] Fig.2a shows a further model of room acoustics with regard to the room impulse response with parameter modifications according to some embodiments;
[0039] Fig.2b shows graphs of the magnitude against frequency showing the effect of the modification to the parameters shown in Fig.2a;
[0040] Fig.3 shows schematically an example reverberator comprising an example feedback delay network (FDN) according to some embodiments;
[0041] Fig.4 shows schematically an example reverberator parameter determiner and modifier configuration according to some embodiments;
[0042] Fig.5 shows schematically an example reverberator parameter determiner as shown in Fig.4 according to some embodiments;
[0043] Fig.6 shows an example flow diagram of the operations of the example apparatus as shown in Fig.4 and 5 according to some embodiments;
[0044] Fig.7 shows schematically an example binaural renderer as shown in Fig.4 in further detail according to some embodiments;
[0045] Fig.8 shows an example flow diagram of the operations of the example reverberation parameter modifier as shown in Fig.4 according to some embodiments;
[0046] Fig.9 shows a further example flow diagram of the operations of the example reverberation parameter modifier as shown in Fig.4 according to some embodiments;
[0047] Fig. 10 shows schematically an example virtual audio scene rendering system comprising the reverberator modification apparatus as shown in Figs.4 and 5 according to some embodiments;
[0048] Fig. 11 shows schematically an example reflection processor and reflection binaural renderer in further detail according to some embodiments;
[0049] Fig. 12 shows an example system within which some embodiments can be implemented; and
[0050] Fig. 13 shows an example device suitable for implementing the apparatus shown in previous figures. DETAILED DESCRIPTION
[0051] The following relates to apparatus, methods and computer programs for rendering or reverberation and modification of reverberation parameters for listener acoustic environments (AE) such that the listener AE acoustic characteristics can be controlled via bitstream parameters provided by the content creator in a metaverse application (or other VR / AR application) where the listener AEs correspond to real physical environments with varying acoustic characteristics.
[0052] Rendering of reverberation based on the listener environment can be achieved in the MPEG-I immersive audio standard by providing listener space information to the renderer, which contains acoustic parameters (diffuse onset, reverberant-to-direct ratio, reverberation times, room geometry and acoustic absorptions) or a room impulse response. The renderer can then adjust its reverberation rendering based on the provided listener room characteristics. This is desirable since in AR applications it is desirable for the virtual content to blend in with the physical content such that a virtual rendering of a sound source sounds the same as if a physical source were present in the same location. This can be achieved if the virtual acoustics effects such as reverberation are rendered for the virtual sound sources by the MPEG-I immersive audio renderer according to acoustic parameters obtained from the physical listener environment.
[0053] However, there is no guarantee that the listener environment has acoustics which create a pleasing listening experience for the content. There can be room resonances or strong reflections or extra reverberation in the environment. Moreover, different content has different requirements for the room acoustics. There is currently defined a limited degree of control available for a content creator to control the rendering and reverberation operations in form of AR configuration parameters. Currently defined parameter fields, for example, include: revARConfigPresent: This field indicates whether reverberation control parameters are present for AR acoustic environments; revTheoreticalRDRtoActualRDRdB: This field provides an offset in decibels to be added to the log base 10 of an RDR value obtained from a statistical diffuse approximation based on the RT60 value when RDR values have not been provided via the LSI or when the bitstream flag revOverridelnputRDR or the LSI parameter overrideRDR is set; revInputRDRtoActualRDRdB: This field provides an offset in decibels to be added to the log base 10 of the RDR value obtained via the LSI; revOverridelnputRDR: This field indicates whether late reverberation shall derive RDR values using a statistical diffuse approximation based on the RT60 value instead of the RDR values obtained via the LSI acoustic parameters or RIR.
[0054] However, these fields only allow adjusting the level of reverberation (RDR) or controlling the means how the RDR is obtained (parameters or RIR or statistical model based on room geometry).
[0055] The following therefore provides apparatus and methods which enable improved control over the bitstream to guide the acoustic modeling for physical acoustic environments. If such guiding is lacking the content creator has no control over how the immersive reverberation sounds which can lead to suboptimal audio quality as the physical room acoustics can interfere with the content or enhance or attenuate certain aspects of content in an uncontrolled manner. An example interference could be one of adding too much reverberation or enhancing certain frequency ranges such that the resulting reverberated content sounds muffled or unclear.
[0056] The concept as discussed in the following embodiments in further detail is one for an apparatus configured to provide (or method of) rendering of reverberation for augmented reality (or mixed reality -- MR or extended reality - XR) in physical listener environments and where control is provided to the renderer or reverberator as to how accurately the renderer renders reverberation according to the physical space characteristics via acoustic modeling guidance parameters.
[0057] In some embodiments this is achieved by: (a) obtaining at least one acoustic modeling guidance parameter; (b) obtaining at least one acoustic parameter value; (c) based on the value of the acoustic modeling guidance parameter, modifying the acoustic parameter value to produce a modified acoustic parameter value; (d) configuring a reverberator using the modified acoustic parameter value; and (e) rendering immersive reverberation using the configured reverberator.
[0058] In some embodiments, the acoustic modeling guidance parameter comprises at least one acoustic parameter tolerance or reference value and an instructional parameter.
[0059] In some embodiments, the instructional parameter indicates whether the tolerance or reference is to be used strictly or as guidance.
[0060] For example, in the case of guidance related instructional parameters, the acoustic parameter is adjusted so that it is within the provided tolerance or is modified to be closer to the reference value.
[0061] In some embodiments a strict mode instructional parameter causes the renderer or reverberator to modify the parameter such that the modified acoustic parameter is closely within the tolerance or substantially matches the reference value.
[0062] In some embodiments there can be several acoustic parameter tolerance or reference values for different acoustic environments or different ranges of acoustic parameter values. For example, in some embodiments there can be sets of acoustic parameter tolerance or reference values for various ranges of room volumes. Thus for example a small room volume with a volume below a first threshold can have a first set of acoustic parameter tolerance or reference values, a medium room volume with a volume above the first threshold but below a second threshold can have a second set of acoustic parameter tolerance or reference values and a large room volume with a volume above the second threshold can have a third set of acoustic parameter tolerance or reference values.
[0063] In some embodiments there can be different acoustic parameter tolerance values for different content and usage scenarios. For example, there can be different values for music, movie, or an immersive communication use cases. This enables the rendered virtual acoustics that adapt to the listening situation.
[0064] The acoustic parameters can relate to the decay, level, and delay of late reverberation or the level, timing, and density of early reflections. The acoustic parameters can also relate to dimensions of the virtual room to be used for parameterizing late reverberation and / or early reflections.
[0065] When the embodiments such as described herein are employed, the content creator can at least partially control or impact the rendering of virtual acoustics for AR or other suitable content. The content creator can therefore decide to strictly force the use of predefined acoustic parameter reference values or just guide the system towards the reference or tolerances.
[0066] In such a manner the content creator, in such embodiments, is able to control the accuracy of the virtual acoustics system in following the real, physical room acoustics and thus improve on a poor-quality sounding output which is not optimized for the designed content and to which the content creator has not had access. The content creator furthermore in these embodiments can also decide whether to select a predetermined set of reference acoustic parameter values when he or she wants to remain in full control of how the content will sound like with acoustic effects.
[0067] This can be compared to examples where the embodiments are not implemented and where the content creator does not have control over how the AR content sounds with reverberation effects applied.
[0068] As different content benefits from different kind of reverberation effects, these embodiments enable the content creator (with knowledge as to how beneficial reverberation effects can be for their content) to control the application of the processing of virtual acoustic effects on the AR / VR / XR content.
[0069] Fig.2a shows an example situation that the following embodiments attempt to implement. Fig 2a shows the impulse response as shown in Fig. 1. The original impulse response shows reverberation parameters such as reverberation time RT6O1 and reverberant-to-direct-ratio (RDR) RDRi 205. Moreover, as shown in fig.2a is diffuse onset value diffuseOnseti 201 denoting the onset of the time when the reverberation should become diffuse. Furthermore, there can be room dimension parameters impacting the delays of the reflections (not shown in Fig 2a). . Furthermore, there can be parameters which characterize the level, timing, and density of the early reflection portion 209 before diffuseOnseti 201.
[0070] The parameters can be provided via listener space information for the renderer as acoustic parameters or as an impulse response from which the parameters can be measured. However, as described above there is no guarantee that content rendered with a reverberator configured with the parameters RT6O1 and RDRi 205 and diffuseOnseti 201 sounds good. This is because the reverberation will affect the quality of the immersive audio. For produced content, a content creator can decide what type of reverberation maximizes the quality of the content. Thus a consumer in a VR environment can have the content optimized as the VR room acoustics are fixed or designed by the content creator. For AR content or environments, this is not possible with current methods as the room acoustics (and their parameters) are obtained from the listener environment which can have widely varying acoustics.
[0071] For example, referring to Fig.2a, listener space acoustic parameters may be RT6O1 and RDRi 205 and diffuseOnseti 201 but the content creator may have designed the content with the acoustic parameters indicated by RT6O2 and RDR2 207 and diffuseOnset2 203 and which results in a more pleasing audio experience.
[0072] Therefore, in some embodiments the aim is to provide a mechanism for the content creator to signal or include one or more reference acoustic parameter sets or tolerances for the renderer to map the rendering acoustic parameters to a provided set of reference acoustic parameters or bring them inside provided tolerance values.
[0073] Thus in some embodiments, the audio experience for the listener could, with certain parameters result in a bad quality experience. For example, using the Fig.2a impulse functions the ‘sensed’ environment could result in reverberation rendering with RT6O1 and RDRi 205 and diffuseOnseti 201 which may not result in a pleasing immersive audio experience. In employing some embodiments, the parameters can be modified or set to RT6O2 and RDR2 207 and diffuseOnset2 203 which are reference parameters carried in the bitstream and with which a content creator has been able to pre-audition using the content. These reference parameters ensure good immersive audio quality as the content creator has been able to verify this by listening to the content using the parameters.
[0074] Fig.2b furthermore depicts another example of the effect of the invention. Two panes are depicted, 211 and 212. The pane 211 contains the spectrum of a reverberated signal 214 which is obtained from original RT60 values. The RT60 at 315Hz is equal to 3.3810s. The spectrum 213 is the spectrum of a reverberated signal where the RT60 value at the frequency 315Hz has been modified to 6.3810s to simulate a case where the acoustic measurement from a room impulse response, or an actual room characteristic, causes there to be a large RT60 value at a certain frequency, which is an example of a situation which the embodiments can attempt to overcome. The visible difference is audible in the audio signal as a ringing resonance which reduces immersion.
[0075] In the pane 212 there are again plotted the original spectrum 216 which is the same as 214, and a new spectrum 215 which is obtained after a tolerance value has been applied to the RT60 value at 315Hz. In the case of the spectrum 215 the RT60 value at 315Hz has been limited to equal the average RT60 across all the frequencies which is 3.5912s. In this case the spectrum 215 is much closer to 216 than 213 to 214.
[0076] In an augmented or virtual acoustics rendering system, reverberation is typically rendered as a combination of at least one direct and two echo-generating components. The echo-generating components can comprise a so-called early reflection echo synthesis component configured to generate a certain number of perceptually distinct echoes, and a late reverberation synthesis component configured to generate a stream of echoes which are relatively indistinct but adhere to the overall decay properties of a stochastic model for the late reverberation.
[0077] With respect to Fig. 10 is shown an augmented audio scene rendering system 1000 incorporating some embodiments. The augmented audio scene rendering system 1000 can comprise a direct sound processor 1003 configured to receive the audio signal 301 and generate direct sound processed audio signal 1006 which are passed to a direct sound binaural renderer 1069.
[0078] The direct sound processor 1003 renders the sound that directly reaches the listener without reflecting or reverberating (in other words the signals shown in Fig. 1, reference 101). The direct sound processor 1003 can therefore apply distance gain attenuation (e.g. attenuation proportional to 1 / r where r is the distance from the sound source to the listener) and air absorption filtering (which is a distance-dependent low-pass filter attenuating high frequencies).
[0079] The augmented audio scene rendering system 1000 can furthermore comprise a direct sound binaural renderer 1069 configured to receive the direct sound processed audio signals 1006 and generate binauralized direct audio signals 1064 which are output.
[0080] The augmented audio scene rendering system 1000 can comprise a reflection processor 1051. With respect to the reverberator and the reflection processor these are configured to generate audio signals associated with echoes within the system. For example, the reflection processor 1051 is configured to produce a discrete number of echoes which are specular with regard to features and geometry of the modelled room and are correspondingly precise and independently varied in their arrival direction, intensity, and coloration, as characterizes early reflections in a room impulse response (in other words the signals shown in Fig.1, reference 103). The echoes produced by the reflection processor are accordingly referred to as reflection echoes. The reflection processor is external to and running in parallel with the reverberator which is configured to produce late reverberation. Herein, the echoes produced by the reflection processor are referred to as reflection echoes.
[0081] There are several ways to calculate or simulate early reflections. As an example, the image source method can be used such as detailed in J. B. Allen and D. A. Berkley, “Image method for efficiently simulating small-room acoustic,” J. Acoust. Soc. Am., vol. 65, pp. 943-950, April 1979 and J. Borish. “Extension of the image model to arbitrary polyhedra.” The Journal of the Acoustical Society of America 75.6 (1984): 1827-1836. Reflection echo synthesis is typically spatially dynamic in that the levels and directions of arrival (DoAs) of the reflections depend on the source and listener positions and orientations.
[0082] Late reverberation, in contrast to early reflections, is not considered to be spatially dynamic in that the echoes produced in late reverberation synthesis do not vary with source-listener orientation. The reverberator 405 of Fig.4 and Fig. 10 is configured to produce, by way of a feedback network, a decaying stream of many echoes which increase in number (density) while decreasing in intensity (loudness) over time, as characterized by the decay properties of stochastic late reverberation (ref. Fig. 1, item 107) according to the characteristics of the virtual or physical room the reverberator relates to.
[0083] This can be achieved by the feedback architecture of the reverberator, in which an input audio signal passes through the network, splitting into numerous paths which form echoes which are separated in time by independent delay lines, all of which subsequently recirculate through the network, being further divided among the delay lines, subsequently splitting into more echoes with each recirculation through the network, and so on.
[0084] These echoes can be made to have only a loose correspondence to the geometry of the virtual room so do not represent geometrically precise (specular) reflections, nor do they convey the attenuation characteristics of specific reflection surfaces.
[0085] The proposed embodiments provide a mechanism where the physical accuracy of acoustic modeling achieved by the reflection processor 1051 of Fig. 10 and the reverberator 405 can be controlled via the bitstream.
[0086] The controlling involves preprocessing the acoustic parameters input to these modules to provide modified parameters, such that the modified parameters match reference parameters or are within certain tolerances provided in the bitstream acoustic modeling accuracy data.
[0087] As a result, control via the bitstream is enabled regards how the output content will sound like as the acoustic modeling will be done according to these control or guidance parameters and not always physically accurately (or rather based on the input.
[0088] Thus, as shown in Fig. 10 the augmented audio scene rendering system 1000 comprises a (late) reverberator and renderer400 which implements embodiments and is described in further detail hereafter. The (late) reverberator and renderer 400 is configured to receive or obtain as inputs: the audio signal 301; the directional configuration specification 412; the reverberation configuration specification 402; the acoustic modeling guidance parameters 404 and the listener position 1100 and generate the reverberant binaural audio signals 414, the binauralized direct signals 1064, and the reflection binaural audio signals 1054.
[0089] With respect to Fig.4 is shown the (late) reverberator and renderer 400 in isolation from the other reflection and direct processing and is suitable for rendering middle- and late-stage reverberation and implementing at least some embodiments.
[0090] In some embodiments, the reverberation configuration specification 402 comprises suitable parameters for configuring the reverberator 405. The reverberation configuration specification 402 in some embodiments comprises parameters such as the reverberation times RT6Q(k) in frequency bands (where k is the frequency band index), reverberant-to-direct ratio RDR(k), pre-delay or diffuse onset time tpre, and / or a virtual space geometry specification.
[0091] In some embodiments as an alternative to the RDR, the diffuse-to-source energy ratio (DSR) can be employed.
[0092] In some embodiments, the acoustic modeling guidance parameters 406 comprise parameters which guide how acoustic modeling is to be performed based on the reverberation configuration specification 402. The acoustic modeling guidance parameters 406 can in some embodiments comprise reference values for the reverberation times RT6Qref(Ji), reverberant-to-direct ratio RDRref(k'), pre-delay or diffuse onset time tpreref, and / or a virtual space geometry specification, e.g., as a reference box or refence mesh.
[0093] The acoustic modeling guidance parameters 406 can alternatively in some embodiments comprise tolerance values for the reverberation times RT6Qtol(k), reverberant-to-direct ratio RDRtoi(k), pre-delay or diffuse onset time tpreto!, and / or a virtual space geometry specification, e.g., as a tolerance values for the box or mesh dimensions.
[0094] The acoustic modeling guidance parameters 406, in some embodiments, can comprise absolute values for the reference or the tolerance values as described above or they can be expressed in percentage of the average, for example.
[0095] The acoustic modeling guidance parameters 406 furthermore in some embodiments can also be expressed in relation to a root mean value or a running average value, so that an individual acoustic parameter value at some frequency can maximally deviate a certain percentage from the average parameter value, or the RMS parameter value, or the median parameter value, or a running mean parameter value with an additional parameter indicating the running mean length.
[0096] Furthermore, in some embodiments the acoustic modeling guidance parameters 406 comprise an additional parameter or field, which can be designated relaxed, which indicates whether the acoustic modeling guidance parameters 406 are to be exactly matched (relaxed^alse or 0) or whether the parameters in the reverberation configuration specification 402 should be modified towards the parameters in the acoustic modeling guidance parameters 406 (relaxed=true or 1).
[0097] The (late) reverberator and renderer 400 or reverberator processing system in some embodiments comprises a reverberation parameter modifier 407 configured to receive the reverberation configuration specification 402 and the acoustic modeling guidance parameters 406. As an output the reverberation parameter modifier 407 produces modified reverberation configuration specification 404 containing modified values of the reverberation configuration specification based on the acoustic modeling guidance parameters 406.
[0098] In some embodiments, the (late) reverberator and renderer 400 (reverberator processing system) comprises a reverberator parameter determiner 403 configured to obtain the modified reverberation configuration specification 404 and the directional configuration specification 412.
[0099] The reverberator parameter determiner 403 can be configured to convert these specifications into suitable reverberator parameters 404 for the reverberator 405.
[0100] In some embodiments, the directional configuration specification 412 can indicate reproduction / rendering directions used to render the reverberation by a suitable rendering scheme that creates a perception of enveloping diffuse reverberation, such as ambisonics or amplitude panning rendering, or simply rendered directly to a surrounding (real or virtual) loudspeaker setup. As an example, the directional configuration may specify a spherical design such as a f-design, Lebedev grid, or other suitable (nearly) uniform spherical layout with D points representing rendering directions or positions (and thus the number of reverberator output channels). The immersive rendering configuration specification is used for reproducing the reverberant signals for the listener AE. The listener AE is the acoustic environment within which the current listener position is.
[0101] The (late) reverberator and renderer 400 (reverberator processing system) further in some embodiments comprises a reverberator 405 configured by the reverberator parameters 404 and generating the reverberant audio signals 410 from the audio signal 301 input.
[0102] The (late) reverberator and renderer 400 in some embodiments further comprises a binaural renderer 409 configured to render reverberant binaural signals 414 from the reverberant audio signals 410. The reverberant binaural signals 414 contain a late reverberation that is perceived according to the reverberant characteristics specified in the reverberation configuration specification 402 modified by the acoustic modeling guidance parameters 406 and directional characteristics specified in the directional configuration specification 412. The reverberation configuration specification 402 and acoustic modeling guidance parameters 406 and directional configuration specification 412 can, for example, be obtained from a bitstream or from a listener space information (LSI) input to the renderer. Since the reverberation rendering is performed with a reverberator configured with reverberator parameters 404 which were derived from modified reverberation configuration specification 404, which took as input the acoustic modeling guidance parameters 406, the apparatus and methods therefore achieve control via the acoustic modeling guidance parameters 406 (which can be transmitted in the bitstream) of how accurately the reverberation is configured based on the reverberation configuration specification 402 which can originate from a measurement of the listener space.
[0103] Fig.3 shows an example reverberator 405 implemented as a feedback reverberator 300. The reverberator 300 comprises a pre-delay line 350, configured to receive and delay the input audio signal 301. The reverberator 300 also comprises a reverberation ratio control filter GEQratio 303 which is configured to receive the pre-delay line output. It further comprises a number D of feedback delay lines z~md 351 and corresponding feedback delay line attenuation filters GEQd 353. The output signals from GEQd 353 are sent to inputs of a feedback matrix 4 357. The outputs of the feedback matrix 4 357 are sent to D signal combiners (adders) 354 which sum the outputs of the feedback matrix 357 with the output of GE Qratio 303 to be used as inputs to each of the feedback delay lines z~md 351. The reverberator thus comprises multiple recirculating delay lines associated with the feedback network 358. The feedback matrix 4 357 is used to control the recirculation gain and routing within the network.
[0104] In these embodiments a signal from the output taps 359 of the feedback loops combiners are routed to D signal multipliers 361 which in turn produce the reverberant audio output signals 310. Outputs of delay line attenuation filters GEQd 353 are routed directly to a corresponding signal multiplier among the multipliers 361.
[0105] In some embodiments (not shown herein) the reverberator 300 comprises a gain element which is configured to apply a source gain to the predelayed input signal to adjust the level of reverberation.
[0106] In some embodiments, the reverberator 300 is configured to receive reverberator parameters which configure and define the performance of the reverberator. The parameters comprise: a delay length mpre, in samples, for pre-delay line z~mvre 305; coefficients of a reverberation ratio control filter GEQratio 303; delay lengths md for each of D feedback delay lines z~md 351; coefficients for each of D feedback attenuation filters GEQd 353; and coefficients for the feedback matrix A 357. The reverberator parameters can furthermore comprise output channel gains gd which are used to configure D signal multipliers 361.
[0107] Thus the reverberator 300 parameters are related to the reverberation characteristics of the acoustic environment or room which is associated with the reverberator (or that the reverberator relates to).
[0108] In some embodiments where an audio scene with several rooms or acoustic environments (AE) there can be several reverberator 300 instances within the reverberators 405. In an embodiment one of the reverberators can be a physical environment reverberator corresponding to the listener physical environment and other reverberators can correspond to virtual AEs.
[0109] In some embodiments the filters GEQd 353 are implemented as graphic equalizer (EQ) filters using M biquad HR band filters and can facilitate controlling the energy decay rate at different frequencies. In the case of octave-band filtering, M = 10. The reverberator parameters corresponding to each graphic EQ filter comprise the feedforward and feedback coefficients for 10 biquad HR filters, the gains for biquad band filters, and the overall gain.
[0110] In some embodiments the coefficients of feedback matrix A 357 form a square lossless matrix and can be hardcoded in software code rather than provided as parameters. The size of the matrix (and the corresponding number of delay lines D) can be adjusted depending on the desired tradeoff between reverberation quality (e.g. modal density, temporal and spatial diffuseness, diffuse onset time) and computational complexity.
[0111] The feedback attenuation filters GEQd 353 can be designed such that they attenuate the signal by the desired amount with each pass through the feedback network (FN) 358 such that the desired or modified reverberation time (RT6Q) is achieved. The reverberation time depends on the reverberation time of the acoustic environment the reverberator models and any modification applied.
[0112] In an embodiment, an implementation with D = 15 delay lines is used, where internal feedback and output channels are indexed by d = 1,2, ...,d,...,D. This makes it possible to define the coefficients of the feedback matrix A 357 in terms of a Galois sequence facilitating efficient implementation, as described in proposed by Rocchesso in Maximally Diffusive Yet Efficient Feedback Delay Networks for Artificial Reverberation, IEEE Signal Processing Letters, Vol. 4. No. 9, Sep 1997. The feedback network (FN) 258 of the reverberator is a feedback delay network (FDN). Reverberator 200 can thus produce 15 nearly uncorrelated outputs which are subsequently encoded to different spatial directions defined by the directional configuration. The output signals are then reproduced using loudspeakers (or alternatively virtual loudspeakers that are convolved with HRTFs or yet alternatively encoded to ambisonics which is then decoded to a binaural or loudspeaker format) that are positioned in the corresponding spatial directions, and the levels of which are controlled with channel gain coefficients. The resulting reverberant audio signals have acoustical characteristics according to the reverberator parameters 404, namely a desired frequency-dependent rate of decay and level.
[0113] Fig.5 further shows an example reverberator parameter determiner 403 as shown in Fig.4 and 10 in further detail. In these embodiments the example reverberator parameter determiner 403 is configured to generate suitable reverberator parameters 404. In some embodiments the example reverberator parameter determiner 403 is configured to receive the directional configuration specification 312 and the modified reverberation configuration specification and from these generate reverberation parameters 404, such as: number of reverberator output channels; feedback delay line lengths; feedback attenuation filter coefficients; reverberation ratio control filter coefficients; and pre-delay line length.
[0114] For example, in some embodiments a reverberator parameter determiner 403 comprises a feedback delay line lengths determiner 501 configured to determine feedback delay line lengths, a feedback attenuation filter parameters determiner 503 configured to determine feedback attenuation filter coefficients, a reverberation ratio control filter parameter determiner 505 configured to determine reverberation ratio control filter coefficients and pre-delay line length determiner 509 configured to determine pre-delay line length.
[0115] In some embodiments the feedback delay line lengths determiner 501 is configured to determine the feedback delay lengths md based on a virtual space (Acoustic Environment) geometry specification. For example, a bounding box that encloses or is aligned with the walls of the physical or virtual room can be defined with dimensions xDim,yDim, zDim. Alternatively, the dimensions can be obtained as three longest orthogonal dimensions in the non-shoebox shaped room, or by a mesh if the bounding box is provided as a mesh, or by another suitable method. When the method is executed in a renderer then the enclosure vertices are obtained from the bitstream (for VR acoustic environments) or the LSDF / LSI (for an AR acoustic environment) and the dimensions can be calculated. Furthermore, the enclosure vertices are modified based on the acoustic modeling guidance parameters 406.
[0116] The feedback delay lengths md can in some embodiments be set proportionally to standing wave resonance frequencies in the virtual room or physical room (the acoustic environment), whose dimensions have been modified using the acoustic modeling guidance parameters 406.
[0117] In some embodiments the reverberator parameter determiner 403 further comprises a feedback attenuation filter parameter determiner 503 which is configured to determine attenuation filter coefficients for feedback attenuation filters GEQd 253. The filter coefficients can be configured so that the rate of attenuation produced by the recirculation through the delay lines results in the desired modified reverberation time ftT60m(fc). This is done in a frequency-dependent manner to ensure the appropriate rate of decay of signal energy at specified frequencies. For a frequency bin k, the desired attenuation per signal sample is ysamp(fc) = -60 / (fs * RT6Q(ky) dB, where fs is the sampling rate. The attenuation in decibels for a delay line pair of aggregate length md, where md = Tnda + mdb, is then Ygeq^ = md * ysamp(k),
[0118] which serves as a target command gain in the design procedure of cascade graphic equalizer filters as described in V. Valimaki and J. Liski, “Accurate cascade graphic equalizer," IEEE Signal Process. Lett., vol. 24, no. 2, pp. 176—180, Feb. 2017 , to produce the attenuation filter coefficients for GEQd. The cited design procedure operates in octave bands, although methods for similar graphic EQ structures can support third octave bands, increasing the number of biquad filters to 31 and providing a better match for detailed target responses, such as detailed in J. Ramb, J. Liski, and V. Valimaki, “Third-Octave and Bark Graphic-Equalizer Design with Symmetric Band Filters,” Applied Sciences, vol. 10, no. 4, p. 1222, Feb. 2020.
[0119] The feedback delay line lengths determiner 501 may set the delay line lengths sum md to be made to be mutually prime integers. This choice minimizes coherent repetition in the impulse response of the FN. The sieve of the Sundaram algorithm can be used to find the prime numbers up to the maximum delay line length and each delay line length can then be mapped to the closest prime number in the obtained set of prime numbers.
[0120] In some embodiments, the pre-delay line length determiner 506 is configured to determine a predelay length mpre in samples based on tprem that denotes an onset timing of the diffuse state of the reverberator after modification using acoustic modeling guidance parameters 406, in which case mpre can be set such that the diffuse onset timing of the FN matches a desired tprejm. The diffuse onset timing of the FN can be estimated by any mixing time or diffuseness estimator or predicted from analytic methods using the modified virtual space geometry specification provided in the modified reverberation configuration specification 404.
[0121] In some embodiments, the reverberation ratio control filter parameter determiner 509 can be configured such that, when the filter GEQratio 203 is applied to the input signal 301, the resultant reverberation will have the desired modified energy ratio defined by the RDRm(k). The input to the design procedure can in some embodiments be the vector of modified reverberant-to-direct (RDR) energy ratio values RDRm(k) obtained by the modified reverberation configuration specification 404. The generated coefficients of GEQratio is designed to match the reverberator spectrum energy to the target spectrum energy. To do this, an estimate of the RDR of the reverberator output is determined by the following procedure.
[0122] The RDR of the reverberator output can be obtained by first rendering a unit impulse through the reverberator that has been configured with the parameters produced by the delay line lengths determiner 501, feedback attenuation filter parameters determiner 503, and tpre set to 0. The input to the reverberator is a buffer of zeros of a sufficient length to capture the reverberation tail, such as the maximum ^60,m(^) among all frequency bands, with a unit impulse written to the head of the buffer. Once rendered, the energy of the reverberator output is measured, along with that of the unit impulse, and the ratio of these energies is calculated. The procedure for measuring signal energy is detailed in the following.
[0123] The monophonic output signal srev(t), which is a function of time t, can be obtained by summation of the outputs of the feedback network 250. A FFT of length / VFFT is calculated over srev(t) and its magnitude spectrum can be obtained as H(kb) = abs(FFT(srev(t)).
[0124] Here, kb are the FFT bin indices. The positive half spectral energy density is W2, ivfft
[0125] where the energy from the negative frequency indices kb is added into the corresponding positive frequency indices kb. The energy of a unit impulse can be calculated or obtained analytically and is denoted as Sunit(kh).
[0126] In some embodiments the energy of each band at index k are calculated as the positive half spectral energy density of the reverberator Srev(kb) and the positive half spectral energy density of the unit impulse Sunit(fcb). Band energies can be calculated as S(k) = ) S(kb), kb~blow
[0127] where biow and bhigh are the lowest and highest bin indices belonging to band k, respectively. The band bin indices can be obtained by comparing the frequencies of the bins to the lower and upper frequencies of each band.
[0128] The reproduced RDR of the reverberator is obtained as RDRrev(k~) = Srev(fc) / Sunit(fc)
[0129] The target linear magnitude response for GEQratio can be obtained as 5GEQrati0 = sqrt( / ?DK(k)) / sqrt( / ?^
[0130] where RDR^k) is the target linear RDR value from modified reverberation configuration specification 404. The target response control gain is then ^GEQratio GO = 20 * IOgiQ)•
[0131] The RDR target response control gain can also be obtained directly in the logarithmic domain as TgecwJ / 0 = 10 * log10CRD / ?m(fc)) - 10 * \ogjRDRrev(ky).
[0132] yG£(?ratio (k) is then provided to the graphic equalizer design routine, previously cited in the description of feedback attenuation filter parameter determiner 403 to produce filter coefficients for GEQratio.
[0133] Fig.6 shows an example flow diagram of the operations of the system shown in Fig.4 with respect to the reverberators and the associated binaural renderer.
[0134] First, the audio signal 301, reverberation configuration specification 402, directional configuration specification 412 and acoustic modeling guidance parameters 406 are obtained as shown by 601.
[0135] Then, the modified reverberation configuration specification is determined based on the reverberation configuration specification 402 and the acoustic modeling guidance parameters 406 as shown by 603.
[0136] Then, the reverberator parameters 404 are determined from the modified reverberation configuration specification 404 and directional configuration specification 412 inputs as shown by 605.
[0137] Then, the reverberator 405 is configured using reverberator parameters 404 as shown by 607.
[0138] Then, the binaural renderer 409 is configured using the directional configuration specification 412 and listener position 1100 as shown by 609.
[0139] Then, reverberant audio signals 310 are generated by processing the audio signal 301 with the configured reverberator 405 as shown by 611.
[0140] Then, reverberant binaural signals 414 are rendered by processing the reverberated audio signals 410 with the binaural renderer 409 as shown by 613.
[0141] Then, reverberant binaural signals 414 are output as shown by 615.
[0142] Fig.7 shows an example binaural renderer 409 as shown in Fig.4 and 10 in further detail. The input to the binaural renderer 409 is the reverberant audio signals 410d srev(t, d) and the directional configuration specification 412 indicating rendering directions for each combined reverberant audio signal. In the example shown in Fig.7 the binaural renderer 409 is organized on a channel-by-channel basis and there is one HRTF processor 701 d per reverberant audio channel. For example, a first channel a HRTF processor 7011 is configured to receive the reverberant audio signal 410i (channel one) and the directional configuration 412i associated with channel one. A second channel a HRTF processor 7012 is configured to receive the reverberant audio signal 4102 (channel two) and the directional configuration 4122 associated with channel two. Also shown is a Dth channel HRTF processor 701 d configured to receive the combined reverberant audio signal 41 Od (channel D) and the directional configuration 41 2d associated with channel D.
[0143] Each of the HRTF processors can comprise an HRTF filter pair hbin(m, i, d), where m is the time index of the filter coefficients, i = 1,2 is the index of the binaural channel, and d is the reverberator output channel index.
[0144] The operation of the dth HRTF processor 701 d is as follows. Using the HRTF filter pairs hbin(m, i, d), reverberant binaural audio signals sbin(t, i,d~) 702d can be determined for each channel of the reverberant audio signals 410d by ^bin^: h-bin^fri, I, d) ® Scor[lb(<t, d)
[0145] where ® denotes convolution (the filtering may also be performed in the frequency domain in some implementations instead of time-domain convolution) and scomb(t, d) is the combined reverberant signal.
[0146] The reverberant binaural audio signals sb-m(t,i,d') 702d can then be passed to a binaural signal combiner 703.
[0147] The reverberant binaural audio signals sbin(t, i, d) 702d can then be combined across channels d in the binaural signal combiner 703 by ^bin^' 0 ^bin^> d
[0148] yielding the reverberant binaural signals sbin(t, i) 414 which is the output.
[0149] Fig.8 shows a flow diagram of the operations of the example reverberation parameter modifier 407 according to some embodiments.
[0150] First, there is obtaining reverberation configuration specification and acoustic modelling guidance parameters as shown in Fig.8 by 801.
[0151] Then, there is determining the value of the relaxed parameter in the acoustic modelling guidance parameters as shown in Fig.8 by 803.
[0152] A check of whether the relaxed parameter is true (yes or no) is shown in Fig.8 by 804.
[0153] If re / axecMrue, then there is moving values of acoustic parameters in the reverberation configuration specification, which have a corresponding reference value in the acoustic modelling guidance parameters, towards the reference value as shown in Fig.8 by 805.
[0154] Then, there is moving values of acoustic parameters in the reverberation configuration specification, which have a corresponding tolerance value in the acoustic modelling guidance parameters, towards the value within tolerance as shown in Fig.8 by 807.
[0155] Then, modified reverberation configuration specification is output as shown by 809.
[0156] If, however, relaxed = false, then after the check step 804 there is setting values of acoustic parameters in the reverberation configuration specification, which have a corresponding reference value in the acoustic modelling guidance parameters, equal to the reference value as shown in Fig.8 by 811.
[0157] Then, there is setting values of acoustic parameters in the reverberation configuration specification, which have a corresponding tolerance value in the acoustic modelling guidance parameters, such that the value is within tolerance as shown in Fig.8 by 813.
[0158] Then, modified reverberation configuration specification is output as shown in Fig.8 by 809.
[0159] An example related to the steps of Fig.8 is as follows. Reverberation configuration specification can indicate RT60 equal to 3.5 s. Acoustic modeling guidance parameter can indicate tolerance of RT60 equal to 2 + / -1 s. If relaxed = false, RT60 is set such that it is within the tolerance, in this case 2+1 s = 3s. If relaxed = true, RT60 can be modified less, for example to 3.2s so that it is closer to the tolerance but not necessarily within.
[0160] Thus, setting relaxed=true can be considered a guidance mode where the input reverberation characteristics are moved closer to the parameter values in the bitstream acoustic modeling guidance parameters. This way the acoustic modeling guidance parameters provide reference values of acoustic parameters or ranges of values with the tolerances which have been tested to work well with the content.
[0161] Setting re / axed=false will cause the system to use the values from the acoustic modeling guidance data. This will enable signaling certain preset values for certain content, as having strictly dictated acoustics can be important for certain content.
[0162] Fig.9 depicts a further example of operations of the reverberation parameter modifier 407. In this case, the system 400 of Fig.4 can obtain as additional input content type indication such as “music”, “movie”, or “speech” or the like.
[0163] First, there is obtaining reverberation configuration specification and acoustic modelling guidance parameters as shown in Fig.9 by 901. In some embodiments, new reverberation configuration specification can be obtained if the environment (room) changes. In some embodiments, new content can arrive having a different type. Also, new acoustic modeling guidance parameters can be obtained. The system then needs to update its behavior to react accordingly.
[0164] Then, there is checking whether there are several alternative acoustic modeling guidance parameters available as shown in Fig.8 by 903.
[0165] If there are only one set of acoustic modeling guidance parameters available, there is then performing the steps 803 through 813 of Fig.8 as shown in Fig.9 by 905.
[0166] If there are several sets of acoustic modeling guidance parameters available, there is then checking whether there are there acoustic modeling guidance parameters available for this content type as shown in Fig.9 by 907.
[0167] If there are acoustic modeling guidance parameters available for this content type, there is then performing the steps 803 through 813 of Fig.8 using these contenttype specific acoustic modeling guidance parameters as shown in Fig.9 by 909. This enables the method to adapt the acoustic modeling guidance depending on content type, such as “speech”.
[0168] If instead there are no acoustic modeling guidance parameters for this type, then there is determining whether there are acoustic modeling guidance parameters available for a room having a size close to the room in reverberation configuration specification as shown in Fig.9 by 911.
[0169] If there are acoustic modeling guidance parameters available for a room having size close to the room in the reverberation configuration specification, then there is performing the steps 803 through 813 of Fig.8 using acoustic modeling guidance parameters closest to room specification as shown in Fig.9 by 913. This enables different acoustic modeling guidance parameters to be included for different room sizes (room volumes) and the method can select the most suitable acoustic modeling guidance parameters for this room size. The benefit of this is that a content creator can during content production create suitable “preset” sets of acoustic modeling guidance parameters which lead to good audio quality in different size rooms which will then be used to make these “presets”.
[0170] If there are no acoustic modeling guidance parameters available for a room having size close to the room in the reverberation configuration specification, then there is performing the steps 803 through 813 of Fig.8 using acoustic modeling guidance parameters which result in least amount of reverberation as shown in Fig.9 by 915. This is an example implementation and using the acoustic modeling guidance parameters which leads into least amount of reverberation (smallest level and / or RT60 time) is a safe choice compared to selecting a set of acoustic modeling guidance parameters which lead to higher amount of reverberation.
[0171] Fig. 11 shows an example reflection processor 1051 and the associated reflection binaural renderer 1059 suitable for using along with the embodiments as discussed herein. There are several ways to calculate or simulate early reflections. As an example, the image source method can be used such as detailed in J. B. Allen and D. A. Berkley, “Image method for efficiently simulating small-room acoustic,” J. Acoust. Soc. Am., vol. 65, pp. 943-950, April 1979 and J. Borish. “Extension of the image model to arbitrary polyhedra.” The Journal of the Acoustical Society of America 75.6 (1984): 1827-1836. In the example early reflection renderer shown in Fig.11, a reflection parameter determiner 1101 is configured to receive the inputs of room geometry 1106, listener position 1100, source position 1102, absorption coefficients 1104 and modified reverberation configuration specification 404 and generate control parameters such as delay 1106, absorption 908, attenuation 1110 and direction of arrival (DoA) 1112 and pass these to the processors described hereafter.
[0172] The role of the modified reverberation configuration specification 404 is to indicate modified values for various early reflection parameters, such as early reflection density, delay, early decay time, clarity, definition, or center time. Early reflection density can refer to the density or amount of early reflections per unit of time. Early reflection delay refers to the delay of early reflections from direct sound. Early decay time refers to the time it takes for the reverberation level to decay the first 10dB. Clarity can refer to the ratio of the early sound energy (between 0 and 50 ms) and the late sound energy (that arrives later than 50 ms). Definition can be obtained as the ratio between the sound energy between 0 and 50 ms and the total energy of the impulse response. Since modified reverberation configuration specification 404 is fed into the early reflection parameter determiner 1101 then early reflection portion of the reverberation can also be rendered according to the acoustic modeling guidance parameters 406 of Fig.4.
[0173] In some embodiments the input audio signal (source output 432) is first fed into a delay line 1103 which buffers audio signal samples and enables picking segments of past samples of the audio signal.
[0174] The reflection signal obtainer 1105 can receive the output of the delay line 1103 and the delay 1106 parameter. The reflection signal obtainer is configured to obtain a past signal sample based on the delay 1106 to obtain a delayed signal.
[0175] A reflection absorption processor 1107 then can filter the selected past signal sample to apply an equalizer filter to model the frequency-dependent absorption data for the reflection to obtain delayed and absorption-filtered signal.
[0176] A reflection attenuation processor 1109 can then attenuate the delayed and absorption-filtered signal by applying a 1 / r attenuation and optionally air absorption to obtain delayed and absorption-filtered and attenuated signal.
[0177] Finally, a reflection binaural renderer 1059 can be configured to spatialize the delayed and absorption-filtered and attenuated signal by HRTF filtering with a left and right HRTF filter corresponding to the desired DoA for this reflection to obtain a reflection binaural signal 1054 containing the synthesized reflection portion. In some situations, the reflection spatializer can be the binauralizer of Fig.7.
[0178] Fig. 12 shows schematically an example system where the embodiments are implemented in an encoder device 1201 which performs part of the functionality; writes data into a bitstream 1221 and transmits that for a renderer device 1241, which decodes the bitstream, performs reverberator processing according to the embodiments and outputs audio for headphone listening.
[0179] The encoder side 1201 of Fig. 12 can be performed on content creator computers and / or network server computers. The output of the encoder is the bitstream 1221 which is made available for downloading or streaming. The decoder / renderer 1241 functionality runs on an end-user-device, which can be a mobile device, personal computer, sound bar, tablet computer, car media system, home HiFi or theatre system, head mounted display for AR or VR, smart watch, or any suitable system for audio consumption.
[0180] The encoder 1201 is configured to receive the virtual scene description 1200 and the spatial audio signals 1904. The virtual scene description 1200 can be provided in the MPEG-I encoder input format (EIF) or in another suitable format. Generally, the virtual scene description contains an acoustically relevant description of the contents of the virtual scene, and contains, for example, the scene geometry as a mesh or as voxels, acoustic materials, acoustic environments with reverberation parameters, positions of spatial audio sound sources, and other audio element related parameters such as whether reverberation is to be rendered for an audio element or not. The encoder 1201 in some embodiments comprises a scene and reverberation payload encoder 1213 configured to generate reverberation parameters and acoustic modeling guidance parameters.
[0181] The encoder 1201 further comprises a MPEG-H 3D audio encoder 1214 configured to obtain the spatial audio signals 1904 and MPEG-H encode them and pass them to a bitstream encoder 1215.
[0182] The encoder 1201 furthermore in some embodiments comprises a bitstream encoder 1215 which is configured to receive the output of the scene and reverberation payload encoder 1213 and the encoded audio signals from the MPEG-H encoder 1214 and generate the bitstream 1221 which can be passed to the bitstream decoder 1241. The bitstream 1221 in some embodiments can be streamed to end-user devices or made available for download or stored.
[0183] The decoder 1241 in some embodiments comprises a bitstream decoder 1251 configured to decode the bitstream.
[0184] The decoder 1241 further can comprise a scene payload decoder 1253 configured to obtain the encoded scene and reverberation parameters and acoustic modeling guidance parameters and decode these in an opposite or inverse operation to the scene encoder 1213.
[0185] The reverberator parameter modifier 407 / 1256 is configured to receive the decoded the decoded reverberation configuration specification and room dimensions and reverberation parameters and acoustic modeling guidance parameters and generated the modified reverberation configuration specification discussed herein.
[0186] The reverberator parameter determiner 403 / 1242 is configured to receive the modified reverberation configuration specification and room dimensions information and generate the reverberator parameters discussed herein.
[0187] 'Furthermore, the head pose generator 1257 receives information from a head mounted device 1270 or similar and generates head pose information or parameters which can be passed to the binaural renderer 409 / 1259, the reflection processor 1051 / 1262, reflection binaural renderer 1059 / 1299 and the direct sound audio binaural renderer 1069 / 1263.
[0188] The decoder 1241 comprise MPEG-H 3D audio decoder 1254 which is configured to decode the audio signals and pass them to the reverberator and reflection processor and direct sound binaural renderer 1069 / 1263.
[0189] The decoder 1241 furthermore comprises reverberator 405 / 1261 configured to implement a suitable reverberation of the audio signals from the MPEG-H 3D decoder.
[0190] The reverberator 405 / 1261 is configured to output reverberated audio based on the reverberator parameters to a binaural renderer 409 / 1259.
[0191] The decoder further comprises a binaural renderer 409 / 1259 configured to generate binaural reverberant audio signals from the reverberant signals output of the reverberator 405 / 1261.
[0192] The decoder furthermore comprises reflection processor 1051 / 1262 configured to obtain the audio signal and generate early reflections as described above and pass these to a reflection binaural renderer 1059 / 1299.
[0193] The decoder further comprises an early reflection (ER) binaural renderer 1059 / 1299 configured to generate binaural early reflection audio signals from the output of the early reflection processor 1051 / 1262.
[0194] Furthermore, in some embodiments the decoder comprises a head orientation determiner which passes the head orientation information to the head pose generator 1257.
[0195] In some embodiments the assignment of reverberator outputs to loudspeaker channels happens during configuration of the reverberator. The assignment can be stored during configuration and provided to the reverberant signal router.
[0196] In some embodiments, the output is a multichannel loudspeaker setup (such as 5.1 or 7.1+4 multichannel loudspeaker setup). In that case, the spatial processing proposed in Fig, .7 can be modified by using the directions of the actual loudspeakers as the directional configuration and omitting the binaural renderers and reproducing the reverberant audio signals from the corresponding loudspeakers of the loudspeaker setup. In the case of loudspeaker output, instead of binaural renderer 409 / 1259 in Fig. 12 there will be a loudspeaker renderer (or panner) which in the simplest case will pass through the output signals to a loudspeaker signal combiner which will replace the binaural signal combiner 1267. Correspondingly, the direct sound part and early reflection part are spatialized with a panner such as vector-base amplitude panning instead of the binaural processors.
[0197] With respect to Fig.13 an example electronic device which may be used as any of the apparatus parts of the system as described above. The device may be any suitable electronics device or apparatus. For example, in some embodiments the device 2000 is a mobile device, user equipment, tablet computer, computer, audio playback apparatus, etc. The device may for example be configured to implement the encoder or the renderer or any functional block as described above.
[0198] In some embodiments the device 2000 comprises at least one processor or central processing unit 2007. The processor 2007 can be configured to execute various program codes such as the methods described herein.
[0199] In some embodiments the device 2000 comprises a memory 2011. In some embodiments the at least one processor 2007 is coupled to the memory 2011. The memory 2011 can be any suitable storage means. In some embodiments the memory 2011 comprises a program code section for storing program codes implementable upon the processor 2007. Furthermore, in some embodiments the memory 2011 can further comprise a stored data section for storing data, for example data that has been processed or to be processed in accordance with the embodiments as described herein. The implemented program code stored within the program code section and the data stored within the stored data section can be retrieved by the processor 2007 whenever needed via the memory-processor coupling.
[0200] In some embodiments the device 2000 comprises a user interface 2005. The user interface 2005 can be coupled in some embodiments to the processor 2007. In some embodiments the processor 2007 can control the operation of the user interface 2005 and receive inputs from the user interface 2005. In some embodiments the user interface 2005 can enable a user to input commands to the device 2000, for example via a keypad. In some embodiments the user interface 2005 can enable the user to obtain information from the device 2000. For example, the user interface 2005 may comprise a display configured to display information from the device 2000 to the user. The user interface 2005 can in some embodiments comprise a touch screen or touch interface capable of both enabling information to be entered to the device 2000 and further displaying information to the user of the device 2000. In some embodiments the user interface 2005 may be the user interface for communicating.
[0201] In some embodiments the device 2000 comprises an input / output port 2009. The input / output port 2009 in some embodiments comprises a transceiver. The transceiver in such embodiments can be coupled to the processor 2007 and configured to enable a communication with other apparatus or electronic devices, for example via a wireless communications network. The transceiver or any suitable transceiver or transmitter and / or receiver means can in some embodiments be configured to communicate with other electronic devices or apparatus via a wire or wired coupling.
[0202] The transceiver can communicate with further apparatus by any suitable known communications protocol. For example, in some embodiments the transceiver can use a suitable universal mobile telecommunications system (UMTS) protocol, a wireless local area network (WLAN) protocol such as for example IEEE 802.X, a suitable short-range radio frequency communication protocol such as Bluetooth, or infrared data communication pathway (IRDA).
[0203] The input / output port 2009 may be configured to receive the signals.
[0204] In some embodiments the device 2000 may be employed as at least part of the renderer. The input / output port 2009 may be coupled to headphones (which may be a headtracked or a non-tracked headphones) or similar.
[0205] It should be understood that the apparatuses may comprise or be coupled to other units or modules used in or for transmission and / or reception. Although the apparatuses have been described as one entity, different modules and memory may be implemented in one or more physical or logical entities.
[0206] It is noted that whilst some embodiments have been described in relation to specific communication networks, similar principles can be applied in relation to other networks and communication systems. Therefore, although certain embodiments were described above by way of example with reference to certain example architectures for wireless networks, technologies and standards, embodiments may be applied to any other suitable forms of communication systems than those illustrated and described herein.
[0207] It is also noted herein that while the above describes example embodiments, there are several variations and modifications which may be made to the disclosed solution without departing from the scope of the present invention.
[0208] As used herein, “at least one of the following: ” and “at least one of ” and similar wording, where the list of two or more elements are joined by “and” or “or”, mean at least any one of the elements, or at least any two or more of the elements, or at least all the elements.
[0209] In general, the various embodiments may be implemented in hardware or special purpose circuitry, software, logic or any combination thereof. Some aspects of the disclosure may be implemented in hardware, while other aspects may be implemented in firmware or software which may be executed by a controller, microprocessor or other computing device, although the disclosure is not limited thereto. While various aspects of the disclosure may be illustrated and described as block diagrams, flow charts, or using some other pictorial representation, it is well understood that these blocks, apparatus, systems, techniques or methods described herein may be implemented in, as non-limiting examples, hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controller or other computing devices, or some combination thereof.
[0210] As used in this application, the term “circuitry” may refer to one or more or all of the following: (a) hardware-only circuit implementations (such as implementations in only analog and / or digital circuitry) and (b) combinations of hardware circuits and software, such as (as applicable): (c) a combination of analog and / or digital hardware circuit(s) with software / firmware and (i) any portions of hardware processor(s) with software (including digital signal processor(s)), software, and memory(ies) that work together to cause an apparatus, such as a mobile phone or server, to perform various functions); and (ii) hardware circuit(s) and or processor(s), such as a microprocessor(s) or a portion of a microprocessor(s), that requires software (e.g., firmware) for operation, but the software may not be present when it is not needed for operation.
[0211] This definition of circuitry applies to all uses of this term in this application, including in any claims. As a further example, as used in this application, the term circuitry also covers an implementation of merely a hardware circuit or processor (or multiple processors) or portion of a hardware circuit or processor and its (or their) accompanying software and / or firmware. The term circuitry also covers, for example and if applicable to the particular claim element, a baseband integrated circuit or processor integrated circuit for a mobile device or a similar integrated circuit in server, a cellular network device, or other computing or network device.
[0212] The embodiments of this disclosure may be implemented by computer software executable by a data processor of the mobile device, such as in the processor entity, or by hardware, or by a combination of software and hardware. Computer software or program, also called program product, including software routines, applets and / or macros, may be stored in any apparatus-readable data storage medium and they comprise program instructions to perform particular tasks. A computer program product may comprise one or more computer-executable components which, when the program is run, are configured to carry out embodiments. The one or more computer-executable components may be at least one software code or portions of it.
[0213] Further in this regard it should be noted that any blocks of the logic flow as in the Figures may represent program steps, or interconnected logic circuits, blocks and functions, or a combination of program steps and logic circuits, blocks and functions. The software may be stored on such physical media as memory chips, or memory blocks implemented within the processor, magnetic media such as hard disk or floppy disks, and optical media such as DVD and the data variants thereof, CD. The physical media is a non-transitory media.
[0214] The term “non-transitory,” as used herein, is a limitation of the medium itself (i.e., tangible, not a signal) as opposed to a limitation on data storage persistency (e.g., RAM vs. ROM).
[0215] The memory may be of any type suitable to the local technical environment and may be implemented using any suitable data storage technology, such as semiconductor based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory and removable memory. The data processors may be of any type suitable to the local technical environment, and may comprise one or more of general purpose computers, special purpose computers, microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASIC), FPGA, gate level circuits and processors based on multi core processor architecture, as non-limiting examples.
[0216] Embodiments of the disclosure may be practiced in various components such as integrated circuit modules. The design of integrated circuits is by and large a highly automated process. Complex and powerful software tools are available for converting a logic level design into a semiconductor circuit design ready to be etched and formed on a semiconductor substrate.
[0217] The scope of protection sought for various embodiments of the disclosure is set out by the independent claims. The embodiments and features, if any, described in this specification that do not fall under the scope of the independent claims are to be interpreted as examples useful for understanding various embodiments of the disclosure.
[0218] The foregoing description has provided by way of non-limiting examples a full and informative description of the exemplary embodiment of this disclosure. However, 5 various modifications and adaptations may become apparent to those skilled in the relevant arts in view of the foregoing description, when read in conjunction with the accompanying drawings and the appended claims. However, all such and similar modifications of the teachings of this disclosure will still fall within the scope of this invention as defined in the appended claims. Indeed, there is a further embodiment 10 comprising a combination of one or more embodiments with any of the other embodiments previously discussed.
Claims
1. An apparatus for rendering at least one reverberant audio signal from at least one audio signal, the apparatus comprising at least one processor and at least one memory storing instructions that, when executed by the at least one processor, cause the system at least to perform:obtaining at least one acoustic parameter;obtaining at least one acoustic modelling parameter;modifying the at least one acoustic parameter based on the at least one acoustic modelling parameter;obtaining the at least one audio signal;configuring at least one reverberator based on the modified at least one acoustic parameter; andrendering at least one reverberant audio signal based on the configured at least one reverberator and the at least one audio signal.
2. The apparatus as claimed in claim 1, caused to perform obtaining at least one acoustic modelling parameter is caused to perform obtaining at least one of:at least one acoustic modeling guidance parameter;at least one acoustic parameter tolerance;at least one reference value; and at least one instructional parameter.
3. The apparatus as claimed in any of claims 1 or 2, caused to perform modifying the at least one acoustic parameter based on the at least one acoustic modelling parameter is caused to perform at least one of:adjusting the at least one acoustic parameter within a provided tolerance within the at least one acoustic modelling parameter;adjusting the at least one acoustic parameter closer to a reference value within the at least one acoustic modeling parameter; andadjusting the at least one acoustic parameter to substantially a value from the at least one acoustic modelling parameter.
4. The apparatus as claimed in any of claims 1 to 3, caused to perform obtaining at least one acoustic modelling parameter is further caused to perform obtaining at least two acoustic modeling parameter values, wherein each of the at least two acoustic modeling parameter values are associated with an acoustic environment parameter.
5. The apparatus as claimed in claim 4, further caused to perform obtaining at least one acoustic environment parameter value, wherein the apparatus caused to perform modifying the at least one acoustic parameter based on the at least one acoustic modelling parameter is further caused to perform:selecting one of the at least two acoustic modeling parameter values based on the obtained at least acoustic environment parameter value; andmodifying the at least one acoustic parameter based on the selected one of the at least two acoustic modeling parameter values.
6. The apparatus as claimed in any of claims 1 to 5, caused to perform obtaining at least one acoustic modelling parameter is further caused to perform obtaining at least two acoustic modeling parameter values, wherein each of the at least two acoustic modeling parameter values are associated with a content or application type.
7. The apparatus as claimed in claim 6, further caused to perform obtaining at least one content or application type value, wherein the apparatus caused to perform modifying the at least one acoustic parameter based on the at least one acoustic modelling parameter is further caused to perform:selecting one of the at least two acoustic modeling parameter values based on the obtained at least one content or application type value; andmodifying the at least one acoustic parameter based on the selected one of the at least two acoustic modeling parameter values.
8. The apparatus as claimed in any of claims 1 to 7, wherein the at least one acoustic parameter comprises at least one of:a decay parameter;a level parameter; anda delay parameter.
9. The apparatus as claimed in any of claims 1 to 8, wherein the at least one acoustic parameter is configured to define an audio scene.
10. A method for an apparatus for rendering at least one reverberant audio signal from at least one audio signal, the method comprising:obtaining at least one acoustic parameter;obtaining at least one acoustic modelling parameter;modifying the at least one acoustic parameter based on the at least one acoustic modelling parameter;obtaining the at least one audio signal;configuring at least one reverberator based on the modified at least one acoustic parameter; andrendering at least one reverberant audio signal based on the configured at least one reverberator and the at least one audio signal.
11. The method as claimed in claim 10, wherein obtaining at least one acoustic modelling parameter comprises obtaining at least one of:at least one acoustic modeling guidance parameter;at least one acoustic parameter tolerance;at least one reference value; andat least one instructional parameter.
12. The method as claimed in any of claims 10 or 11, wherein modifying the at least one acoustic parameter based on the at least one acoustic modelling parameter comprises at least one of:adjusting the at least one acoustic parameter within a provided tolerance within the at least one acoustic modelling parameter;adjusting the at least one acoustic parameter closer to a reference value within the at least one acoustic modeling parameter; andadjusting the at least one acoustic parameter to substantially a value from the at least one acoustic modelling parameter.
13. The method as claimed in any of claims 10 to 12, wherein obtaining at least one acoustic modelling parameter comprises obtaining at least two acoustic modeling parameter values, wherein each of the at least two acoustic modeling parameter values are associated with an acoustic environment parameter.
14. The method as claimed in claim 13, further comprising obtaining at least one acoustic environment parameter value, wherein modifying the at least one acoustic parameter based on the at least one acoustic modelling parameter further comprises: selecting one of the at least two acoustic modeling parameter values based on the obtained at least acoustic environment parameter value; andmodifying the at least one acoustic parameter based on the selected one of the at least two acoustic modeling parameter values.
15. The method as claimed in any of claims 10 to 14, wherein obtaining at least one acoustic modelling parameter further comprises obtaining at least two acoustic modeling parameter values, wherein each of the at least two acoustic modeling parameter values are associated with a content or application type.
16. The method as claimed in claim 15, further comprising obtaining at least one content or application type value, wherein modifying the at least one acoustic parameter based on the at least one acoustic modelling parameter further comprises: selecting one of the at least two acoustic modeling parameter values based on the obtained at least one content or application type value; andmodifying the at least one acoustic parameter based on the selected one of the at least two acoustic modeling parameter values.
17. The method as claimed in any of claims 10 to 16, wherein the at least one acoustic parameter comprises at least one of:a decay parameter;a level parameter;a delay parameter.
18. The method as claimed in any of claims 10 to 17, wherein the at least one acoustic parameter is configured to define an audio scene.
Citation Information
Patent Citations
ViewGB2616280AonEspacenetopensinnewtab
ViewUS2025/0106584A1onEspacenetopensinnewtab
ViewWO2024/115031A1onEspacenetopensinnewtab
ViewUS2024/0196159A1onEspacenetopensinnewtab
ViewEP4192038A1onEspacenetopensinnewtab
