Spatial audio scene description and rendering

Abstract

Spatial audio processing systems and techniques allow for processing of audio objects. The systems and techniques allow for object-based immersive audio decoders and renderers at the consumer playback level, allowing for an object-based immersive audio signal recording or transmission format that enables creators to encode and distribute linear audio content. The end user is then free to select a desired loudspeaker listening configuration and the output may be adjusted accordingly.

Description

Spatial Audio Scene Description and RenderingCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of US Provisional Patent Application 63 / 536,645 by Jot et al., entitled: "Spatial Audio Scene Description and Rendering", filed on 2023-09-05, which is incorporated herein by reference in its entirety for all purposes.FIELD OF TECHNOLOGY

[0002] This patent application relates generally to digital audio processing, and more specifically to spatially immersive digital audio processing.BACKGROUND

[0003] Immersive audio authoring, distribution and rendering technology is essential in broad commercial and industrial application domains, including entertainment, communication, health, design and tourism. A general aspect of immersive audio technology is that its purpose is to enable an author to create a specified listening experience and deliver it to an end user. The end user will hear, simultaneously or in future, during a given time period, an immersive audio signal by means of an audio playback system based on loudspeakers or headphones, such that the spatial distribution of sounds composing the audio signal match the author's intent. In many applications of immersive audio technology, the created audio experience is accompanied by visual media delivered synchronously to the end user.

[0004] Applications of immersive audio technology fall broadly into two categories, depending on whether the end-user listening experience is linear or nonlinear. Linear application scenarios are those in which a temporal sequence of events composing the audio or audiovisual media specified by the author is fixed and cannot be modified by the end user. Linear application scenarios are typical in the distribution of recorded media such as music, movies, and podcasts.

[0005] Nonlinear application scenarios support end-user interaction and include video games, virtual reality (VR) experiences and teleconferencing. In such scenarios, the end user may be placed in the role of a virtual listener located in a virtual acoustic environment containing one or more virtual sound sourcefs], some of which may be the voices of characters or avatars of other end users. Types of end-user interaction may simulate six-degree-of-freedom (6-DoF) movement of the virtual listener and / or virtual sound sources, including both translation and three-degree-of-freedom (3-DoF) rotations.

[0006] Nonlinear audio creation frameworks typically adhere to physical representations of virtual acoustic environments that do not enable levels of sound design expressiveness comparable to what is routinely expected by creators engaged in the production of linear audio content such as music or movie soundtracks.

[0007] Linear audio playback systems do not typically support flexible end-user interaction because linear audio content typically includes synthetic reflected sound associated with prescribed sound elements and sound design choices determined at content creation.

[0008] As a result, today's immersive audio content offerings are split into two consumer experience categories, linear vs. nonlinear, which are mutually exclusive in terms of content creation software frameworks, commercial distribution platforms, playback devices, and consumer interactivity or personalization.SUMMARY

[0009] In certain embodiments, system and techniques may be disclosed that provide for audio processing that overcomes the current fragmentation of immersive audio content, distribution and device ecosystem by (a) defining a generic immersive audio rendering description enabling any compliant device to display nonlinear or linear audio content received in a suitable object-based immersive audio format, and (b) for any immersive audio generation program, pairing with it a scene description interface enabling automatic on-the-fly orfuture translation to the rendering description, thereby ensuring compatibility with any compliant end-user device.

[0010] Clause 1. An object-based spatial audio renderer comprising: a reverberation processor, configured to simulate reflected sound in a first virtual room; and a per-source processor, wherein the object-based spatial audio renderer is configured to perform operations comprising: receiving, with the per-source processor, a first audio object waveform of a first audio object; and determining first audio object scene description metadata of the first audio object, wherein the first audio object scene description metadata is determined based on a first immersive audio generation program; receiving, with the per- source processor, a second audio object waveform of a second audio object; and determining second audio object scene description metadata of the second audio object, wherein the second audio object scene description metadata is determined based on a second immersive audio generation program; and generating, with the reverberation processor based on the first audio object waveform and the first audio object scene description metadata, first reflected sound associated with the first audio object; outputting an audio output comprising first audio and second audio, the first audio comprising the first audio object waveform and the first reflected sound, the second audio comprising the second audio waveform.

[0011] Clause 2. The object-based spatial audio renderer of clause 1, wherein the operations further comprise: generating, with the reverberation processor based on the second audio object waveform and the second audio object scene description metadata, second reflected sound associated with the second audio object, wherein the second audio further comprises the second reflected sound.

[0012] Clause 3. The object-based spatial audio renderer of clause 1, wherein: the determining the first audio object scene description metadata comprises translating data generated by the first immersive audio generation program to a rendering representation by a scene description interface; and the determining the second audio object scene description metadata comprises translating datagenerated by the second immersive audio generation program to the rendering representation by the scene description interface.

[0013] Clause 4. The object-based spatial audio renderer of clause 1, further comprising: the first immersive audio generation program; and the second immersive audio generation program.

[0014] Clause 5. An object-based audio file format encoding method comprising: receiving a first audio object waveform; determining, for the first audio object waveform, a rendering representation level that informs processing performed by an audio renderer; generating, based on the first audio object waveform and the rendering representation level and with an audio renderer, a first audio object rendering representation comprising first audio renderer parameter values, wherein the first audio renderer parameter values comprise parameters of a reverberation processor configured to simulate reflected sound in a first virtual room; generating, based on the first audio object rendering representation, first audio output; and outputting the first audio output.

[0015] Clause 6. The object-based audio file format encoding method of clause5, wherein the rendering representation level comprises a description of first radiation properties of a first virtual sound source associated with the first audio object waveform, and wherein the method further comprises: substituting, with the audio renderer, the first audio object waveform with a second audio waveform, the second audio waveform associated with a second virtual sound source.

[0016] Clause 7. The object-based audio file format encoding method of clause6, wherein the second virtual sound source comprises second radiation properties different from the first radiation properties.

[0017] Clause 8. The object-based audio file format encoding method of clause 5, wherein the rendering representation level comprises orientation coordinates of a first virtual sound source associated with the first audio object waveform, and wherein the method further comprises: modifying, with the audio renderer, the orientation coordinates.

[0018] Clause 9. The object-based audio file format encoding method of clause 5, wherein the rendering representation level comprises a perceived distance of a first virtual sound source associated with the first audio object waveform, and wherein the method further comprises: modifying, with the audio renderer, the perceived distance.

[0019] Clause 10. The object-based audio file format encoding method of clause 5, further comprising: receiving a second audio object waveform; generating, based on the second audio object waveform and with the audio renderer, a second audio object rendering representation comprising second audio renderer parameter values, wherein the second audio renderer parameter values comprise parameters associated with a moving position, a direct-sound Doppler factor and an indirect-sound Doppler factor; generating, based on the second audio object rendering representation, second audio output; and outputting the second audio output.

[0020] Clause 11. The object-based audio file format encoding method of clause 10, wherein: the first audio object and the second audio object are both configured to, for the respective first audio output and the second audio output, generate direct sound and reflected sound within the first virtual room; the direct-sound Doppler factor is configured to affect a pitch-shifting effect affecting the direct sound of the second audio object; and the indirect -sound Doppler factor is configured to affect a pitch-shifting effect affecting the reflected sound of the second audio object.

[0021] Clause 12. The object-based audio file format encoding method of clause 11, wherein the direct-sound Doppler factor is configured to not affect a pitch of the reflected sound of the second audio object and / or a pitch of the direct sound and / or reflected sound of the first audio object, and wherein the indirect-sound Doppler factor is configured to not affect a pitch of the direct sound of the second audio object and / or a pitch of the direct sound and / or reflected sound of the first audio object.

[0022] Clause 13. An object-based spatial audio renderer method comprising: a reverberation processor; and a per-source processor, wherein the object-basedspatial audio Tenderer is configured to perform operations comprising: receive, with the per-source processor, a first audio object waveform of a first audio object; and receive, with the per-source processor, a first audio object rendering representation of the first audio object, the first audio object rendering representation comprising first renderer parameter values, the first renderer parameter values associated with a fixed position; receive, with the per-source processor, a second audio object waveform of a second audio object; and receive, with the per-source processor, a second audio object rendering representation of the second audio object, the second audio object rendering representation comprising second renderer parameter values, the second renderer parameter values associated with a moving position and comprising a direct-sound Doppler factor and an indirect-sound Doppler factor; determine, with the per-source processor and based on the direct-sound Doppler factor, a pitch-shifting effect for direct sound of the second audio object; determine, with the per-source processor and based on the indirect- sound Doppler factor, a pitch-shifting effect for indirect sound of the second audio object; generate, with the reverberation processor, first reflected sound in the first virtual room based on the first audio signal; and generate, with the reverberation processor, second reflected sound in the first virtual room based on the second audio signal.

[0023] Clause 14. The object-based spatial audio renderer of clause 13, wherein a pitch of direct sound of the first audio object, a pitch of the first reflected sound, and / or a pitch of the second reflected sound are not affected by direct- sound Doppler factor.

[0024] Clause 15. The object-based spatial audio renderer of clause 13, wherein a pitch of direct sound of the first audio object, a pitch of the first reflected sound, and / or a pitch of the direct sound of the second audio object are not affected by direct-sound Doppler factor.

[0025] These and other embodiments are described further below with reference to the figures.BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The included drawings are for illustrative purposes and serve only to provide examples of possible structures and operations for the disclosed inventive systems, apparatus, methods, and computer program products for spatial audio processing. These drawings in no way limit any changes in form and detail that may be made by one skilled in the art without departing from the spirit and scope of the disclosed implementations.

[0027] FIG. la depicts a functional diagram of an audio generation program in accordance with embodiments of the present disclosure.

[0028] FIG. lb depicts a simplified signal flow diagram of an immersive audio renderer in accordance with embodiments of the present disclosure.

[0029] FIG. lc depicts a graph of a room impulse response recorded at the position of a listener for a particular location of a sound source emitting an impulsive sound, in accordance with embodiments of the present disclosure.

[0030] FIG. Id depicts a graph illustrating the data contained in an object-based immersive audio signal format for encoding multiple virtual sound sources as audio objects, in accordance with embodiments of the present disclosure.

[0031] FIG. 2 depicts an audio signal flow diagram describing an immersive audio renderer employed in the creation and playback of nonlinear audio content, in accordance with embodiments of the present disclosure.

[0032] FIG. 3 depicts an audio signal flow diagram illustrating a generic immersive audio renderer, in accordance with embodiments of the present disclosure.

[0033] FIG. 4 depicts an audio signal flow diagram illustrating a spatial audio object workstation employed in the creation of linear audio content, in accordance with embodiments of the present disclosure.

[0034] FIG. 5 depicts a functional diagram of an end-to-end interoperable ecosystem for the creation, transmission and playback of immersive audio content, in accordance with embodiments of the present disclosure.

[0035] FIG. 6 depicts a flowchart illustrating the creation, encoding and playback of immersive audio content, in accordance with embodiments of the present disclosure.

[0036] FIG. 7 illustrates a block diagram of an example computing system, in accordance with embodiments of the present disclosure.DETAILED DESCRIPTIONIntroduction

[0037] The various figures described herein may utilize certain solid and dashed lines to signify transmission of various data, such as audio waveforms, metadata, parameters, and / or other such data. Such lines are shown to travel between various modules and components and may be modified (e.g., shifted, combined, and / or otherwise transformed) within one or more of the modules. For the purposes of this disclosure and to improve legibility due to the amount of lines that symbolize such transmission of data, the individual data streams are not labeled with ordinal indicators. Such data may be transmitted through any appropriate wired or wireless technique. Description of the various modules and how they interact and / or transform such data may be described. It is also appreciated that fewer or more data streams than that shown in the figures may be implemented. For clarity, multichannel audio waveforms or processing functions may be indicated with thick lines.

[0038] It is appreciated that, for the purposes of this disclosure, when an element includes a plurality of similar elements distinguished by a letter or follow-on numeral following the ordinal indicator (e.g., "211") and reference is made to only the ordinal indicator itself (e.g., "211"), such a reference is applicable to all the similar elements.

[0039] FIG. la depicts a functional diagram of an audio generation program in accordance with embodiments of the present disclosure. The audio generation program 100 includes an immersive audio renderer 120. The immersive audio renderer outputs an immersive audio signal 130 to a listener employing a set of loudspeakers in a listening room or wearing a pair of headphones. The audiorenderer receives a plurality of audio object signals each representing a virtual sound source perceived by a virtual listener in a virtual acoustic environment. The audio object signals are generated by functional block 110. Each audio object signal comprises an audio object waveform signal 111 and audio object metadata 112 that determine computations executed by the audio renderer.

[0040] FIG. lb depicts a simplified signal flow diagram of an embodiment of immersive audio renderer 130 in accordance with the present disclosure. The immersive audio output signal 130 is computed based on the output of the reverberation processor 170 and the output of the pan and mix block 161 included in per-source processor 160. For example, the outputs of reverberation processor 170 and per-source processor 160 may be additively mixed, or may be mixed together through any other appropriate technique (e.g., the outputs may be subtracted from one another, transformed through certain algorithms such as algorithms that have upper and lower bounds that define when signals are to be added together, and / or through another such technique).

[0041] The pan & mix block 161 receives a first object waveform signal 111 processed by the cascade of a filter 164, an amplitude gain scaler 169, and a shifter 167. Additionally, per-source processor 160 feeds reverberation processor 170 as follows: object waveform signal 111 feeds the cascade combination of shifter 166, amplitude gain scaler 168 and filter 165. The output of filter 165 feeds the pan & mix block 163 and the mix block 162, which respectively feed the reflections block 171 and the reverb block 172, whose outputs are combined to form the output of reverberation processor 170.

[0042] The processing functions executed in processors 160 and 170 are determined by the object metadata 112. Pan & mix blocks 161 and 163 affect, respectively, the apparent direction of arrival of the direct sound and of the reflections heard by the listener. Shifter 167 controls the time and pitch shift of the direct sound. Shifter 166 controls the time and pitch shift of the reflected sound, including reflections and reverberation ("reverb").

[0043] Optionally, audio Tenderer 120 processes a second audio object representing a second virtual sound source located in the virtual room that contains the virtual listener and a first virtual sound source represented by the first audio object. The second audio object waveform signal feeds a second instance of the per-source processor 160, within which all signal processing processing operations are controlled by the second audio object metadata, which also control the processing operation executed in reverberation processor 170.

[0044] In one embodiment of the present disclosure, the first and second virtual sound sources are located in the same virtual room, and therefore the reverberation processor is shared by the first and second objects, whose respective contributions are combined (e.g., additively or via another technique described herein) respectively by way of the mix function 162 and the pan & mix functions 161 and 163. Similarly, by including additional instances of the per-source processor 160, audio renderer 120 may be employed to render a plurality of additional virtual sound sources located in the virtual room that contains the virtual listener.

[0045] Taken together, the processing functions included in the audio renderer 120 enable presenting to the end user an immersive audio experience simulating multiple virtual sound sources situated in a virtual room. In embodiments of the present disclosure, the virtual acoustic environment description, represented by the audio object metadata, includes the positions and orientations of the virtual listener and of all virtual sound sources, as well as the frequency-dependent radiation pattern of each virtual sound source.

[0046] FIG. lc depicts a graph 180 of an impulse response recorded at the position of a virtual listener for a particular location of a virtual sound source emitting an impulsive sound in a virtual room, in accordance with embodiments of the present disclosure. The impulse response is graphed in decibels vs. time. The impulse response is divided into the direct sound 181, and the reflected sound received following it. The direct sound results from the shortest sound propagation path from source to listener in the virtual room. The reflectedsound results from indirect propagation paths, and is divided into early reflections 182, and reverb 183. The time of arrival of the direct sound depends on the source-to-listener distance, and so do the amplitude and the power spectrum of each section in the impulse response (181, 182, and 183). The reflections_delay 184 is the time of arrival (or "onset time") of the first reflection relative to the direct-sound arrival time. The reverb_delay 185 is the onset time of the reverb relative to the direct-sound arrival time. The decaytime 186 denotes the rate of decay of the reverb over time, which typically varies with audio frequency.

[0047] Object-based immersive audio signal transmission or recording formats, such as Dolby Atmos, DTS:X or MPEG-H, have been developed and commercially deployed. FIG. Id depicts a graph illustrating the data contained in an audio recording stored or transmitted in an object-based immersive audio signal format, in accordance with embodiments of the present disclosure. The audio content is carried as a plurality of audio object signals, each representing a virtual sound source and comprising an audio object waveform 111 (such as a sound file or stream reference) and audio object metadata 112. Each audio object 191 is present in the audio content during a lifetime spanning from a start time 192 to an end time 193, during which the audio object waveform may be audible or silent. In various embodiments, audio object 191 may be sound produced from a sound source. The audio object 191 may include all sound produced from the sound source or sound produced from the sound source for a limited period of time.

[0048] The audio object metadata represent a set of properties of the virtual sound source, such as its spatial position and orientation, some or all of which may vary during the audio object's lifetime. When the object-based audio signal is received and played back by audio renderer 120, as in FIG. la and lb., the audio object metadata collectively instruct the signal processing operations executed by the audio renderer.

[0049] By provisioning an object-based immersive audio decoder and renderer at the consumer playback end, an object-based immersive audio signalrecording or transmission format enables creators to encode and distribute linear audio content such that the end user remains free to select a desired loudspeaker listening configuration (including both the number and the geometrical layout of the loudspeakers). Additionally, when listening over headphones, the end-user's head orientation may be automatically compensated in order to produce a spatial audio experience simulating loudspeaker-based playback.Nonlinear immersive audio authoring and rendering

[0050] Nonlinear authoring software programs for game or Virtual Reality (VR) content creation, such as Unreal or Unity, commonly rely on physically based virtual world descriptions. A nonlinear authoring software program comprises an audio generation program driven by a world description including a physically-based audio environment description from which the authoring software program generates a graphical representation displayed to the author or developer during content creation. This physically-based audio environment description includes geometrical and acoustical properties assigned e.g. to virtual wall, floor or ceiling boundaries and employed by the authoring software program to automatically generate synthetic reflected sound for each audio object by simulating the propagation of sound from each virtual sound source to the virtual listener, including acoustic reflections and reverberation in the virtual audio environment.

[0051] Nonlinear content distribution solutions require that the end-user playback program incorporate audio renderer functionally equivalent to the one employed in the authoring software program during content creation. For instance, a VR application program intended for playback on the Meta Quest 3 VR headset must be developed specifically for that device and cannot be readily consumed by an end user employing a Sony Playstation 5 video game console, in such a manner that an identical listening experience be delivered to the end user via either device.

[0052] FIG. 2 depicts an audio signal flow diagram illustrating an immersive audio renderer 200 suitable for the creation or playback of nonlinear audio in avideo game or VR scenario. As with the renderer of FIG. lb, the processing of a single virtual sound source is represented. The renderer can process any number of additional audio objects by inputting a respective audio object waveform signal 111 to a respective additional instance of the per-source processing block 160.

[0053] The immersive audio output signal 130 is produced by a virtualizer block 240 that receives the direct-sound contribution via one of the buses Left-Right (253), Standard (252) or Diffuse (251) together with one of the panners 212N. Selector 215 is configured automatically according to the positional audio rendering technique implemented by the renderer according to constraints dictated by the end-user's loudspeaker or headphone configuration. Selector 215, thus, allows for applications of different shifters and / or panners based on whether the positional audio rendering mode is standard, diffuse, or binaural (using the left and right output buses).

[0054] Additionally, the virtualizer 240 receives reflected sound via the reverberation processor output bus 254, including reflections produced by reflections block 171 and one or more reverb blocks 172, each of which can model the reverb of the virtual room containing the virtual listener or the reverb of an adjacent virtual room in the virtual acoustic environment. The output of reverb block 172 is processed by a panning module 216 that can apply an adjustable directional emphasis to the reverb output signal in order to reproduce the perception of reverberation emanating from an adjacent room via an acoustic opening such as a door. Additionally, the output of reflections block 171 or a reverb block 172 is processed by a respective filter 217 that applies a spectral correction suitable to reproduce the perception of reflected sound emanating from an adjacent virtual room via transmission through a virtual wall.

[0055] As in the audio renderer of FIG. lb, the per-source processor 160 includes a panner that can provide a directional emphasis to the early reflections generated by reflections block 171. Per-source processing block 160 also includes a set of filters and gain scalers 211N each cascaded with acorresponding shifter 213N feeding one of the reflections or reverb blocks via the reflection send bus 231 or a reverb send bus 232, respectively. Shifter 214 is configured to apply the same pitch shift to its left and right outputs while allowing for the insertion of a time delay between the left and right signals.

[0056] The parameters of all audio signal processing functions included in the per-source processor 160 are determined by the audio object metadata 112. Taken together, the processing functions included in the audio renderer 200 enable presenting to the end user an immersive audio listening experience, parametrized by the audio object metadata 112, simulating multiple virtual sound sources situated in a virtual acoustic environment comprising several connected virtual rooms, in the context of a video game or VR application scenario, derived from the physically-based description of a virtual audio environment inhabited by the virtual listener. The physically-based description includes the geometry of room walls, ceiling, floor and obstacles and their acoustical properties such as absorption, reflections and diffusion coefficients. Linear immersive audio creation

[0057] Linear audio content or soundtrack authoring workflows such as for the production of music, movie or podcast audio commonly rely on Digital Audio Workstations (a.k.a. DAWs). These DAWs, such as Avid Pro Tools or Steinberg Nuendo, incorporate audio processing software functions that enable perceptually-based sound design capabilities which are not functionally restricted by a requirement to match a physically-based acoustic environment description, and can instead be purely motivated by the sound author's aesthetic sound design intent. An essential sound design function of DAWs is the application of synthetic reflected sound, including reflections and / or reverb.

[0058] Linear audio content is commonly distributed from the creation studio to the consumer by exporting from the DAW software program one or several fixed recorded versions of the audio content. For this purpose, it is advantageous for the author to employ an object-based immersive audio recording format where audio object metadata include the apparent position of each audio object relative to a reference position and orientation of thelistener's head. At the playback end, an object-based Tenderer can then produce an immersive audio output signal that will faithfully convey to the listener the spatial distribution of sound elements created by its author, accounting for the end-user's loudspeaker or headphone configuration.

[0059] In existing object-based immersive audio signal formats, the audio object metadata 112 do not include synthetic reflected sound properties. Consequently, a playback-end object-based audio Tenderer is not able to apply to any individual audio object the synthetic reflected sound effects (reflection and reverberation) prescribed by the authoring program. Reflected sound effects designed by the author remain fixed in the recording as exported originally during the content creation process, and can only be included in the content transmitted to an end-user by ensuring that the object-based immersive audio signal 160 comprises one or more audio objects that include them in their audio object waveform signal. Consequently, when receiving and playing such linear immersive audio content, it is not practical for the end-user to perform end-user interactions such as suppressing or substituting audio objects or displacing the position of the virtual listener within the virtual audio environment.

[0060] In order to overcome this limitation in end-user interaction with linear audio content, it is, in theory, possible to employ an existing nonlinear audio creation and distribution solution suitable for video game or VR applications. However, this approach is impractical for the author because it requires restricting distribution to specific video game or VR end-user devices, instead of ensuring de-facto compatibility with any end-user device capable of playing back linear audio content.Generic 6-degree-of-freedom spatial audio Tenderer

[0061] FIG. 3 depicts an audio signal flow diagram illustrating a generic immersive audio Tenderer suitable for the creation or playback of linear or nonlinear object-based immersive audio signals, in accordance with embodiments of the present disclosure. FIG. 3 illustrates a configuration of Tenderer 300 that may include certain systems that are similar to that ofrenderer 200. For the purposes of this disclosure, similar elements may be denoted with the same ordinal indicator and the disclosure of such elements may apply to all such elements sharing the ordinal indicator within all figures. It is appreciated that, in various embodiments, such elements may include variations between different configurations.

[0062] As in FIG.2, the per-source processor 160 processes audio object waveform signal 111 parametrized by audio object metadata 112. In one embodiment of the present disclosure, at least one of the immersive audio objects is a reverberant audio object, i.e. an audio object where the audio object metadata 112 include all information needed to configure a reverberation processor such as the reverberation processor of FIG. lb, including one or more reflections blocks 171 and / or reverb blocks 172, each associated in cascade with a panning block 216 and / or a filtering block 217.

[0063] In embodiments of the present disclosure, renderer 300 processes a second reverberant audio object. The second audio object waveform 111 feeds a second instance of the per-source processor 160, where all signal processing functions are controlled by the second audio object metadata 112. The second audio object metadata, combined with the first object metadata, together define the configuration of the effect slots 370. If the reverb parameters in the second reverberant audio object metadata are identical to those in the first audio object metadata, including panning and filtering parameters, the two objects can share the same set of reverb blocks and reverb send buses. This simplification may occur, typically, if the two objects represent virtual sound sources situated in the same virtual room. Similarly, the two objects may share a single reflections block and the reflections send bus that feeds it.

[0064] Similarly, in embodiments of the present disclosure, renderer 300 may process a multiplicity of reverberant audio objects, by instantiating additional reverb or reflection blocks in the effect slots 370 and the respective summing lines in the effect send bus 230. Upon the end of the lifetime of one of the reverberant audio objects, the processing resources employed by this object inthe effect slots 370 or in the effect send bus 230 may be freed up unless another reverberant audio object requires them.

[0065] In embodiments of the present disclosure, the audio signal processing capabilities of audio renderer 300 are fully defined by its rendering description, i.e. a container hosting the parameter fields whose values must be specified for any reverberant audio object to determine, via its audio object metadata, the configuration and parameter settings of the processing applied in the persource processor and in the effect slots.

[0066] FIG. 4 depicts an audio signal flow diagram illustrating a Spatial Audio Object Workstation 400 suitable for authoring linear immersive audio content. In embodiments of the spatial audio object workstation, its audio signal processing functionality is exactly equivalent to that of generic immersive audio renderer 300. In FIG. 4, this functionality is represented according to the conventional topology of a DAW, with each audio object waveform 111 representing a sound source such as musical instrument or a human voice recorded with a microphone.

[0067] During the linear audio creation process, according to embodiments of the present disclosure, an author will adjust the parameters exposed by a DAW application, and any such adjustment will translate to reverberant audio object metadata that can be recorded or transmitted as part of an object-based immersive audio signal. As a result, according to an embodiment of the present disclosure, the author of a linear immersive audio experience can perceive the created object-based immersive audio content exactly as it will be perceived by an end-user listening to an object-based immersive audio signal containing the same set of reverberant spatial audio objects.Universal end-to-end 6-degree-of-freedom immersive audio ecosystem

[0068] FIG. 5 depicts a functional diagram of an end-to-end interoperable ecosystem for the creation, transmission and playback of immersive audio content in accordance with embodiments of the present disclosure. Immersive audio content generated by programs 551a and 551b is translated, transmitted, and displayed to the end user over loudspeakers or headphones by use of anend-user playback program 510, which outputs the immersive audio signal 130 generated by audio renderer 120.

[0069] Immersive audio generation program 551a generates an object-based audio scene description 560a in the form of an object-based immersive audio signal comprising a set of reverberant audio objects, each comprising an audio object waveform signal and audio object metadata. The scene description is transmitted to a scene description interface block 520a, which translates all of the audio object metadata in the scene description 560a to audio object metadata 111 compliant with the rendering description and transmitted to the end-user playback program 510.

[0070] In embodiments of the present disclosure, the immersive audio generation program is a physically-based nonlinear immersive audio generation program such as a video game or a VR experience. In other embodiments, it is a perceptually-based linear immersive audio creation program including a spatial audio object workstation.

[0071] In embodiments of the present disclosure, a second immersive audio generation program 551b generates a second scene description 560b comprising additional immersive audio objects to be included in the immersive audio presentation delivered to the end user via audio renderer 120. For this purpose, a second scene description interface block 520b translates the audio object metadata in scene description 560b to audio object metadata compliant with the rendering description. In embodiments of the present disclosure, the first immersive audio generation program may be a virtual reality application representing a live music concert stage and the second immersive audio generation program may be a communication application enabling remote interaction between avatars of event attendees.

[0072] Optionally, the translated scene description output by block 520a, optionally combined with the scene description output by block 520b, is encoded by encoder 530 before transmission to the end-user playback program 510. The function of encoder 530 may include, for instance, low-bit-rate audio coding of some of the audio object waveform signals 112. In playback program510, the rendering interface block 540 decodes and parses the incoming objectbased immersive audio signal and dynamically configures the audio renderer and / or updates its processing parameter settings as instructed by the audio object metadata 111.

[0073] As a result, as illustrated in the flowchart of FIG. 6, any immersive audio generation program that is paired with a scene description interface that is compliant with the rendering description, which describes the audio Tenderer's signal processing capabilities, is interoperable with any rendering device that is compliant with this same rendering description. In various embodiments of the present disclosure, the rendering device may be, for instance, a TV, a mobile computing device, a car audio system, a pair of headphones or earbuds, or a virtual or augmented reality headset.

[0074] In 610, immersive audio generation program (551a) generates an object-based immersive audio scene description (560a). In 620, immersive audio scene description (560a) is translated by scene description interface block to ensure compliance with the rendering description. In optional 630, audio objects included in a second scene description (560b) are juxtaposed with audio objects from the first scene description. In optional 640, the translated scene descriptions are encoded and are transmitted to the end-user playback program. Additionally or alternatively, the combined scene descriptions are encoded and are transmitted to the end-user playback program. In option 650, the incoming object-based immersive audio signal is modified in response to end-user interaction. In 660, the object-based immersive audio signal is rendered and displayed to the end user over loudspeakers or headphones.

[0075] Therefore, according to embodiments of the present disclosure, it is possible to encode reverberant audio objects in object-based immersive audio content creation and distribution, as illustrated in FIG. 6, indifferently of whether the immersive audio generation program employs a physically-based spatial audio scene model or a perceptually-based one.

[0076] At the receiving end, the object-based renderer can reproduce each reverberant audio object exactly as specified by the immersive audiogeneration program employed to create it, even in a use case where the audio content comprises a plurality of audio objects individually generated by several independent immersive audio generation programs (which may be e.g. independent programs such as video games, VR applications, online music or video services, teleconferencing applications, or social communication applications).

[0077] Optionally, the rendering interface block 540 modifies the object-based audio signal received by playback program 510 by removing, substituting, inserting or modifying some of the audio objects, in response to end-user interaction. The audio renderer 120 can perform processing that differs from the parameter settings supplied in the audio object metadata 111. The enduser playback program 510 allows end-user interaction with one or multiple elements in the immersive audio output signal 130, potentially overriding or modifying the value of any one of the audio signal processing parameters applied in the audio renderer 120.Rendering representation levels in reverberant audio object metadata

[0078] A 6-DoF immersive audio rendering interface as described herein may provide for one or more of: (a) allowing maximum flexibility in per-object sound design by the content creator employing the Immersive audio generation program; (b) ensuring faithful playback of immersive audio scenes to the end user; (c) letting the end user take advantage of the interactivity and personalization benefits enabled by the object-based audio scene representation, within limits authorized by the content creator; (d) enabling both the content creator and the playback program developer to select among several possible levels of end-user interaction and implementation complexity.

[0079] According to embodiments of the present disclosure, audio object metadata 111 include, for each reverberant object, a rendering representation level that determines a level of end-user interaction enabled upon rendering. According to embodiments of the present disclosure, reverberant audio object metadata are categorized into a hierarchy of functionality profiles, exposing increasing levels of playback-end immersive audio scene interaction.

[0080] In one embodiment of the present disclosure, rendering representation level 1 is a minimal set of audio object metadata that ensures that the end-user listening experience can reproduce exactly any immersive audio scene description compliant with the rendering interface. The parameters included in the level 1 reverberant audio object metadata set cover all features of the audio signal processing functions performed in the per-source processor 160 and in the effect slots 370.

[0081] In one embodiment, rendering representation level 1 includes (a) the direction and spatial extent of the direct sound as well as its power spectrum vs. frequency, (b) the direction, spatial extent, onset time and power spectrum of the reflections, and (c) the direction, spatial extent, onset time, power spectrum and reverb decay rate vs. frequency. Direction and spatial extent may be represented by a 3-dimensional vector whose magnitude is 1.0 for a spatially focused sound, and decreases towards 0.0 to represent a more spatially diffuse sound. Frequency dependent parameters may be expressed by values at low, mid and high frequencies. Additionally, as authorized by the author, rendering representation level 1 optionally allows gain scaling, removal, substitution, or 3- DoF repositioning of any individual reverberant audio object.

[0082] In one embodiment, rendering representation level 2 includes additional audio object metadata that provide spectral correction (amplitude scaling at low, mid and high frequencies) of the axial and of the diffuse sound radiated by the virtual sound source represented by the audio object. For a virtual sound source pointing towards the virtual listener, these spectral corrections scale, respectively, the amplitude of the direct sound and the amplitude of the reflected sound.

[0083] In embodiments of the present disclosure, virtual sound source radiation properties (i.e. the axial and diffuse spectral correction parameters) are included in the reverberant audio object metadata. This enables user-end interaction scenarios such as remixing or language replacement, where an incoming reverberant audio object (representing a human's voice, for instance) may be replaced by an audio object supplied locally by the end-user playbackprogram 510 to the rendering interface block 540. In this case, the virtual sound source's radiation properties (i.e. the axial and diffuse spectral correction parameters) may be exchanged along with the object waveform, while the other incoming audio object metadata remain unaffected.

[0084] In one embodiment, rendering representation level 3 includes additional audio object metadata that facilitate playback-end 6-DoF navigation of the virtual listener or 6-DoF repositioning of individual audio objects, including (a) orientation coordinates and off-axis directivity pattern of the virtual sound source represented by an audio object; (b) source-listener distance expressed in meters; and (c) direct-sound and indirect-sound Doppler factor parameter values.

[0085] One particular capability of a 6-DoF nonlinear or linear audio rendering engine is the simulation of a moving virtual listener and / or a moving virtual sound source, such that the variation in source-to-listener distance may cause an audible pitch-shift often referred to as Doppler effect. For this purpose, each of the shifters in the generic spatial audio renderer of FIG. 3 can shift (increase or reduce) the pitch of the audio signal independently of the duration of the delay that it introduces. For each audio object, the audio renderer 300 of FIG. 3 applies separate shifter function blocks to its direct sound and its reflected sounds. Therefore, the Doppler effect affecting the direct sound for each audio object may be scaled by the direct-sound Doppler factor parameter while the pitch shift affecting the reflected sound produced for this object by the reflection or reverberation block may be scaled independently by the indirect- sound Doppler factor parameter.Computing System Example

[0086] FIG. 7 illustrates a block diagram of an example computing system, in accordance with embodiments of the present disclosure. According to various embodiments, a system 700 suitable for implementing embodiments described herein includes a processor 702, a memory module 704, a storage device 706, an interface 712, and a bus 716 (e.g., a PCI bus or other interconnection fabric.) System 700 may operate as a variety of devices such as a server system such asan application server and a database server, a user device such as a laptop, desktop, smartphone, tablet, wearable device, set top box, etc., or any other device or service described herein.

[0087] Although a particular configuration is described, a variety of alternative configurations are possible. The processor 702 may perform operations such as those described herein. Instructions for performing such operations may be embodied in the memory 704, on one or more non-transitory computer readable media, or on some other storage device. Various specially configured devices can also be used in place of or in addition to the processor 702. The interface 712 may be configured to send and receive data packets over a network. Examples of supported interfaces include, but are not limited to: Ethernet, fast Ethernet, Gigabit Ethernet, frame relay, cable, digital subscriber line (DSL), token ring, Asynchronous Transfer Mode (ATM), High-Speed Serial Interface (HSSI), and Fiber Distributed Data Interface (FDDI). These interfaces may include ports appropriate for communication with the appropriate media. They may also include an independent processor and / or volatile RAM. A computer system or computing device may include or communicate with a monitor, printer, or other suitable display for providing any of the results mentioned herein to a user.

[0088] Any of the disclosed embodiments may be embodied in various types of hardware, software, firmware, computer readable media, and combinations thereof. For example, some techniques disclosed herein may be implemented, at least in part, by non-transitory computer-readable media that include program instructions, state information, etc., for configuring a computing system to perform various services and operations described herein. Examples of program instructions include both machine code, such as produced by a compiler, and higher-level code that may be executed via an interpreter. Instructions may be embodied in any suitable language such as, for example, Java, Python, C++, C, HTML, any other markup language, JavaScript, ActiveX, VBScript, or Perl. Examples of non-transitory computer-readable media include, but are not limited to: magnetic media such as hard disks and magnetic tape;optical media such as flash memory, compact disk (CD) or digital versatile disk (DVD); magneto-optical media; and other hardware devices such as read-only memory ("ROM") devices and random-access memory ("RAM") devices. A non- transitory computer-readable medium may be any combination of such storage devices.

[0089] In the foregoing specification, various techniques and mechanisms may have been described in singular form for clarity. However, it should be noted that some embodiments include multiple iterations of a technique or multiple instantiations of a mechanism unless otherwise noted. For example, a system uses a processor in a variety of contexts but can use multiple processors while remaining within the scope of the present disclosure unless otherwise noted. Similarly, various techniques and mechanisms may have been described as including a connection between two entities. However, a connection does not necessarily mean a direct, unimpeded connection, as a variety of other entities (e.g., bridges, controllers, gateways, etc.) may reside between the two entities.

[0090] While various embodiments have been described herein, it should be understood that they have been presented by way of example only, and not limitation. For example, some techniques and mechanisms are described herein in the context of fulfillment. However, the disclosed techniques apply to a wide variety of circumstances. Particular embodiments may be implemented without some or all of the specific details described herein. In other instances, well known process operations have not been described in detail in order not to unnecessarily obscure the techniques disclosed herein. Accordingly, the breadth and scope of the present application should not be limited by any of the embodiments described herein, but should be defined only in accordance with the claims and their equivalents.Conclusion

[0091] Although the foregoing concepts have been described in some detail for purposes of clarity of understanding, it will be apparent that certain changes and modifications may be practiced within the scope of the appended claims. It should be noted that there are many alternative ways of implementingprocesses, systems, and apparatuses. Accordingly, the present embodiments are to be considered illustrative and not restrictive.

Claims

AMENDED CLAIMS received by the International Bureau on 17 February 2025 (17.02.2025)1. An object-based spatial audio renderer comprising: a reverberation processor, configured to simulate reflected sound in a first virtual room; and a per-source processor, wherein the object-based spatial audio renderer is configured to perform operations comprising: receiving, with the per-source processor, a first audio object waveform of a first audio object; and determining first audio object scene description metadata of the first audio object, wherein the first audio object scene description metadata is determined based on a first immersive audio generation program; receiving, with the per-source processor, a second audio object waveform of a second audio object; and determining second audio object scene description metadata of the second audio object, wherein the second audio object scene description metadata is determined based on a second immersive audio generation program; and generating, with the reverberation processor based on the first audio object waveform and the first audio object scene description metadata, first reflected sound associated with the first audio object; outputting an audio output comprising first audio and second audio, the first audio comprising the first audio object waveform and the first reflected sound, the second audio comprising the second audio waveform.

2. The object-based spatial audio renderer of claim 1, wherein the operations further comprise: generating, with the reverberation processor based on the second audio object waveform and the second audio object scene description metadata, second reflected sound associated with the second audio object, wherein the second audio further comprises the second reflected sound.

3. The object-based spatial audio renderer of claim 1, wherein: the determining the first audio object scene description metadata comprises translating data generated by the first immersive audio generation program to a rendering representation by a scene description interface; and the determining the second audio object scene description metadata comprises translating data generated by the second immersive audio generation program to the rendering representation by the scene description interface.

4. The object-based spatial audio renderer of claim 1, further comprising: the first immersive audio generation program; and the second immersive audio generation program.

5. An object-based audio file format encoding method comprising: receiving a first audio object waveform; determining, for the first audio object waveform, a rendering representation level that informs processing performed by an audio renderer; generating, based on the first audio object waveform and the rendering representation level and with an audio renderer, a first audio object rendering representation comprising first audio renderer parameter values, wherein the first audio renderer parameter values comprise parameters of a reverberation processor configured to simulate reflected sound in a first virtual room; generating, based on the first audio object rendering representation, first audio output; and outputting the first audio output.

6. The object-based audio file format encoding method of claim 5, wherein the rendering representation level comprises a description of first radiation properties of a first virtual sound source associated with the first audio object waveform, and wherein the method further comprises:substituting, with the audio renderer, the first audio object waveform with a second audio waveform, the second audio waveform associated with a second virtual sound source.

7. The object-based audio file format encoding method of claim 6, wherein the second virtual sound source comprises second radiation properties different from the first radiation properties.

8. The object-based audio file format encoding method of claim 5, wherein the rendering representation level comprises orientation coordinates of a first virtual sound source associated with the first audio object waveform, and wherein the method further comprises: modifying, with the audio renderer, the orientation coordinates.

9. The object-based audio file format encoding method of claim 5, wherein the rendering representation level comprises a perceived distance of a first virtual sound source associated with the first audio object waveform, and wherein the method further comprises: modifying, with the audio renderer, the perceived distance.

10. The object-based audio file format encoding method of claim 5, further comprising: receiving a second audio object waveform; generating, based on the second audio object waveform and with the audio renderer, a second audio object rendering representation comprising second audio renderer parameter values, wherein the second audio renderer parameter values comprise parameters associated with a moving position, a direct-sound Doppler factor and an indirect-sound Doppler factor; generating, based on the second audio object rendering representation, second audio output; and outputting the second audio output.

11. The object-based audio file format encoding method of claim 10, wherein:the first audio object and the second audio object are both configured to, for the respective first audio output and the second audio output, generate direct sound and reflected sound within the first virtual room; the direct-sound Doppler factor is configured to affect a pitch-shifting effect affecting the direct sound of the second audio object; and the indirect-sound Doppler factor is configured to affect a pitch-shifting effect affecting the reflected sound of the second audio object.

12. The object-based audio file format encoding method of claim 11, wherein the direct-soundDoppler factor is configured to not affect a pitch of the reflected sound of the second audio object and / or a pitch of the direct sound and / or reflected sound of the first audio object, and wherein the indirect-sound Doppler factor is configured to not affect a pitch of the direct sound of the second audio object and / or a pitch of the direct sound and / or reflected sound of the first audio object.

13. An object-based spatial audio renderer comprising: a reverberation processor; and a per-source processor, wherein the object-based spatial audio renderer is configured to perform operations comprising: receive, with the per-source processor, a first audio object waveform of a first audio object; and receive, with the per-source processor, a first audio object rendering representation of the first audio object, the first audio object rendering representation comprising first renderer parameter values, the first renderer parameter values associated with a fixed position; receive, with the per-source processor, a second audio object waveform of a second audio object; and receive, with the per-source processor, a second audio object rendering representation of the second audio object, the second audio object renderingrepresentation comprising second renderer parameter values, the second renderer parameter values associated with a moving position and comprising a direct-soundDoppler factor and an indirect-sound Doppler factor; determine, with the per-source processor and based on the direct-soundDoppler factor, a pitch-shifting effect for direct sound of the second audio object; determine, with the per-source processor and based on the indirect-soundDoppler factor, a pitch-shifting effect for indirect sound of the second audio object; generate, with the reverberation processor, first reflected sound in a first virtual room based on the first audio signal; and generate, with the reverberation processor, second reflected sound in the first virtual room based on the second audio signal.

14. The object-based spatial audio renderer of claim 13, wherein a pitch of direct sound of the first audio object, a pitch of the first reflected sound, and / or a pitch of the second reflected sound are not affected by direct-sound Doppler factor.

15. The object-based spatial audio renderer of claim 13, wherein a pitch of direct sound of the first audio object, a pitch of the first reflected sound, and / or a pitch of the direct sound of the second audio object are not affected by direct-sound Doppler factor.

Images