Method, apparatus and system for processing an audio scene for audio rendering

Abstract

The present disclosure relates to a method for processing audio scene information for audio rendering, the method comprising the steps of receiving an audio scene description, the audio scene description including a representation of a three-dimensional audio scene and information about a source position of a sound source within the audio scene, receiving an indication of a listener position of a listener within the audio scene, obtaining diffraction information about acoustic diffraction paths within the audio scene between the sound source position and the listener position, performing audio rendering for the sound source based on the diffraction information, and outputting a representation of the diffraction information. The present disclosure further relates to corresponding apparatus, computer programs and computer readable media.

Description

[Technical field]

[0001] [CROSS REFERENCE TO RELATED APPLICATIONS] This application claims priority to U.S. Provisional Application No. 63 / 318,080, filed March 9, 2022, and U.S. Provisional Application No. 63 / 413,719, filed October 6, 2022, the entire contents of which are incorporated by reference herein.

[0002] [Technical field] This disclosure relates to techniques for processing audio scene information for audio rendering, and in particular, to voxel-based scene representation and audio rendering. [Background technology]

[0003] MPEG (Moving Picture Experts Group) is an association of working groups established jointly by the International Organization for Standardisation (ISO) and the International Electrotechnical Commission (IEC) to set standards for media coding, including audio coding. MPEG is organised under ISO / IEC SC 29, and the audio group is now identified as working group (WG) 6. WG 6 is currently working on a new audio standard (MPEG-I Immersive Audio, also known as ISO / IEC 23090-4).

[0004] The new MPEG-I standard enables virtual reality (VR), augmented reality (AR), mixed reality (MR) and / or extended reality (XR) applications to experience audio from different viewpoints and / or perspectives or listening positions by supporting various movements of a scene and around such a scene, such as movements using various degrees of freedom, such as three degrees of freedom (3DOF) or six degrees of freedom (6DoF). 6DoF interaction extends the 3DoF spherical video / audio experience, which is limited to head rotation (pitch, yaw and roll), to include translational movements (forward / back, up / down and left / right), allowing navigation within the virtual environment in addition to head rotation (e.g., physical walking in a room).

[0005] For audio rendering in VR, AR, MR and XR applications, an object-based approach is widely used by representing a complex auditory scene as multiple separate audio objects, each of which is associated with parameters or metadata that define the location / position and trajectory of that object in the scene. Alternatively, audio rendering in such environments also uses higher order ambisonics (HOA). However, new uses of "voxels" for rendering audio scenes are currently being explored, such as for use in new immersive audio experiences. Voxels for audio rendering are relevant for both hardware and software implemented media environments, such as video games and / or VR, AR, MR and XR environments.

[0006] A voxel is a spatial volume that is assigned acoustic properties or audio rendering instructions. The voxel size may be an encoder configuration parameter, which can be selected (manually or automatically) according to the scene geometry level of detail (e.g., in the range of 10 cm to 1 m).

[0007] Voxels for audio rendering can be obtained by: Voxelization (or conversion) of mesh-based scene representations From a scene representation used for scene generation (or video rendering) (e.g. by downsampling to smaller sized voxels)

[0008] However, traditional approaches for using voxels to provide realistic sound for user experiences (including those involving motion) in VR, AR, MR and XR environments remain challenging and computationally complex.

[0009] Typical techniques for diffraction modeling in three-dimensional audio scenes, such as computer-mediated reality applications, require recalculation of diffraction paths and other diffraction information each time either the audio scene, the user position, or the sound source position changes. For example, the diffraction paths may change as the user and / or sound source move through the three-dimensional audio scene. Furthermore, the diffraction paths may change when the audio scene itself changes, for example, by showing doors or windows that open or close. Frequent recalculation of diffraction paths can be computationally expensive, which may require relatively powerful computing devices to implement the computer-mediated reality application and / or may in some cases adversely affect the user experience.

[0010] Thus, there is a need for improved techniques for diffraction modeling in three-dimensional audio scenes, particularly in three-dimensional audio scenes that utilize voxels, and in particular techniques that can reduce the computational burden on devices (e.g., decoders, renderers) that implement such techniques. Summary of the Invention

[0011] In view of this need, the present disclosure provides a method for processing audio scene information for audio rendering (in particular voxel-based audio scene information), an apparatus for processing audio scene information for audio rendering, a computer program and a computer-readable storage medium, having the features of the respective independent claims.

[0012] One aspect of the disclosure relates to a method for processing audio scene information for audio rendering. The method may include receiving an audio scene description. The audio scene description may include a representation of a three-dimensional audio scene and information regarding a source position of a sound source within the audio scene. The method may further include receiving an indication of a listener position of a listener within the audio scene. The method may further include obtaining diffraction information regarding acoustic diffraction paths in the audio scene between the sound source position and the listener position. The method may further include performing audio rendering for the sound source based on the diffraction information. The method may further include outputting a representation of the diffraction information. The output of the representation of the diffraction information may be to at least one external (e.g., shared) data source or repository, enabling reuse of the diffraction information by external rendering or decoder instances.

[0013] In addition to local reuse, the proposed method provides an interface for sharing the generated / calculated diffraction information between rendering instances (or decoder instances) (e.g. a data interface for exchanging data, including a predefined format for the representation of the diffraction information). The interface may be implemented in a software format that can be executed on one or more hardware platforms. For example, the interface may be a graphical user interface that displays the information and / or allows user interaction. This can be used to establish a framework of rendering instances that share locally generated (or externally retrieved) diffraction information between them. Thereby, the computational power required for rendering at each rendering instance can be reduced. Importantly, the diffraction information is generated at the decoder side, thus automatically ensuring that it is applicable to real use cases and situations. In particular, a large amount of diffraction information is made available for listener positions that are frequently visited by real users in the audio scene. Thus, the amount of storage (e.g. shared / physical storage or bitstream) for storing the representation of the diffraction information can be used efficiently and is used to store mostly representations of substantially related diffraction information.

[0014] In some embodiments, outputting the representation of the diffraction information may include outputting data elements including the diffraction information and information about the scene state. The scene state may include an audio scene description and a listener position. The output of the data elements may be to a bitstream or storage. In particular, the output may be to at least one external (e.g. non-local, in particular shared) data source or repository (e.g. shared / cloud memory or bitstream), allowing reuse of the diffraction information by external rendering instances.

[0015] The diffraction information may be reused at a later point in time by the same device / decoder / renderer or may be used by other devices / decoders / renderers. The scene state contained in the data elements can be used by the device / decoder / renderer to determine whether the available diffraction information is applicable to a given configuration of audio scenes and listeners therein, or in other words, whether diffraction information is available for a given configuration.

[0016] In some embodiments, the representation of the diffraction information may be output to a bitstream (an output bitstream) and / or to storage.

[0017] In some embodiments, the diffraction information may be output for later reuse for audio rendering by the same rendering instance, or for later reuse by a different rendering instance.

[0018] In some embodiments, a representation of the diffraction information may be output as part of the voxSceneDiffractionPreComputedPathData() syntax element in accordance with the MPEG-I standard or any subsequent version of the MPEG-I standard.

[0019] In some embodiments, the diffraction information may indicate a virtual source location of a virtual sound source. This virtual sound source may be selected to "encapsulate" the application of diffraction and / or occlusion effects to the sound source, so that the virtual sound source, when rendered directly, sounds the same or substantially the same as the sound source when rendered with diffraction and / or occlusion processing. The virtual sound source location may have the same orientation (e.g., azimuth, or azimuth and elevation) as a first location (diffraction corner) on or near the acoustic diffraction path where the diffraction path changes direction and the straight line from the diffraction corner voxel to the listener voxel location is unobstructed, as viewed from the listener position. The virtual sound source distance may correspond to the length of the diffraction path. The diffraction information may be expressed as C, defined below: vox and rin In brief, C vox denotes the position of the diffraction corner (e.g., voxel position), and r in denotes the diffraction path length.

[0020] In some embodiments, the representation of the three-dimensional audio scene may be a voxel-based representation. The representation of the three-dimensional audio scene may then include one or more indications of a cuboid volume within a voxel grid, and each such indication may include information about pairs of extreme-corner voxels that define the cuboid and about common voxel properties of the voxels within the cuboid volume. This allows for a more efficient voxel-based representation of the three-dimensional audio scene.

[0021] In some embodiments, the information about pairs of end-corner voxels of a cuboid volume may include an indication of the respective voxel indices assigned to the end-corner voxels, where voxels of the voxel-based audio scene representation may have uniquely assigned consecutive voxel indices, allowing for more efficient representation of voxel coordinates.

[0022] In some embodiments, the representation of the three-dimensional audio scene may be a voxel-based representation. The diffraction information may then comprise an indication of a position of a voxel located on or near the diffraction path and an indication of a length of the diffraction path. Furthermore, the indication of a position of a voxel located on or near the diffraction path may be an indication of a voxel index assigned to that voxel, and voxels of the voxel-based audio scene representation may have uniquely assigned consecutive voxel indices.

[0023] In some embodiments, the method may further comprise determining a (current) scene state based on the audio scene description and the listener position, which can then be used to determine whether pre-calculated diffraction information is available for the current configuration of the audio scene and the current listener position.

[0024] In some embodiments, the method may further comprise determining whether the current scene state corresponds to a known scene state from which pre-calculated diffraction information may be retrieved, for example from the bitstream (the input bitstream) or from storage (including in particular external storage such as shared / cloud storage).

[0025] In some embodiments, determining whether the current scene state corresponds to the known scene state may include determining a hash value based on the current scene state, which may further include comparing the determined hash value to the hash value of the known scene state.

[0026] In some embodiments, the method may further include, if it is determined that the current scene state corresponds to a known scene state, determining diffraction information by retrieving pre-computed diffraction information for the known scene state from the bitstream or storage (e.g., local memory, cache, external memory, shared memory, cloud implemented memory, etc.) In particular, the pre-computed diffraction information may be retrieved, at least in part, from an external data source or repository.

[0027] In some embodiments, the method may further include determining diffraction information using a pathfinding algorithm based on the sound source position, the listener position and the representation of the three-dimensional audio scene if it is determined that the current scene state does not correspond to the known scene state.

[0028] In some embodiments, the method may further comprise receiving a look-up table or look-up table entries from a bitstream or storage, the look-up table comprising a plurality of items of pre-calculated diffraction information, each associated with a respective known scene state. Thus, the LUT may be associated with or comprise a plurality of such data items. The known scene states may comprise a known audio scene description and a known listener position.

[0029] In some embodiments, the representation of the three-dimensional audio scene may be a voxel-based representation.

[0030] According to another aspect, a method of compressing an audio scene for three-dimensional audio rendering is provided. The method may include obtaining a voxelized representation of the audio scene, the voxelized representation including a plurality of voxels arranged in a voxel grid, each voxel having an associated voxel characteristic. The method may further include determining, among the voxels of the voxelized representation, a set of voxels that form a connected geometric region on the voxel grid, the voxels within the geometric region sharing a common voxel characteristic. The method may further include generating a representation of the audio scene based on the determined set of voxels.

[0031] In some embodiments, the geometric region may have a cuboid shape. The method may then further include determining at least a first bounding voxel and a second bounding voxel for the set of voxels from the plurality of voxels of the voxelized representation, where the first bounding voxel and the second bounding voxel may define the cuboid shape of the geometric region.

[0032] In some embodiments, the voxel characteristics of each voxel may include acoustic characteristics associated with that voxel and / or a set of audio rendering instructions assigned to that voxel. Additionally, the common voxel characteristics of voxels in a geometric region may include common acoustic characteristics associated with those voxels and / or a common set of audio rendering instructions assigned to those voxels.

[0033] In some embodiments, the method may further comprise determining, for the geometric region, at least one scene element parameter comprising one or more of a scene element identifier, an acoustic property identifier and / or an audio rendering instruction set identifier, and indices of corresponding first and second bounding voxels defining the geometric region, where a scene element is understood to be associated with a geometric region together with its corresponding voxel properties (e.g. material properties and / or rendering instructions) and, optionally, an identifier (e.g. scene element identifier).

[0034] In some embodiments, the method may further include applying entropy coding and / or lossy coding to at least one scene element parameter for the geometric region in a sequential (taking all data for processing) or progressive (taking a portion of the data) manner.

[0035] In some embodiments, the method may further include a step of outputting a bitstream comprising at least one scene element parameter for determining, based on the determined set of voxels, a set of voxels associated with a geometric domain for the compressed representation of the audio scene.

[0036] In some embodiments, the geometrical domain may relate to a scene element within an audio scene.

[0037] In some embodiments, the audio scene may comprise a larger scene represented by the determined set of voxels. The larger scene may comprise a set of sub-scenes. Each of the sub-scenes may correspond to a subset of the determined set of voxels. The method may then further comprise determining, within the determined set of voxels, a subset of voxels for a corresponding sub-scene.

[0038] In some embodiments, the method may further comprise applying interpolation of audio properties and renderer instructions of voxels in time and / or space, and / or applying filtering.

[0039] In some embodiments, the method may further include redefining voxel properties for a subset of the set of voxels associated with the scene sub-element within the geometric domain to overwrite the subset with the redefined voxel properties, where a scene sub-element is understood to relate to a scene element having a geometric domain that is contained (e.g., completely contained) within the geometric domain of another scene element.

[0040] In some embodiments, the method may further include determining a superset of voxels that includes the determined set of voxels. The determined set of voxels may be associated with a scene subelement within the geometric domain. The method may then further include assigning new voxel properties to the determined set of voxels and overwriting the voxel properties of the determined set of voxels with the new voxel properties.

[0041] In some embodiments, the method may further include determining a voxel size for representing the geometric region, the voxel size being based on a number of voxels along a scene dimension.

[0042] According to another aspect, there is provided an apparatus for processing audio scene information for audio rendering, the apparatus may include a processor and a memory coupled to the processor for storing instructions for the processor, the processor may be configured to perform all steps of the above aspects and methods according to these embodiments.

[0043] According to a further aspect, a computer program is described, which may include executable instructions for performing the methods or steps of the methods outlined throughout this disclosure when executed by a computing device (e.g., a processor).

[0044] According to another aspect, a computer-readable storage medium is described. The storage medium may store a computer program adapted for execution on a computing device (e.g., a processor) and, when executed on the computing device, to perform the methods or method steps outlined throughout this disclosure.

[0045] It should be noted that the methods and systems, including preferred embodiments thereof, as outlined in this disclosure may be used independently or in combination with other methods and systems disclosed in this document. Furthermore, all aspects of the methods and systems outlined in this disclosure may be combined in any manner. In particular, the features of the claims may be combined with each other in any manner.

[0046] It will be appreciated that apparatus features and method steps may be interchanged in many ways. In particular, as those skilled in the art will appreciate, details of the disclosed methods can be implemented by corresponding apparatus, and vice versa. Furthermore, it will be understood that any statements made above with respect to methods (and, e.g., steps thereof) equally apply to corresponding apparatus (and, e.g., blocks, stages, units thereof), and vice versa. [Brief description of the drawings]

[0047] The invention will now be described, by way of example only, with reference to the accompanying drawings, in which: [Figure 1] 2 illustrates generally an example of a processing chain for processing audio scene information for audio rendering. [Diagram 2] 1 illustrates a schematic of example diffraction paths for a source position and a listener position in a voxel-based 3D audio scene. [Diagram 3] 1 is a flowchart illustrating an example method for processing audio scene information for audio rendering, according to an embodiment of the present disclosure. [Figure 4] 4 is a flowchart illustrating an example of implementation details of the method of FIG. 3 according to an embodiment of the present disclosure. [Diagram 5] 2 illustrates a schematic diagram of an example of a processing chain for processing audio scene information for audio rendering according to an embodiment of the present disclosure. [Figure 6] 2 illustrates a schematic diagram of an example of a processing chain for processing audio scene information for audio rendering according to an embodiment of the present disclosure. [Figure 7] 2 illustrates a schematic diagram of an example of a processing chain for processing audio scene information for audio rendering according to an embodiment of the present disclosure. [Figure 8] FIG. 1 illustrates a diagram showing complexity measures as a function of time for different operational modes / implementations of processing audio scene information for audio rendering, according to an embodiment of the present disclosure. [Figure 9] 1 illustrates a schematic example of a possible use case for the technique according to an embodiment of the present disclosure. [Figure 10A] 1 illustrates a schematic diagram of an example of a portion of a voxel-based audio scene according to an embodiment of the present disclosure. [Figure 10B] 1 illustrates a schematic diagram of an example of a portion of a voxel-based audio scene according to an embodiment of the present disclosure. [Figure 10C] 1 illustrates a schematic diagram of an example of a portion of a voxel-based audio scene according to an embodiment of the present disclosure. [Figure 11] 1 illustrates a schematic example of a voxel-based audio scene to which embodiments of the present disclosure may be applied; [Figure 12] 1 is a block diagram that illustrates a schematic example of an apparatus for implementing a method according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0048] Exemplary embodiments of the present disclosure will now be described with reference to the accompanying drawings, in which the same elements may be indicated by the same reference numerals, and repeated descriptions may be omitted.

[0049] [Voxel-based audio scene representation] First, an overview of voxel-related concepts for the representation of audio scenes is given.

[0050] What are Voxels for Audio Rendering? A voxel is understood as a spatial volume that is assigned acoustic properties or audio rendering instructions.

[0051] What is the voxel size for audio rendering? The voxel size may be an encoder configuration parameter, which may be selected (manually or automatically) according to the scene geometry level of detail (e.g., in the range of 10 cm to 1 m).

[0052] How large an audio scene can it handle? A large audio scene does not necessarily result in a large number of voxels and high rendering complexity. For example, a large audio scene can be represented as follows: A set of independent sub-scenes (and a way to "teleport" between these representations without "restarting" the renderer) A set of scene updates (based on user position)

[0053] How can we deal with the discontinuity problem caused by voxel granularity? Any strong discontinuities in the sound level (and jumps in the direction of the diffraction signal) can be avoided by applying interpolation (eg, in time and space).

[0054] How to represent a voxel-based audio scene? Any voxel-based representation of an audio scene may include an indication of voxels that are not transparent voxels (e.g., occluder voxels), i.e., voxels through which sound cannot or cannot propagate freely, i.e., a representation of the occlusion geometry. This indication may be associated with an indication of the coordinates (e.g., center coordinates, corner coordinates, etc.) of the respective voxels. The coordinates of these voxels may be represented, for example, by a grid index. Furthermore, the voxel-based representation may include an indication of the material properties of the voxels that are not transparent voxels, such as absorption coefficients, reflection coefficients, etc. In addition to occluder voxels, the voxel-based representation may also indicate transparent voxels (e.g., air voxels), i.e., voxels through which sound can propagate, i.e., a representation of the sound propagation medium. Some implementations of a voxel-based representation of an audio scene may therefore include an indication of the respective material properties for each voxel within a predefined section of space (e.g., within a boundary surrounding the audio scene).

[0055] [Technology for processing audio scene information] 1 shows a schematic of a processing chain 100 that can be used to process audio scene information for audio rendering. In particular, the processing chain 100 can be used to convert voxel-related data into parameters and signals required for sonification (or audio rendering in general). The processing chain 100 may be implemented in software, hardware or a combination thereof. For example, the processing chain 100 may be implemented by a renderer / decoder coupled to an AR / VR / MR / XR device such as AR / VR / MR / XR goggles. Particular implementations may include game consoles, set-top boxes, personal computers, etc.

[0056] The processing chain receives an audio scene description 20 from a bitstream (or storage / memory) 10. The audio scene description 20 may contain a representation of a 3D audio scene and information about source positions of sound sources within the audio scene. The representation of the 3D audio scene may for example be voxel-based.

[0057] The processing chain 100 further receives an indication of a user position (listener position) 30 of a user (listener) within the audio scene. The audio scene description 20 and the user position 30 are provided to a diffraction direction calculation block (diffraction calculation block) 40 for determining (e.g., calculating) diffraction information. The diffraction information may relate to acoustic diffraction paths within the audio scene between the sound source position and the listener position. The diffraction information is then provided to a diffraction modeling tool 50 for applying diffraction modeling and optionally occlusion modeling based on the diffraction information. The occlusion modeling calculates attenuation gains for a straight line between the listener and the sound source. The diffraction modeling tool 50 may output auralized audio data (3DoF auralizer data), for example, including the position, orientation, and frequency-dependent gain of the object to be rendered. The output of the diffraction modeling tool may be further processed by other rendering stages, such as Doppler, directivity, distance attenuation, etc. In general, it can be said that the diffraction modeling tool 50 outputs diffraction information, as described in more detail below. The auralized audio data may then be used, for example, for audio playback.

[0058] In summary, a processing chain such as that shown in Fig. 1 may be used to convert voxel-related data into parameters for sonification and parameters for the signal. The diffraction direction calculation block 40 and the diffraction modeling tool 50 may be considered as non-limiting examples of a rendering tool. In general, the rendering tool may generate 3DoF auralizer data.

[0059] As described above, the scene description may include voxel matrices and associated coefficients (e.g., reflection coefficients, occlusion coefficients, absorption coefficients, transmission coefficients, etc.). These coefficients may indicate the material or material properties of each voxel. The rendering tools may include, for example, occlusion and diffraction modeling tools. The 3DoF auralizer data may include, for example, object positions, orientations, and frequency-dependent gains.

[0060] As mentioned above, a voxel-based representation of a 3D audio scene defines the psychoacoustically relevant geometric elements and the sound propagation medium. In some implementations, the scene description may provide information to a rendering tool using the following parameters / interfaces (e.g., the following agreed data formats, or agreed points of data exchange):

[0061] Scene Size: - Absolute units (e.g. meters) - Number of voxels and / or voxel size Scene anchor: - Coordinate anchors (to map absolute coordinates to voxel indices) - Scene anchors (to map subscenes to subsets of voxels) Scene Content Data: - Reference to material properties (e.g. transmission, reflection, etc. coefficients) that approximate the acoustic effects caused by occluders (sound obstacles) located in the corresponding volume -References to sound propagation medium properties (e.g. speed of sound, energy absorption, distance attenuation curves, etc.) that approximate the acoustic effects caused by the medium located in the corresponding volume - Rendering control parameters that describe the intended occlusion modeling effect For example, a "global" or "local" occluder type that determines the length (and shape) of the occlusion effect shadow behind this voxel. - Rendering control parameters that describe the intended sound diffraction modeling effect For example, voxel types that control / cause a change in sound direction (i.e. diffracted sound paths cannot penetrate this volume) -Content control parameters describing audio signal relevance and scene authoring For example, audio signal ID and / or signal gain determine which signals are perceptually relevant (rendered) at the corresponding volume. - Rendering control parameters that describe the intended reverberation modeling effect For example, the voxel type that controls the reverberation settings (e.g. RT60, DDR, RIR, etc.) Scene content update: - Referenced to update trigger events

[0062] All data can be made audio object dependent (to support content creator intent in flexible audio scene authoring).

[0063] 3DoF auralizer data may include the following information: - parameters for a set of audio objects (and HOAs) and associated signals Parameters include the metadata output of the rendering tool (i.e. position, orientation and gain to simulate the effects of occlusion, diffraction, early reflections, parameters for reverberation coefficients, IR, etc.) The associated signal represents the audio output of the rendering tool (i.e. the downmixed or duplicated audio signal). -Scene state identifier (i.e. metadata that allows mapping scene description and user input to 3DoF auralizer data)

[0064] 2 shows an example of a possible scene state and the diffraction paths for this scene state, it being understood that the scene state relates to or includes a listener position 210 and an audio scene description (including a representation of a three-dimensional audio scene and sound source positions 220).

[0065] The example of Fig. 2 relates to a voxel-based representation of a three-dimensional audio scene. This voxel-based representation includes "air" or empty voxels (i.e. voxels through which sound can propagate or transparent voxels) 230 and occluder voxels 240 (i.e. voxels through which sound cannot propagate or cannot propagate freely). Thus, occluder voxels may be understood to relate to voxels that are filled with a material other than air and can reflect, block or otherwise modify the propagation of sound. For the occluder voxels 240, the representation may further indicate the respective transmission, reflection and potentially absorption coefficients for the material properties of these voxels. These coefficients may be linked to the ID or index of these respective voxels in the voxel-based representation. In general, the voxel-based representation may define the psychoacoustically relevant geometric elements and sound propagation media in the audio scene.

[0066] The listener position 210 is the parameter L VOX and the sound source position 220 is represented by another parameter S VOX As shown by:

[0067] The diffraction path between the sound source position 220 and the listener position 210 may be determined using a pathfinding algorithm that takes as input the listener position 210, the sound source position 220, and a representation of the three-dimensional audio scene (or a two-dimensional representation derived therefrom, e.g., a 2D projection or a 2D matrix). For example, an algorithm for determining diffraction information may take as input the listener position 210, the sound source position 220, and a representation of the three-dimensional audio scene, and VOX and the position of the diffraction corner 250, denoted by in For example, the diffraction information may be determined based on: [C VOX ,r in ]=DiffractionDirectionCalculation(L VOX ,S VOX ,VoxDataDiffractionMap) where DiffractionDirectionCalculation is an algorithm for determining diffraction information (the "pathfinding algorithm"), and VoxDataDiffractionMap is a voxel-based representation of the 3D audio scene or a processed version thereof (e.g., a 2D projection or a 2D matrix derived from it). VOX It will be understood that denotes the coordinates of the diffraction corners (eg, the coordinates, voxel / grid coordinates or voxel / grid indexes of the respective voxels that include the diffraction corners).

[0068] Here, DiffractionDirectionCalculation may include any viable pathfinding algorithm, such as, for example, the Fast Traversal Algorithm for Ray Tracing (see Amanatides, J. and A. Woo, A Fast Voxel Traversal Algorithm for Ray Tracing. Proceedings of EuroGraphics, 1987. 87.) and the JPS algorithm (see Harabor, DD and A. Grastien, Online Graph Pruning for Pathfinding On Grid Maps. Proceedings of the Twenty-Fifth AAAI Conference on Artificial Intelligence, 2011). Furthermore, using the voxel-based scene representation, a 3D pathfinding algorithm may be directly applied to obtain the shortest path between the sound source position 220 and the listener position 210. Alternatively, a 2D pathfinding algorithm may be applied for this task using an appropriate 2D projection plane of the 3D voxel-based scene representation. For indoor (e.g., multi-room) sound simulation, the corresponding 2D projection plane may be similar to a floor plan describing the "sound propagation path topology". For outdoor sound simulation scenarios, it may be important to consider a second (e.g., perpendicular) 2D projection plane to account for the diffraction paths of the sound beyond obstacles or occlusion structures. The pathfinding technique remains the same for all projection planes, but its application provides additional paths that can be used for diffraction modeling.

[0069] The pathfinding algorithm is assumed to output a diffracted path consisting of multiple straight path segments (line segments) that connect the sound source position 220 to the listener position and are linked sequentially end-to-end. Each transition from one path segment to another is associated with a change in the direction of the diffracted path.

[0070] According to the algorithm for determining the diffraction information, the diffraction corner C VOX is a set of voxels C that are on or near the diffraction path and represent corner voxels on the diffraction map (shown by a voxel-based representation). set voxels adjacent to a corner voxel (in the VOX is the set of voxels that form the diffraction path (P set ), the route (P set ) to change the direction of (C set voxels that are close to the "visible" (from the listener position Lc) corner voxel (belonging to P set ) is selected.

[0071] In general, a diffraction path algorithm may be said to determine diffraction information about sound diffraction paths in an audio scene between a sound source position and a listener position.

[0072] This diffraction information may be sufficient for the renderer to recover / determine a virtual source position for a virtual sound source that encapsulates the effects of the acoustic diffraction effect. This is called the diffraction corner C VOX and the diffraction path length r in For example, the virtual sound source position may be reconstructed by calculating the direction (e.g., azimuth, or azimuth and elevation) of the diffraction corner as seen from the listener position. Using this direction, the path length r in The virtual sound source position can be determined by regarding this as the virtual sound source distance to the listener position.

[0073] Note that the diffraction information can be represented in different ways. One option is to express the diffraction information in terms of the path length r as described above. in and diffraction corner C VOX The diffraction information includes / stores coordinates (e.g., grid coordinates, etc.).

[0074] Based on the above, the following data elements may be defined:

[0075] An example of a scene state N1 may be represented by: N1={L VOX ,S VOX ,VoxDataDiffractionMap} That is, the listener position L in the audio scene VOX , sound source position S VOX , and may relate to or include a voxel-based representation (e.g., VoxDataDiffractionMap).

[0076] The scene state identifier for scene state N1 may be defined as follows: SceneStateIdentifier=HASH(N1) where HASH is a hash function that generates a hash value of the scene state N1, e.g., mapping scene states to fixed size values. In general, a scene state identifier may be said to indicate or identify a particular scene state.

[0077] Further, an example of the diffraction information N2 may be represented by the following: N2={C VOX ,r in} where r in is the path length of the diffraction path, and C VOX denotes the location (eg, voxel position) of the diffraction corner, as described above.

[0078] The quantized version of the diffraction information N2 may be denoted by N3. N3=voxSceneDiffractionPreComputedPathData(N1) N3(~=N2)=DiffractionDirectionCalculation(N1) Here, voxSceneDiffractionPreComputedPathData() is a bitstream syntax that parses the bitstream and extracts pre-calculated (stored and quantized) diffraction information (e.g., generated by processing chain 500 in FIG. 6 ), and DiffractionDirectionCalculation() refers to a function that performs online calculation of diffraction information, which may be implemented, for example, in diffraction direction calculation block 40 in FIGS. 5 and 7 .

[0079] User's position voxel coordinate L VOX is fixed, so for example, C VOX and r in Such diffraction information may be considered as related to the 3DOF auralizer data.

[0080] An example of the syntax element voxSceneDiffractionPreComputedPathData() according to the MPEG-I standard is given by Table 1. This voxel payload data structure may have the following elements:

[0081] numberOfVoxDiffractionPathData This element represents the number of pre-calculated diffraction path data sets.

[0082] voxDiffractionPathStartVoxelPacked This element is the path start voxel of a pre-calculated diffraction path (e.g., S in Figure 2). VOX Or L VOX ) voxDiffractionPathStartVoxel, which may be, for example, a 2D position on the diffraction map indicating the start position of the diffraction path.

[0083] voxDiffractionPathEndVoxelPacked This element is the path end voxel of the pre-calculated diffraction path (e.g., S in Figure 2). VOX Or L VOX ) voxDiffractionPathEndVoxel may be, for example, a 2D position on the diffraction map indicating the end position of the diffraction path.

[0084] voxDiffractionPathDataExistFlag This factor indicates whether a diffraction path exists or not.

[0085] voxDiffractionSourceDirectionPacked This element represents the packed form of the variable voxDiffractionSourceDirection that indicates the voxel index of the voxel for which to determine the diffracted source azimuth value. VOX may also correspond to

[0086] voxDiffractionPathLength This element represents the diffraction path length on the diffraction map 2D matrix. This can be, for example, in may also correspond to

[0087] Note: voxSceneDimensions Number of voxels for each scene dimension escapedValue() This element implements a generic method for transmitting integer values ​​using a variable number of bits. It features a two-level escape mechanism that allows extending the representable range of values ​​by the successive transmission of further bits. The syntax of escapedValue() shall be as defined in ISO / IEC 23003-3. [Table 1]

[0088] The variables extracted from voxSceneDiffractionPreComputedPathData() (e.g., as shown in Figure 6) are further processed to output diffraction information N3.

[0089] A technical benefit and advantage of the techniques of this disclosure is that a scene state identifier or other information derived from a scene state may be used to avoid the application of diffraction modeling or rendering tools if corresponding processing has already been performed for this scene state and diffraction information or 3DoF auralizer data is available. In this scenario, the renderer: -Reusing previously calculated (pre-calculated) data, or -Applying data calculated by another renderer allows access to diffraction information / 3DoF auralizer data (for known scene conditions) without applying rendering tools.

[0090] Thus, the technical benefit and advantage is that the techniques disclosed herein relate to lossless functionality for low complexity modes (complexity vs. bitrate).

[0091] To fully implement such a scheme, the present disclosure proposes to provide an interface in a processing chain for processing audio scene information for audio rendering (e.g. in a decoder / renderer) to provide / output diffraction information for later use or for use by a different decoder / renderer. This interface is understood to be a data interface for outputting data in a predefined format, in particular to enable consistent reuse by other decoders / renderers. The interface may be implemented and / or utilized in any combination of software and hardware. In particular, this may relate to providing / outputting a data element that includes the diffraction information and information about the scene state, such as for example a scene state identifier. The data element may for example have a predefined format with predefined data fields. Using this interface, the processing chain can provide the calculated diffraction information or 3DoF auralizer data together with the scene state identifier to other decoders / renderers and / or store it for later reuse.

[0092] Example 1: If the decoder / renderer has obtained diffraction information (e.g., diffraction path) for a given user position (listener position), it can reuse it until the user leaves the corresponding voxel volume (or the scene description is updated).

[0093] Example 2: If the computed diffraction information corresponds to a scene state that is unknown to other decoders, the other decoders may reuse the diffraction information and avoid running their own diffraction modeling or rendering tools.

[0094] The exchange and sharing of diffraction information between different decoders can be done using a database that can be included in the bitstream (eg, accessed via an application request).

[0095] 3 is a flow chart illustrating an example of a method 300 for processing audio scene information for audio rendering, according to an embodiment of the present disclosure. The method 300 may be implemented in software, hardware, or a combination thereof. For example, the processing chain 100 may be implemented by a renderer / decoder coupled to an AR / VR / MR / XR device, such as an AR / VR / MR / XR goggle. Particular implementations may include a game console, a set-top box, a personal computer, etc.

[0096] The method 300 includes steps S310 to S350 that may be executed, for example, by a decoder / renderer. These steps may be executed, for example, every time a scene state changes. Since the scene state is understood as relating to or including the listener position 210 and the audio scene description (including the representation of the three-dimensional audio scene and the sound source positions 220), implemented, for example, by the above-mentioned scene state N1, a change in the scene state may relate to one or more of a change in the listener position 210, a change in the sound source positions and a change in the (representation of) the three-dimensional audio scene. Alternatively, steps S310 to S350 may be executed for each of multiple processing cycles of the decoder / renderer. However, if the audio scene description has not changed, step S310 may be omitted. It is also understood that steps S310 to S350 do not have to be executed in the order shown in FIG. 3.

[0097] In step S310, an audio scene description is received. The audio scene description includes a representation of a three-dimensional audio scene and information about source positions of sound sources within the audio scene. For example, the audio scene description may be represented by, for example, the elements S defined above. VOX and VoxDataDiffractionMap.

[0098] In step S320, information of the listener's position in the audio scene is received. The listener position can be, for example, determined by the element L defined above. VOX may also correspond to

[0099] In step S330, diffraction information is obtained regarding sound diffraction paths in the audio scene between the sound source position and the listener position. The obtained diffraction information may indicate a virtual source position of a virtual sound source. For example, the virtual sound source position may be located at a diffraction corner C VOX The virtual source distance may have the same direction (e.g., azimuth, or azimuth and elevation) as the diffraction path length r in Therefore, the diffraction information may correspond to the C VOX and r in The instruction may include:

[0100] In step S340, audio rendering is performed for the sound source based on the diffraction information, which may include, for example, diffraction modeling.

[0101] To this end, a virtual source position of the virtual sound source may be determined based on the diffraction information. The virtual sound source may be a source that encapsulates the effects of acoustic diffraction between the source position and the listener position in the three-dimensional audio scene. For example, the virtual sound source position may be determined based on the diffraction information. Diffraction corner C as seen from the listener position VOX Determining the direction (e.g., azimuth, or azimuth and elevation) of Using the determined direction as the virtual sound source direction of the virtual sound source as seen from the listener position; and Diffraction path length r in of ○ The virtual source distance of the virtual sound source from the listener position, or Virtual source gain compensation derived from diffraction and direct path lengths Use as By C VOX and r in It may be determined based on.

[0102] Audio rendering may then include, for example, rendering the virtual sound sources at the virtual sound source positions.

[0103] In step S350, a representation of the diffraction information is output. For example, outputting the representation of the diffraction information may include outputting a data element that includes the diffraction information and information about the scene state. The scene state may be represented by an audio scene description (e.g., S VOX and VoxDataDiffractionMap) and the listener position (e.g., L VOX ) may also be included.

[0104] The output may be provided to a look up table (LUT). The LUT contains as its entries different items of diffraction information indexed with information about the respective scene state (e.g. indexed with the respective scene state identifier). This LUT may be said to contain diffraction information and information about the scene state. The LUT may be stored and / or provided for later retrieval, e.g. by application request, e.g. by other decoders, from the bitstream or from shared storage (e.g. cloud or server based). A hash value of the scene state or the scene state identifier can be used to retrieve the actual desired entry from the LUT.

[0105] Further, the representation of the diffraction information may be output to a bitstream (e.g., an output bitstream) and / or storage (e.g., memory, cache, file, etc.). The storage may be local or shared (e.g., cloud-based). In general, the representation of the diffraction information may be output to any suitable medium for storing digital or computer-related information. The output may be directed at least in part to an external or shared data source or data repository.

[0106] In some implementations, a representation of the diffraction information may be output as part of the voxSceneDiffractionPreComputedPathData() syntax element according to ISO / IEC 23090-4 (Coded representation of immersive media - Part 4: MPEG-I immersive audio, https: / / www.iso.org / standard / 84711.html) or any future standard derived therefrom.

[0107] For example, the voxSceneDiffractionMap() syntax element may be given by Table 2. [Table 2]

[0108] voxSceneDiffractionMap() provides a compact representation of a 2D diffraction map (VoxDataDiffractionMap), similar to the 3D representation used for voxel-based 3D audio scenes.

[0109] A MapElement is defined by two points (x,y indices) on the diffraction map and a corresponding value. The two points span a rectangle and all covered grid cells are assigned the value voxDiffractionMapValue.

[0110] The bitstream element numberOfVoxDiffractionMapElements indicates the number of MapElements.

[0111] The bitstream element voxDiffractionMapValue indicates a binary value that controls the pathfinding algorithm. This value is useful because it indicates whether a path can pass through a grid cell. This value is defined for every entry on the diffraction map.

[0112] The bitstream element voxDiffractionMapPosPackedS indicates a packed representation of the two indices of the starting grid cell of a MapElement, which may be an array indicating the collection of all starting grid cells.

[0113] The bitstream element voxDiffractionMapPosPackedE is a packed representation of the two indices of the end grid cells of a MapElement. It may also be an array indicating the collection of all end grid cells.

[0114] Both voxDiffractionMapPosPackedS and voxDiffractionMapPosPackedE are useful because they allow a compact representation of the data where a single voxDiffractionMapValue is used for all grid cells between these two variables.

[0115] 4 is a flow chart illustrating an example of a method 400 including steps that may be executed to implement the steps of the method 300. The method 400 includes steps S410 to S460, among which steps S410 to S450 may implement step 330 of the method 300. Furthermore, step S460 may correspond to step S350.

[0116] In step S410, the current scene state is determined based on the audio scene description and the listener position.

[0117] In step S420, it is determined whether the current scene state corresponds to a known scene state for which pre-calculated diffraction information is available (e.g., can be retrieved). The pre-calculated diffraction information may be retrieved, for example, from a bitstream (the input bitstream) or storage (including external or shared storage, among others). Determining whether the current scene state corresponds to a known scene state may include determining a hash value based on the current scene state. This may further include comparing the hash value of the current scene state against a hash value of a known (e.g., previously encountered) scene state.

[0118] If it is determined that the current scene state corresponds to the known scene state (YES in step S430), the method proceeds to step S440.

[0119] In step S440, the diffraction information is determined by extracting pre-calculated diffraction information for known scene states from the bitstream or storage, which may relate to local storage (e.g., memory, cache, file, etc.) or shared storage (e.g., cloud storage, server storage).

[0120] Extracting the pre-calculated diffraction information for the known scene state may include receiving a look-up table or look-up table entries from a bitstream (input bitstream) or storage. The look-up table may be considered as a representation of the diffraction information. It may include a number of items of pre-calculated diffraction information, each associated with a respective known scene state. The pre-calculated diffraction information and the associated known scene state may correspond to the data elements mentioned above. The known scene state may include or indicate a known audio scene description and a known listener position.

[0121] Selecting the relevant entry of the received lookup table, or selecting the relevant entry to be received (if only that entry and not all of the lookup table is received), may involve using a hash value, as described above.

[0122] On the other hand, if it is determined that the current scene state does not correspond to the known scene state (NO in step S430), the method proceeds to step S450.

[0123] In step S450, the diffraction information is determined using a pathfinding algorithm based on the sound source positions, the listener position and the representation of the three-dimensional audio scene, which may be done according to the procedure described above with reference to FIG.

[0124] In step S460, the diffraction information obtained via step S440 or step S450 is output. This step may correspond to step S350 above.

[0125] In summary, the proposed method may include (among other things): Check whether pre-computed diffraction path information (i.e. pre-calculated diffraction information) can be retrieved from the cache and re-applied for the current scene state and listener position · Check whether pre-computed diffraction path information for the current scene state (i.e. pre-computed diffraction information) can be retrieved from the memory cache or the bitstream and re-applied.

[0126] Here, the scene state is the input parameter L of the function DiffractionDirectionCalculation() that contains the pathfinding algorithm. VOX , S VOX , VoxDataDiffractionMap, Voxel C VOX is defined via a selection and diffraction path length estimation step.

[0127] The diffraction path information is output as the output parameter C VOX , r in This diffraction path information, if available, can be obtained directly from the bitstream syntax voxSceneDiffractionPreComputedPathData() for the corresponding scene state to avoid calling the function DiffractionDirectionCalculation().

[0128] Diffraction path information C VOX , r in is the current scene state L VOX、 S VOX、 When obtained for a VoxDataDiffractionMap, this information can be cached in memory (or provided externally to the renderer) for later reuse by the renderer (or other renderer instances).

[0129] In other words, “diffraction path finding” according to the present disclosure (eg, as embodied by method 300 and / or method 400) may involve the following process. -Check whether pre-computed diffraction path information for the current scene state can be retrieved from the memory cache or the bitstream and re-applied. The scene state is the input parameter L of the function DiffractionDirectionCalculation() that contains the pathfinding algorithm. VOX , S VOX , VoxDataDiffractionMap, Voxel C VOX is defined via a selection and diffraction path length estimation step. [C VOX ,r in ]=DiffractionDirectionCalculation(L VOX ,S VOX ,VoxDataDiffractionMap) The diffraction path information is output as the output parameter C VOX , r inThis diffraction path information, if available, can be obtained directly from the bitstream syntax voxSceneDiffractionPreComputedPathData() for the corresponding scene state to avoid calling the function DiffractionDirectionCalculation(). -Diffraction path information C VOX , r in is the current scene state L VOX、 S VOX、 When obtained for a VoxDataDiffractionMap, this information can be cached in memory (or provided externally to the renderer) for later reuse by the renderer (or other renderer instances).

[0130] In the above, the bitstream syntax definition may be written in function() style in the MPEG standard document. It defines how to read / parse data (bitstream elements) from the bitstream. In this case, it is used to obtain the variables / information needed to recover the diffraction path information.

[0131] 5, 6 and 7 show an example of a processing chain 500 according to the above that can be used to process audio scene information for audio rendering. In particular, the processing chain 500 can be used to convert voxel-related data into parameters and signals required for auralization.

[0132] Figure 5 relates to the case where the current scene state is an unknown scene state: unlike the processing chain 100 of Figure 1, the bitstream / memory 510 further comprises data elements containing the diffraction information and the associated scene state, for example in the form of a look-up table, as described above.

[0133] Similar to processing chain 100, processing chain 500 receives an audio scene description 20 from a bitstream (or storage / memory) 510. Processing chain 500 further receives an indication of a user position (listener position) 30 of a user (listener) within the audio scene.

[0134] The diffraction direction calculation block (diffraction calculation block) 40 for determining (e.g., calculating) diffraction information, and the diffraction modeling tool 50 for applying diffraction modeling and optionally occlusion modeling based on the diffraction information may be the same as for processing chain 100.

[0135] However, unlike the processing chain 100 of FIG. 1, the audio scene description 20 and the listener position 30 are used to determine a scene state 515 or a scene state identifier. This scene state 515 (e.g., scene state N1 defined above) or a scene state identifier (e.g., HASH(N1)) is provided / input to a scene state analysis block 520 which determines whether the current scene state 515 corresponds (YES in block 535) to a known scene state 530 or not (NO in block 535). In this example, the current scene state 515 does not correspond to a known scene state (i.e., the current scene state 515 is an unknown scene state). Thus, the audio scene description 20 and the listener position 30 are input to a diffraction direction calculation block 40 in the same way as for the processing chain 100 to generate diffraction information 550. The diffraction information 550 is then used for rendering / diffraction modeling as in the processing chain 100. In addition, however, the diffraction information 550 (e.g. the diffraction information N2 as defined above or its quantized version N3) is output via an interface for later reuse by a renderer or other (external) rendering instances. In particular, the diffraction information 550 may be output together with the corresponding scene state to the bitstream (or memory / storage) 510 via an output interface 555.

[0136] FIG. 6 relates to the case where the current scene state 515 is a known scene state.

[0137] Similarly, the current scene state 515 is provided / input to the scene state analysis block 520 to determine whether the current scene state 515 corresponds to the known scene state 530. In this example, the current scene state 515 corresponds to the known scene state. Thus, instead of inputting the audio scene description 20 and the listener position 30 to the diffraction direction calculation block 40 to calculate / generate the diffraction information, the diffraction information is extracted / received from the bitstream (or storage / memory) 510 as described above (e.g. via step S450 of the method 400). Furthermore, even if the diffraction information is not calculated locally, it may be output to the bitstream (or memory / storage) 510, as in the case of FIG. 5. The reason for this is that if the diffraction information is obtained from one source (e.g. in the bitstream or storage), it can be made available to the respective other source in this way.

[0138] FIG. 7 shows the complete processing chain 500 including the data paths for both known scene states 515 and unknown scene states.

[0139] FIG. 8 illustrates the complexity measures of different implementations of processing audio scene information or audio rendering as a function of time, assuming a simple maze as audio scene. It is further assumed that the user moves randomly through the maze, and thus revisits previously visited positions. Graph 810 relates to the case where no precomputed diffraction information is available at all (e.g. no diffraction information is provided in the bitstream and memory / cache is disabled). In this case, the computational load of the renderer is substantially constant and relatively high. Graph 820 relates to the case where precomputed diffraction information is locally available (e.g. no diffraction information is provided in the bitstream and local memory / cache is enabled). In this case, the processing load of the renderer decays over time, as more items of diffraction information are accumulated locally. In other words, the encountered scene states are more and more related to the (locally) known scene states. Finally, graph 830 relates to the case where precomputed diffraction information is provided externally (e.g. complete diffraction information is provided in the bitstream). In this case, the computational load on the renderer is always low since a significant portion of the scene state relates to a known scene state and the diffraction information can be retrieved externally (e.g., from the bitstream or by request from external / shared storage) without local computation.

[0140] FIG. 9 shows a schematic example of a possible use case for the technique according to an embodiment of the present disclosure. Two listeners (users) A and B are shown at different positions in an audio scene (e.g., a house with different areas and levels). Users A and B may be users exploring a VR environment including the audio scene individually or jointly, e.g., as part of a game, a virtual tour, etc. Users exploring a common VR environment may be performing, for example, social VR. By having different listener positions in the audio scene, users A and B generate different rendering results and different diffraction information. The present disclosure foresees each user (or their respective devices / decoders / renderers) making their calculated diffraction information available to the other users. When user B enters an area of ​​the audio scene where user A was previously present, user B can benefit from user A's pre-calculated diffraction information and vice versa. For example, user A's diffraction information may be made available to user B via a LUT that indexes different items of diffraction information with corresponding scene states or scene state identifiers. By exchanging diffraction information between different devices / decoders / renderers, depending on these movement patterns in the audio scene, the computational load of both users' devices / decoders / renderers can be reduced.

[0141] Furthermore, since users (listeners) tend to behave in the same way, diffraction information (diffraction data) is accumulated, especially for relevant (e.g., frequently occurring) scene states. This is very difficult to achieve with encoder-side pre-computation of diffraction information, since the encoder does not have access to the actual listener positions and can therefore only assume these. Furthermore, the use of data storage (e.g., physical / shared storage or bitstream bandwidth) is much less efficient for encoder-side pre-computation in this case, due to the part of the pre-computed diffraction information that pertains to irrelevant or less relevant scene states.

[0142] For example, the proposed features and techniques can create a LUT that corresponds to the user's actual 6DoF behavior (not the expected behavior on the encoder side) and may therefore be said to relate to smart user-oriented LUT creation.

[0143] [Efficient Voxel-Based Representation of Audio Scenes] Current representation formats for voxel-based scenes include, for example, *.vox, *.binvox, etc.

[0144] The present disclosure provides the following compression technique, for example for the MPEG-I Audio standard: A set of voxels having the same acoustic properties (e.g., the same material properties) or the same audio rendering instructions can be identified by two points forming a rectangular region on the voxel grid. All voxels in this rectangular region share the acoustic properties or audio rendering instruction set assigned to the corresponding two points as follows:

[0145] That is, the scene geometry is determined by a set of such point pairs (as an example of a representation of a geometric domain), e.g. <voxbox id="V_ID" material="P_ID" Point_S="X1 Y1 Z1" Point_E="X2 Y2 Z2" / > where V_ID is a rectangular voxel block element identifier, P_ID is an acoustic property or audio rendering instruction set identifier (e.g., occlusion, reflection, RT60 data), and X1, Y1, Z1 and X2, Y2, Z2 are the grid indices of the corresponding two points that define the rectangular voxel block element. Thus,<VoxBox id, material, Point_S, Point_E / > may correspond to a scene element as defined above, i.e. a geometric region (e.g. defined by a pair of points) together with its voxel properties (e.g. P_ID) and optionally its identifier (e.g. V_ID).

[0146] The voxel size may be determined by the number of voxels as follows: <voxsize size="N" / > where N is the number of voxels along the first longest scene dimension.

[0147] Thus, a voxel-based representation of an audio scene according to an embodiment of the present disclosure may include representations or indications of one or more cuboid geometric regions (cuboid spatial regions, cuboid volumes) that have identical (i.e., the same, common) acoustic properties (e.g., materials, absorption coefficients, reflectance coefficients, etc.) or identical rendering instructions. The acoustic properties or rendering instructions for a given voxel may be non-limiting examples of voxel properties of a given voxel. Representations of indications of cuboid geometric regions may, for example, be related to scene elements. It is understood that the cuboid regions each contain two or more voxels and are each significant in the sense that they consist of a connected (i.e., contiguous) set of voxels.

[0148] The shape of each of these geometric regions can be defined by first and second boundary voxels (e.g., pairs of points as described above). These first and second boundary voxels may be associated with the corners of a rectangular prism's diameter (end corner voxels), such as, for example, the end corner voxel with the smallest x, y, z coordinate values ​​or coordinate index, and the end corner voxel with the largest x, y, z coordinate values ​​or coordinate index. Other choices of end corner voxels in the diametric direction are possible as well.

[0149] Further, in the voxel-based representation, each geometric region may be represented by at least an indication of first and second bounding voxels and an indication of a common voxel characteristic of the voxels within the geometric region. Further, the representation of the geometric region may include an identifier (ID) for the geometric region.

[0150] Figure 10A shows an example of a geometric region 1002 within a voxel grid 1007. The geometric region 1002 includes a number of voxels 1003 that form a rectangular parallelepiped. The shape or size of the geometric region can be represented by first and second bounding voxels 1004-1, 1004-2, such as end corners or end corner voxels, which in this example correspond to the lower left front corner and the upper right rear corner, respectively. Figure 10B shows a side view of the geometric region 1002 as seen from the front right of Figure 10A.

[0151] For the proposed compression technique, each subsequent pair of points (i.e., each subsequent geometric region) may redefine the voxel properties in the corresponding cuboid region. That is, the voxel properties of the subsequent geometric region may overwrite the previously assigned voxel properties for the voxels of the subsequent geometric region. In one implementation, a smaller geometric region that is fully contained within a larger geometric region may redefine or overwrite the voxel properties of the voxels in the smaller geometric region with the voxel properties of the smaller geometric region, where it is understood that the corresponding voxel properties are overwritten and other voxel properties are maintained. For example, if a subsequent geometric region defines acoustic properties for its voxels, these acoustic properties are used to overwrite the acoustic properties defined for the voxels of the previous geometric region, but the rendering instructions of any of the voxels of the previous geometric region are maintained.

[0152] As noted above, the ordering between the geometric regions may be derived, for example, from whether the geometric regions are completely contained within one another, from the order of representations or indications of the geometric regions in the bitstream, or from a predefined order that references identifiers (IDs) of the geometric regions.

[0153] FIG. 10C illustrates diagrammatically how voxel properties assigned to a first geometric region defined by end corner voxels 1004-1, 1004-2 may be overwritten by voxel properties of a second geometric region 1002.

[0154] 11 shows a schematic example of a voxel-based description of a three-dimensional audio scene that includes multiple cuboid volumes of common voxel properties, potentially with smaller cuboid sub-volumes that redefine the voxel properties. The audio scene may include voxels 1101 that are local sound occluders that occlude the interior of a given acoustic environment of the audio scene. The audio scene may further include voxels 1102 that are local sound occluders that separate the acoustic environments from each other.

[0155] [Voxel coordinate / index representation] The following efficient representation of the voxel index may be applicable for example to the transmission or storage of both the voxel grid and the diffraction map (VoxDataDiffractionMap) entries. This may substitute for any fixed-length representation of the voxel index (voxel coordinates).

[0156] The following steps may be performed in the context of the proposed representation.

[0157] Step 1: Determine the amount of bits (i.e., number, count) required for the current grid resolution / diffraction map dimension. For a 3D voxel grid and a 2D diffraction map, these numbers NbitsVox and NbitsMap may be determined, for example, as follows, respectively: NbitsVox=ceil(log2(L*W*H-1)) NbitsMap=ceil(log2(L*W-1)) where L, W, H (length, width, height) are the dimensions of the voxel grid and the diffraction map. These values ​​may be different for the voxel grid and the diffraction map.

[0158] Step 1 may be applied on both the encoder and decoder sides.

[0159] Step 2: The voxel index (x,y,z) and the diffraction map index (x,y) are mapped to a packed representation index (Idx) and encoded using Nbits_vox and Nbits_map bits, respectively. In one embodiment, (x,y,z) are zero-based and the packed representation index may range from 0 to L*W*H-1 for voxels and 0 to L*W-1 for diffraction maps. The mapping from index (x,y,z) to packed representation index may be, for example, as follows: Idx(x,y,z)=(x-1)+((y-1)*L)+((z-1)*L*W)

[0160] Step 2 may be performed only on the encoder side.

[0161] In the above, a packed representation index is an index that can uniquely identify a voxel in a voxel grid or a diffraction map. In other words, voxels in a voxel grid may be assigned unique consecutive indices such that each voxel in the voxel grid can be uniquely identified by a single integer. Thus, a packed representation index may be used to indicate either a voxel location in a voxel grid or in a two-dimensional map. In particular, a packed representation index may be used to indicate any voxel location referred to throughout this disclosure.

[0162] The assignment of unique indices to the voxels may follow a predefined pattern, for example, the voxel grid may be scanned / traversed in x, y and z directions in that order to successively assign a unique index to each voxel.

[0163] The mapping from the packed representation indices to the voxel and diffraction map indices may be done, for example, as follows: x=floor(Idx)%L+1 y=floor(Idx / L)%W+1 z=floor(Idx / L / W)%H+1 where % denotes the modulo operator.

[0164] In accordance with the above, the bitstream payload element voxSceneDiffractionPreComputedPathData() given in Table 1 may use the following pseudocode: [Table 3] Here, voxDiffractionMapPosPackedS and voxDiffractionMapPosPackedE indicate packed representation indexes.

[0165] [Entropy coding] Entropy coding methods can be applied to sequences of integers representing acoustic features, voxel grid coordinates, voxel grid indices, and so on.

[0166] For example, the entropy coding method can be applied to the sequence of integers above (representing the acoustic properties or audio rendering instruction set references P_ID and grid indices X1, Y1, Z1 and X2, Y2, Z2) or a packed representation thereof. Furthermore, entropy coding may be applied to sequences where the integers are derived from the above representation of the diffraction path information.

[0167] Thus, it is a technical benefit and advantage of the voxel-based scene representation described herein that it is possible to create complex dynamic scenes and encode them efficiently.

[0168] [Device] Although methods and processing chains have been described above, it will be appreciated that the present disclosure also relates to apparatus (e.g., computing devices or devices having processing capabilities in general) for implementing these methods and processing chains (or techniques in general).

[0169] An example of such an apparatus 1200 is shown diagrammatically in Fig. 12. The apparatus 1200 includes a processor 1201 and a memory 1202 coupled to the processor 1201. The memory 1202 may store instructions for execution by the processor 1201. The processor 1201 may be adapted to implement processing chains described throughout this disclosure and / or to perform methods described throughout this disclosure (e.g., methods of processing audio scene information for audio rendering). The apparatus 1200 may receive inputs (e.g., audio scene descriptions, listener positions, etc.) and generate outputs (e.g., representations of diffraction information, etc.).

[0170] [interpretation] Aspects of the systems described herein may be implemented in a suitable computer-based sound processing network environment (e.g., a server or cloud environment) for processing digital or digitized audio files. Some of these systems may include one or more networks including any desired number of individual machines, including one or more routers (not shown) that function to buffer and route data transmitted between the computers. Such networks may be built on a variety of different network protocols and may be the Internet, a Wide Area Network (WAN), a Local Area Network (LAN), or any combination thereof.

[0171] One or more of the components, blocks, processes, or other functional components may be implemented through a computer program that controls the execution of a processor-based computing device of the system. It should also be noted that the various functions disclosed herein may be described as data and / or instructions embodied in various machine-readable or computer-readable media using any number of combinations of hardware, firmware, and / or in terms of their behavior, register transfers, logical components, and / or other characteristics. Computer-readable media on which such formatted data and / or instructions may be embodied include, but are not limited to, various forms of physical (non-transitory), non-volatile storage media, such as optical, magnetic, or semiconductor storage media.

[0172] In particular, it should be understood that the embodiments may include hardware, software, and electronic components or modules, which for purposes of discussion may be illustrated and described as if the majority of the components were implemented solely in hardware. However, those skilled in the art will recognize, based on reading this detailed description, that in at least one embodiment, the electronic-based aspects may be implemented in software (e.g., stored in a non-transitory computer-readable medium) executable by one or more electronic processors, such as a microprocessor and / or an application-specific integrated circuit ("ASIC"). Thus, it should be noted that multiple hardware and software-based devices and multiple different structural components may be utilized to implement the embodiments. For example, the computer-implemented neural network described herein may include one or more electronic processors, one or more computer-readable media modules, one or more input / output interfaces, and various connections (e.g., a system bus) connecting the various components.

[0173] Although one or more implementations have been described in terms of specific embodiments by way of example, it is to be understood that the one or more implementations are not limited to the disclosed embodiments. On the contrary, it is intended to cover various modifications and similar arrangements as would be apparent to those skilled in the art. Therefore, the scope of the appended claims should be accorded the broadest interpretation so as to encompass all such modifications and similar arrangements.

[0174] It is also to be understood that the phraseology and terminology used herein is for purposes of description and should not be considered as limiting. The use of "including," "comprising," or "having" and variations thereof is meant to encompass the items listed thereafter and equivalents thereof, as well as additional items. Unless otherwise specified or limited, the terms "mounted," "connected," "supported," and "coupled" and variations thereof are used broadly and include both direct and indirect mounting, connecting, supporting, and coupling.

[0175] [List of exemplary embodiments] Various aspects of the implementation of the present invention can be discerned from the following enumerated exemplary embodiments (EEE) which are not claimed.

[0176] EEE1. A method for processing audio scene information for audio rendering, comprising: receiving an audio scene description, said audio scene description comprising a representation of a three-dimensional audio scene and information regarding source positions of sound sources within said audio scene; receiving an indication of a listener's position within the audio scene; obtaining diffraction information about sound diffraction paths in the audio scene between the sound source position and the listener position; performing an audio rendering for the sound source based on the diffraction information; outputting a representation of the diffraction information; The method includes:

[0177] EEE2. The method of EEE1, wherein the step of outputting the representation of the diffraction information includes the step of outputting a data element comprising the diffraction information and information regarding a scene state, the scene state comprising the audio scene description and the listener position.

[0178] EEE3. The method of EEE1 or EEE2, wherein the representation of the diffraction information is output to a bitstream and / or storage.

[0179] EEE4. The method of any one of EEE1 to EEE3, wherein the diffraction information is output for subsequent reuse for audio rendering with the same rendering instance or for subsequent reuse by another rendering instance.

[0180] EEE5. The method of any one of EEE1 to EEE4, wherein the representation of the diffraction information is output as part of a voxSceneDiffractionPreComputedPathData() syntax element in accordance with ISO / IEC 23090-4 or any standard derived therefrom.

[0181] EEE6. The method of any one of EEE1 to EEE5, wherein the diffraction information indicates a virtual source position of a virtual sound source.

[0182] EEE7. The representation of the three-dimensional audio scene is a voxel-based representation; The method of any one of EEE1 to EEE6, wherein the representation of the three-dimensional audio scene includes one or more indications of rectangular volumes within a voxel grid, each such indication including information about pairs of end corner voxels of the rectangular volume and information about common voxel characteristics of voxels within the rectangular volume.

[0183] EEE8. The method according to EEE7, wherein the information relating to pairs of end corner voxels of the rectangular volume includes an indication of respective voxel indices assigned to the end corner voxels, and the voxels of the voxel-based audio scene representation have uniquely assigned consecutive voxel indices.

[0184] EEE9. The representation of the three-dimensional audio scene is a voxel-based representation; the diffraction information includes an indication of positions of voxels located on or near the diffraction path and an indication of a length of the diffraction path; The method of any one of EEE1 to EEE6, wherein the indication of the position of the voxel located on or near the diffraction path is an indication of a voxel index assigned to that voxel, the voxels of the voxel-based audio scene representation having uniquely assigned consecutive voxel indices.

[0185] EEE10. The method of any one of EEE1 to EEE9, further comprising determining a current scene state based on the audio scene description and the listener position.

[0186] EEE11. The method of EEE10, further comprising the step of determining whether the current scene state corresponds to a known scene state from which pre-calculated diffraction information can be retrieved.

[0187] EEE12. The method of EEE11, wherein determining whether the current scene state corresponds to a known scene state comprises determining a hash value based on the current scene state.

[0188] EEE13. The method of EEE11 or EEE12, further comprising the step of determining the diffraction information by extracting the pre-calculated diffraction information for the known scene state from a bitstream or storage if it is determined that the current scene state corresponds to a known scene state.

[0189] EEE14. The method of any one of EEE11 to EEE13, further comprising the step of determining the diffraction information using a pathfinding algorithm based on the sound source position, the listener position and the representation of the three-dimensional audio scene if it is determined that the current scene state does not correspond to a known scene state.

[0190] EEE15. The method of any one of EEE1 to EEE14, further comprising the step of receiving a lookup table or lookup table entries from a bitstream or storage, the lookup table comprising a plurality of items of pre-calculated diffraction information each associated with a respective known scene state, the known scene states comprising a known audio scene description and a known listener position.

[0191] EEE16. The method of any one of EEE1 to EEE15, wherein the representation of the three-dimensional audio scene is a voxel-based representation.

[0192] EEE17. A method for compressing an audio scene for three-dimensional audio rendering, comprising: obtaining a voxelized representation of the audio scene, the voxelized representation comprising a plurality of voxels arranged in a voxel grid, each voxel having associated voxel properties; determining, among the voxels of the voxelized representation, a set of voxels that form a connected geometric region on the voxel grid, the voxels within the geometric region sharing a common voxel characteristic; generating a representation of the audio scene based on the determined set of voxels; The method includes:

[0193] EEE18. The geometric region has a rectangular parallelepiped shape; The method as described in EEE17, further comprising determining at least a first and a second bounding voxel for the set of voxels from the plurality of voxels of the voxelized representation, the first and second bounding voxels defining the rectangular prism shape of the geometric region.

[0194] EEE19. The method of EEE17 or EEE18, wherein the voxel characteristics of each voxel include acoustic characteristics associated with that voxel and / or a set of audio rendering instructions assigned to that voxel, and the common voxel characteristics of the voxels in the geometric region include common acoustic characteristics associated with these voxels and / or a common set of audio rendering instructions assigned to these voxels.

[0195] EEE20. The method of any one of EEE17 to EEE19, further comprising determining, for the geometric region, at least one scene element parameter comprising one or more of a scene element identifier, an acoustic feature identifier and / or an audio rendering instruction set identifier, and indices of corresponding first and second bounding voxels defining the geometric region.

[0196] EEE21. The method of EEE20, further comprising the step of applying entropy coding to said at least one scene element parameter for said geometric region.

[0197] EEE22. The method according to EEE20 or EEE21, further comprising the step of outputting a bitstream including the at least one scene element parameter for determining, based on the determined set of voxels, a set of voxels associated with the geometric region for the compressed representation of the audio scene.

[0198] EEE23. The method of any one of EEE17 to EEE22, wherein the geometric domain relates to a scene element within the audio scene.

[0199] EEE24. The audio scene comprises a larger scene represented by the determined set of voxels, the larger scene comprising a set of sub-scenes, each of the sub-scenes corresponding to a subset of the determined set of voxels; The method of any one of EEE17 to EEE23, further comprising determining, among the determined set of voxels, a subset of voxels for the corresponding sub-scene.

[0200] EEE25. The method of any one of EEE17 to EEE24, further comprising applying interpolation of audio voxels in time and / or space.

[0201] EEE26. The method of any one of EEE17 to EEE25, further comprising the step of redefining voxel properties for the subset of the set of voxels associated with a scene subelement within the geometric region to overwrite the subset with the redefined voxel properties.

[0202] EEE27. Determining a superset of voxels including said determined set of voxels, said determined set of voxels being associated with scene subelements within said geometric domain; The method of any one of EEE17 to EEE26, further comprising the step of assigning new voxel properties to the determined set of voxels and overwriting the voxel properties of the determined set of voxels with the new voxel properties.

[0203] EEE28. The method of any one of EEE17 to EEE27, further comprising determining a voxel size for representing said geometric region, said voxel size being based on a number of voxels along a scene dimension of said geometric region.

[0204] EEE29. An apparatus including a processor and a memory coupled to the processor that stores instructions for the processor, 13. An apparatus, the processor adapted to perform a method according to any one of claims EE1 to EE28.

[0205] EEE30. A program comprising instructions which, when executed by a processor, cause the processor to carry out a method according to any one of EEE1 to EEE28.

[0206] A computer-readable storage medium storing a program according to EEE31.EEE30.

Claims

1. 1. A method for processing audio scene information for audio rendering, comprising the steps of: receiving an audio scene description, said audio scene description comprising a representation of a three-dimensional audio scene and information regarding source positions of sound sources within said audio scene; receiving an indication of a listener's position within the audio scene; obtaining diffraction information about sound diffraction paths in the audio scene between the sound source position and the listener position; performing an audio rendering for the sound source based on the diffraction information; outputting a representation of the diffraction information; The method includes:

2. 2. The method of claim 1 , wherein outputting the representation of the diffraction information comprises outputting a data element comprising the diffraction information and information about a scene state, the scene state comprising the audio scene description and the listener position.

3. The method of claim 1 or 2, wherein the representation of the diffraction information is output to a bitstream and / or storage.

4. The method of claim 1 , wherein the diffraction information is output for subsequent reuse for audio rendering with the same rendering instance, or for subsequent reuse with another rendering instance.

5. 5. The method of claim 1, wherein the representation of the diffraction information is output as part of a voxSceneDiffractionPreComputedPathData() syntax element in accordance with a standard set by ISO / IEC, such as the MPEG-I standard.

6. The method of claim 1 , wherein the diffraction information is indicative of a virtual source position of a virtual sound source.

7. the representation of the three-dimensional audio scene is a voxel-based representation; 7. A method according to claim 1, wherein the representation of the three-dimensional audio scene comprises one or more indications of a rectangular volume within a voxel grid, each such indication comprising information about pairs of end corner voxels of the rectangular volume and information about common voxel properties of voxels within the rectangular volume.

8. 8. The method of claim 7, wherein the information regarding the pairs of end-corner voxels of the rectangular volume includes an indication of respective voxel indices assigned to the end-corner voxels, and the voxels of the voxel-based audio scene representation have uniquely assigned consecutive voxel indices.

9. the representation of the three-dimensional audio scene is a voxel-based representation; the diffraction information includes an indication of positions of voxels located on or near the diffraction path and an indication of a length of the diffraction path; 7. The method of claim 1, wherein the indication of the position of the voxel located on or near the diffraction path is an indication of a voxel index assigned to that voxel, the voxels of the voxel-based audio scene representation having uniquely assigned consecutive voxel indices.

10. The method of any one of claims 1 to 9, further comprising the step of determining a current scene state based on the audio scene description and the listener position.

11. The method of claim 10 , further comprising the step of determining whether the current scene state corresponds to a known scene state from which pre-calculated diffraction information can be retrieved.

12. The method of claim 11 , wherein determining whether the current scene state corresponds to a known scene state comprises determining a hash value based on the current scene state.

13. 13. The method of claim 11 or 12, further comprising: if it is determined that the current scene state corresponds to a known scene state, determining the diffraction information by extracting the pre-calculated diffraction information for the known scene state from a bitstream or storage.

14. 14. The method of claim 11, further comprising the step of determining the diffraction information using a pathfinding algorithm based on the sound source position, the listener position and the representation of the three-dimensional audio scene if it is determined that the current scene state does not correspond to a known scene state.

15. 15. The method of claim 1, further comprising the step of receiving a look-up table or look-up table entries from a bitstream or storage, the look-up table comprising a plurality of items of pre-calculated diffraction information each associated with a respective known scene state, the known scene states comprising a known audio scene description and a known listener position.

16. The method of claim 1 , wherein the representation of the three-dimensional audio scene is a voxel-based representation including information about positions and material properties of a number of occluder voxels.

17. 1. A method of compressing an audio scene for three-dimensional audio rendering, comprising the steps of: obtaining a voxelized representation of the audio scene, the voxelized representation comprising a plurality of voxels arranged in a voxel grid, each voxel having associated voxel properties; determining, among the voxels of the voxelized representation, a set of voxels that form a connected geometric region on the voxel grid, the voxels within the geometric region sharing a common voxel characteristic; generating a representation of the audio scene based on the determined set of voxels; The method includes:

18. the geometrical region has a rectangular parallelepiped shape; 20. The method of claim 17, further comprising determining, from the plurality of voxels of the voxelized representation, at least a first and a second bounding voxel for the set of voxels, the first and second bounding voxels defining the rectangular prism shape of the geometric region.

19. 19. The method of claim 17 or 18, wherein the voxel characteristics of each voxel include acoustic characteristics associated with that voxel and / or a set of audio rendering instructions assigned to that voxel, and the common voxel characteristics of the voxels in the geometric region include common acoustic characteristics associated with these voxels and / or a common set of audio rendering instructions assigned to these voxels.

20. 20. A method according to any one of claims 17 to 19, further comprising determining, for the geometric region, at least one scene element parameter comprising one or more of a scene element identifier, an acoustic feature identifier and / or an audio rendering instruction set identifier, and indices of corresponding first and second bounding voxels defining the geometric region.

21. The method of claim 20 , further comprising applying entropy coding to the at least one scene element parameter for the geometric region.

22. 22. The method of claim 20 or 21, further comprising the step of outputting a bitstream comprising the at least one scene element parameter for determining, based on the determined set of voxels, a set of voxels associated with the geometric domain for the compressed representation of the audio scene.

23. 23. The method of any one of claims 17 to 22, wherein the geometrical domain relates to a scene element within the audio scene.

24. the audio scene comprises a larger scene represented by the determined set of voxels, the larger scene comprising a set of sub-scenes, each of the sub-scenes corresponding to a subset of the determined set of voxels; 24. The method according to any one of claims 17 to 23, further comprising the step of determining, among the determined set of voxels, a subset of said voxels for said corresponding sub-scene.

25. 25. The method according to any one of claims 17 to 24, further comprising the step of applying an interpolation of audio voxels in time and / or space.

26. 26. The method of claim 17, further comprising the step of redefining voxel properties for the subset of the set of voxels associated with scene subelements within the geometric domain to overwrite the subset with the redefined voxel properties.

27. determining a superset of voxels that includes the determined set of voxels, the determined set of voxels being associated with scene subelements within the geometric domain; 27. The method of claim 17, further comprising the step of assigning new voxel properties to the determined set of voxels and overwriting the voxel properties of the determined set of voxels with the new voxel properties.

28. 28. The method of claim 17, further comprising determining a voxel size for representing the geometrical domain, the voxel size being based on a number of voxels along a scene dimension of the geometrical domain.

29. 1. An apparatus comprising: a processor; and a memory coupled to the processor that stores instructions for the processor, the memory comprising: Apparatus, wherein the processor is adapted to perform a method according to any one of claims 1 to 28.

30. A program comprising instructions which, when executed by a processor, cause the processor to carry out a method according to any one of claims 1 to 28.

31. A computer-readable storage medium storing the program according to claim 30.

Images