Methods and apparatus for processing voxel-based scene representations

Abstract

The disclosure relates to methods of processing voxel-based audio scene representations comprising a voxel matrix, wherein entries of the voxel matrix are associated with respective voxels and take values relating to respective material identifiers. The method comprises, for a given value relating to a material identifier: generating a binary matrix with entries indicating whether a corresponding entry in the voxel matrix has the given value; selecting an entry of the binary matrix for which the corresponding entry in the voxel matrix has the given value; initializing a set of entries of the binary matrix with the selected entry; iteratively selecting dimensions from previously unselected dimensions of the binary matrix and updating the set of entries based on the selected dimensions; and generating a representation of the resulting set of entries for encoding or rendering. The method further comprises, at each iteration: determining a respective maximum range along each of the previously unselected dimensions relative to a given entry in the binary matrix, so that for each entry in the set of entries, all entries within the maximum range relative to the respective entry indicate that the corresponding entries in the voxel matrix have the given value; selecting that dimension that yields the largest maximum range; and updating the set of entries by adding, for each entry in the set of entries, all entries of the binary matrix within the maximum range for the selected dimension relative to the respective entry. The disclosure further relates to corresponding apparatus, computer programs, and computer-readable storage media.

Description

METHODS AND APPARATUS FOR PROCESSING VOXEL-BASED SCENE REPRESENTATIONS Cross-Reference to Related Applications

[0001] This application claims the benefit of priority from U.S. Provisional Application No. 63 / 582,620, filed on 14 September 2023, and European Application No.23207674.5, filed on 3 November 2023, each of which is incorporated by reference herein in its entirety. Technical Field

[0002] The present disclosure relates to techniques for processing voxel-based scene representations of audio scenes, for example for storage, transmission, and / or audio rendering. In particular, the present disclosure is directed to computationally efficient, real-time conversion between voxel-based scene representations of audio scenes. Background

[0003] The Moving Picture Experts Group (MPEG) is an alliance of working groups established jointly by the International Organization for Standardisation (ISO) and International Electrotechnical Commission (IEC), that sets standards for media coding, including audio coding. MPEG is organized under ISO / IEC JTC 1 / SC 29, and the audio group is presently identified as working group (WG) 6. WG 6 is currently working on a new audio standard (also known as MPEG-I Immersive Audio, ISO / IEC 23090-4).

[0004] The new MPEG-I standard enables an acoustic experience from different viewpoints and / or perspectives or listening positions by supporting scenes and various movements around such scenes, such as movements using various degrees of freedom such as three degrees of freedom (3DOF) or six degrees of freedom (6DoF) in Virtual reality (VR), augmented reality (AR), mixed reality (MR) and / or extended reality (XR) applications. A 6 DoF interaction extends a 3 DoF spherical video / audio experience that is limited to head rotations (pitch, yaw, and roll) to include translational movement (forward / back, up / down, and left / right), to allow for navigation within a virtual environment (e.g., physically walking inside a room), in addition to the head rotations.

[0005] For audio rendering in VR, AR, MR and XR applications, object-based approaches have been widely employed by representing a complex auditory scene as multiple separate audio objects, each of which is associated with parameters or metadata defining a location / position and trajectory of that object in the scene. Alternatively audio rendering in such environments also uses higher order ambisonics (HOA). However, a new usage of “voxels” for rendering audio scenes is now being explored, such as for use of new immersive audio experiences. Voxels for audio rendering are relevant for media environments implemented in both hardware and software, such as video game and / or VR, AR, MR and XR environments.

[0006] A Voxel is a space volume with acoustic properties or audio rendering instructions assigned to it. Voxel size may be an encoder configuration parameter, and it can be (manually or automatically) selected according to a scene geometry level of details (e.g., in the range of 10 cm – 1 m).

[0007] Voxels for audio rendering can be obtained by: • voxelization (or conversion) of a mesh-based scene representation • from scene representation used for scene generation (or even video rendering) (e.g., by up-sampling of voxels of smaller size)

[0008] Voxel-based scene representations typically are given in the form of two- or three- dimensional voxel matrices. Each entry of the voxel matrix is uniquely associated with a respective voxel. Values of the entries of the voxel matrix are indicative of material identifiers of respective voxels. In the simplest case, entries of the voxel matrix may take values from {0,1}, where 0 indicates a sound propagation medium (e.g., air voxel) and 1 indicates an occluding medium in which sound cannot propagate. In other implementations, entries of the voxel matrix may take values from a predefined set of (integer) values, with each value in the set representing a respective material identifier (material type identifier), such as air, wood, concrete, etc.

[0009] On the other hand, the renderer interface defined in MPEG-I Immersive Audio is not capable of processing voxel matrices, but relies on input of a predefined compressed representation for voxel-based audio scenes, as defined for instance in patent application PCT / EP2023 / 055380, titled “Methods, Apparatus, and Systems for Processing Audio Scenes for Audio Rendering” and published as WO2023 / 169934. As of today, techniques for conversion from voxel matrices to compressed representations compatible with the MPEG-I rendererinterface are computational expensive and do not allow for real-time conversion of audio scene information on conventional encoder or decoder devices.

[0010] There is thus need for improved techniques for converting (e.g., compressing) voxel- based audio scene information. There is a particular need for such techniques that can operate in real-time, even on devices with limited processing capability. Summary

[0011] In view of this need, the present disclosure provides methods of processing voxel-based scene representations of audio scenes, apparatus for processing voxel-based scene representations of audio scenes, computer programs, and computer-readable storage media, having the features of the respective independent claims.

[0012] One aspect of the present disclosure relates to a method of processing a voxel-based scene representation of an audio scene. The voxel-based scene representation may include a voxel matrix. The entries of the voxel matrix may be (uniquely) associated with respective voxels in the audio scene and take values from a set of values relating to respective material identifiers (material type identifiers) of the voxels. The method may include, for a given value in the set of values (i.e., for a given material identifier), generating a binary matrix with entries indicating whether a corresponding entry in the voxel matrix has the given value. The method may further include, for the given value in the set of values, selecting an entry of the binary matrix that indicates that the corresponding entry in the voxel matrix has the given value. The method may further include, for the given value in the set of values, initializing a set of entries of the binary matrix with the selected entry. Equivalently, a set of voxels may be initialized with a voxel associated with the selected entry. For example, the set of entries (or equivalently, voxels) may be initialized to consist of the selected entry (or equivalently, voxel associated with the selected entry). The method may further include, for the given value in the set of values, iteratively selecting dimensions (dimension directions) from previously unselected dimensions (dimension directions) of the binary matrix and updating the set of entries (or equivalently, voxels) based on the selected dimensions (dimension directions). The method may yet further include, for the given value in the set of values, generating a representation of the resulting set of entries (or equivalently, voxels) for encoding or rendering. The method may further include, at each iteration, determining a respective maximum range along each of the previously unselecteddimensions (dimension directions) relative to a given entry of the binary matrix (or equivalently, voxel), so that for each entry (or equivalently, voxel) in the set of entries, all entries (or equivalently, voxels) within the maximum range relative to the respective entry indicate that the corresponding entries in the voxel matrix have the given value. The method may further include, at each iteration, selecting that dimension (dimension direction) among the previously unselected dimensions that yields the largest maximum range. The method may yet further include, at each iteration, updating the set of entries (or equivalently, voxels) by adding, for each entry in the set of entries, all entries in the binary matrix within the maximum range for the selected dimension relative to the respective entry.

[0013] Thereby, a matrix-based representation (e.g., 2-dimensional diffraction map or 3- dimensional voxel matrix) can be converted to a compressed format, such as a format compatible with the MPEG-I Immersive Audio renderer interface. Moreover, this conversion can be performed in real time for state-of-the-art audio encoding and audio rendering devices, enabling real-time updates of metadata describing the audio scene at the audio renderer.

[0014] In some embodiments, the representation of the resulting set of entries (or equivalently, voxels) may be indicative of a start entry of the resulting set of entries, and end entry of the resulting set of entries, and the material identifier for the entries in the set of entries. For example, the voxel associated with the start entry (i.e., start voxel) may be a corner voxel of the cuboid formed by the set of voxels associated with the entries in the set of entries, and the voxel associated with the end entry (i.e., end voxel) may be the opposite corner voxel of the cuboid. In particular, the start entry may be the entry in the set with the smallest coordinate values in each of the dimensions (dimension directions) of the binary matrix, and the end entry may be the entry in the set of entries with the largest coordinate values in each of the dimensions (dimension directions) of the binary matrix. Thereby, a simple, yet compression-efficient representation of a cuboid group of voxels can be achieved.

[0015] In some embodiments, the method may further include outputting the generated representation for encoding. For example, the generated representation may be output to a bitstream. The bitstream may be in a format suitable for processing by a rendering device, such as a rendering device compatible with the MPEG-I Immersive Audio standard.

[0016] In some embodiments, the entry of the binary matrix may be selected as the first entry in a predetermined scan order along the entries of the binary matrix that indicates that thecorresponding entry in the voxel matrix has the given value. The predetermined scan order may for example correspond to a grid scan order along the x-, y-, and z-directions, in that order.

[0017] In some embodiments, the method may further include setting entries in the binary matrix that are included in the resulting set of entries for the given value in the set of values to a value indicating that the corresponding entries in the voxel matrix do not have the given value. In other words, entries already included in a determined set of entries may be removed from the binary matrix (e.g., in the sense that they are nulled), and thereby, from the voxel-based scene representation. The resulting sets of entries generated by the proposed method therefore are overlap-free, so that a reconstructed audio scene at, for example, a rendering device is independent of an order in which the representations of sets of entries are processed.

[0018] In some embodiments, the method may further include selecting another entry in the binary matrix that indicates that the corresponding entry in the voxel matrix has the given value.

[0019] In some embodiments, entries of the binary matrix may be selected for iterative processing until all entries of the binary matrix that indicate that the corresponding entries in the voxel matrix have the given value have been assigned to sets of voxels. That is, successive selection of entries in the binary matrix for iterative processing may be repeated until the binary matrix does not anymore include entries indicating that corresponding entries in the voxel matrix have the given value.

[0020] In some embodiments, the method may further include outputting a set of representations of sets of entries for encoding or rendering. Here, it is understood that the set of representations (of sets of entries may) allow for a full (i.e., lossless) reconstruction of the binary matrix (and by extension, of the voxel-based audio scene representation). In some implementations, the set may be an ordered set. A reconstructed scene representation may not depend on the particular order, since the sets of entries are overlap-free.

[0021] In some embodiments, the method may further include processing all values in the set of values according to a fixed processing order. For example, the values in the set of values may be processed in numerical order, from small to large, excluding 0.

[0022] In some embodiments, the method may be performed at an audio rendering device and the output of the method may be used for input to an audio rendering process. In some implementations, the audio rendering process may be performed at the same rendering device. Inother implementations, the audio rendering process may be performed at another rendering device, after transmission of the output of the method.

[0023] In some embodiments, the voxel matrix may be a 2-dimensional matrix. In some embodiments, the voxel matrix may relate to an audio diffraction map. For example, the encoded output may be provided to another audio renderer for audio rendering. The audio diffraction map may have been generated based on a 3-dimensional voxel matrix, for example by means of projection to a given plane or by intersection with a given plane.

[0024] In some embodiments, the voxel matrix may be a 3-dimensional matrix. In some embodiments, the voxel matrix may relate to 3-dimensional representation of the audio scene.

[0025] According to another aspect, an apparatus for processing audio scene information is provided. The apparatus may include a processor and a memory coupled to the processor and storing instructions for the processor. The processor may be configured to perform the methods or method steps outlined throughout the present disclosure.

[0026] According to another aspect, an audio rendering apparatus is provided. The apparatus may include a processor and a memory coupled to the processor and storing instructions for the processor. The processor may be configured to perform the methods or method steps outlined throughout the present disclosure.

[0027] According to another aspect, an audio encoder apparatus is provided. The apparatus may include a processor and a memory coupled to the processor and storing instructions for the processor. The processor may be configured to perform the methods or method steps outlined throughout the present disclosure.

[0028] According to a further aspect, a computer program is described. The computer program may comprise executable instructions for performing the methods or method steps outlined throughout the present disclosure when executed by a computing device (e.g., processor).

[0029] According to another aspect, a computer-readable storage medium is described. The storage medium may store a computer program adapted for execution on a computing device (e.g., processor) and for performing the methods or method steps outlined throughout the present disclosure when carried out on the computing device.

[0030] It should be noted that the methods and apparatus including its preferred embodiments as outlined in the present disclosure may be used stand-alone or in combination with the other methods and apparatus disclosed in this document. Furthermore, all aspects of the methods andapparatus outlined in the present disclosure may be arbitrarily combined. In particular, the features of the claims may be combined with one another in an arbitrary manner.

[0031] It will be appreciated that apparatus features and method steps may be interchanged in many ways. In particular, the details of the disclosed method(s) can be realized by the corresponding apparatus, and vice versa, as the skilled person will appreciate. Moreover, any of the above statements made with respect to the method(s) (and, e.g., their steps) are understood to likewise apply to the corresponding apparatus (and, e.g., their blocks, stages, units), and vice versa. Brief Description of the Drawings

[0032] The invention is explained below in an exemplary manner with reference to the accompanying drawings, wherein

[0033] Fig. 1 schematically illustrates an example of an interface between a voxel-based scene geometry engine and an audio renderer to which techniques according to embodiments of the disclosure may be applied;

[0034] Fig. 2 schematically illustrates an example of an interface between first and second audio renderers to which techniques according to embodiments of the disclosure may be applied;

[0035] Fig. 3 schematically illustrates an example of an interface between a voxel-based scene geometry engine and an audio renderer according to embodiments of the disclosure;

[0036] Fig. 4 schematically illustrates an example of an interface between first and second audio renderers according to embodiments of the disclosure;

[0037] Fig. 5 is a flowchart illustrating an example of a method of processing a voxel-based audio scene representation according to embodiments of the disclosure;

[0038] Fig. 6 is a flowchart illustrating an example of a method implementing a step of the method of Fig. 5 according to embodiments of the disclosure;

[0039] Fig. 7 to Fig. 10 schematically illustrate examples of iteratively updating a set of matrix entries associated with respective voxels of the voxel-based audio scene representation, according to embodiments of the disclosure; and

[0040] Fig. 11 is a block diagram schematically illustrating an example of an apparatus suitable for implementing methods according to embodiments of the disclosure.Detailed Description

[0041] In the following, example embodiments of the disclosure will be described with reference to the appended figures. Identical elements in the figures may be indicated by identical reference numbers, and repeated description thereof may be omitted. Voxel-based Audio Scene Representations

[0042] First, an overview over voxel-related concepts for representation of audio scenes will be given. What is a voxel for audio rendering?

[0043] A voxel is understood as a space volume with acoustic properties or audio rendering instructions assigned to it. What is a voxel size for audio rendering?

[0044] The voxel size may be an encoder configuration parameter. It may be (manually or automatically) selected according to a scene geometry level of details (e.g., in the range of 10 cm – 1 m). How large audio scenes can be handled?

[0045] Large audio scenes do not necessarily result in a large number of voxels and high rendering complexity. For example, a large audio scene can be represented as • a set of independent sub-scenes (and method for “teleport” between these representations without a renderer “re-start”) • a set of scenes updates (based on the user position) How can discontinuity issues caused by voxel granularity be handled?

[0046] Any strong discontinuities in sound levels (and jumps of diffracted signal direction) can be avoided by application of interpolation (e.g., in time and space). How to represent voxel-based audio scenes?

[0047] Any voxel-based representation of an audio scene may contain an indication of voxels that are not transmission voxels (e.g., that are occluder voxels), i.e., voxels in which sound cannot propagate or cannot freely propagate – a representation of occluding geometries. This indication may relate to an indication of coordinates (e.g., center coordinates, corner coordinates, etc.) of therespective voxels. The coordinates of these voxels may be represented by grid indices, for example. Additionally, the voxel-based representation may include indications of material properties of the voxels that are not transmission voxels, such as absorption coefficients, reflection coefficients, etc. In addition to the occluder voxels, the voxel-based representation may also indicate transmission voxels (e.g., air voxels), i.e., voxels in which sound can propagate – a representation of sound propagation media. Accordingly, some implementations of voxel-based representations of audio scenes may include, for each voxel in a predefined section of space (e.g., within boundaries enclosing the audio scene), and indication of a respective material property. Examples of Use Cases

[0048] Broadly speaking, the present disclosure provides a solution for converting a voxel-based scene matrix representation (voxel-based scene representation comprising a voxel matrix), for example as used in state-of-the-art file formats and voxel scene engine memory, to the MPEG-I Immersive Audio metadata syntax.

[0049] In particular, the present disclosure provides a lossless and real-time capable (meta)data conversion and compression method for (integer) 3D and 2D matrices, for example representing voxel scenes or acoustic diffraction projection planes, for encoding and / or audio rendering.

[0050] The proposed techniques (e.g., metadata conversion and compression methods) can be used for example • at the audio encoder side, for example for online or offline processing of voxel- based audio scene representations, • at the audio renderer side for online or offline processing of voxel-based audio scene representations, in particular representations of acoustic diffraction projection planes, for transmission to another audio renderer, and • at the renderer side for real-time processing of voxel-based audio scene representations, for audio rendering.

[0051] Illustrative examples of feasible use cases for techniques according to embodiments of the disclosure will be described next.Example Use Case (A): Conversion of Voxel Scene Geometry (3D voxel matrix)

[0052] In this example, a user may experience a virtual environment that is (procedurally) generated by a scene engine in real-time. A scene part of the virtual environment may be created around the user position during scene presentation time. State-of-the-art scene engines represent all scene parts by voxels sorted as a 3D matrix M used for video rendering, for example via generation of meshes visible to the user. A similar 3D matrix representation MVcontaining entries describing (or referencing) material properties (e.g., occlusion and reflection coefficients) can be used for audio rendering.

[0053] Fig. 1 schematically illustrates an example of an interface between a voxel-based scene geometry engine 110 and an audio renderer 140, which is an MPEG-I Immersive Audio renderer. The voxel-based scene geometry engine 110 generates the 3D matrix representation MV120. The audio renderer 140 receives the scene matrix data using the metadata input interface defined by the bitstream syntax. This interface supports the compressed format of the voxel scene representation. The corresponding voxel scene V is defined as a set of vectors CV130 describing voxel scene elements stored in the compressed format in the payload element voxSceneElements(), CV ={MatID, PackedPosS, PackedPosE}.

[0054] Vector CVindicates a group of voxels filling a cuboid volume. These voxels are of the same type or material. MatID denotes a material ID of the group of voxels, PackedPosS indicates a representation of a start voxel of the group of voxels (e.g., a corner of the cuboid volume), and PackedPosE indicates a representation of an end voxel of the group of voxels (e.g., opposite corner of the cuboid volume).

[0055] The conversion process from CVto MV(e.g., decompression) is given in the MPEG-I Immersive Audio specification as described in Annex A.1. Example Use Case (B): Diffraction Projection Map (2D matrix)

[0056] In this example, the user experiences a virtual environment that is rendered by one instance (A) of an MPEG-I Immersive Audio renderer. It calculates (and caches) audio diffraction metadata containing the diffraction projection map represented by the 2D voxel matrix MD. This matrix MDcan be used for acoustic diffraction pathfinding and subsequent generation of all (virtual)diffraction audio sources. The diffraction metadata can be shared with another instance (B) of the MPEG-I Immersive Audio renderer (e.g., in real-time or offline) to reduce computational complexity. In other words, the MPEG-I Immersive Audio renderer can act as an encoder of the diffraction metadata for itself or other audio renderer(s).

[0057] Fig. 2 schematically illustrates an example of an interface between two audio renderers for diffraction metadata. The first MPEG-I Immersive Audio renderer (A) 210 generates the 2D voxel matrix MD220. The second MPEG-I Immersive Audio renderer (B) 240 receives the diffraction matrix data using the metadata input interface defined by the bitstream syntax. This interface supports the compressed format of the voxel projection map representation. The corresponding diffraction projection map D is defined as a set of vectors CD 230 describing voxel scene elements stored in a compressed format in the payload element voxSceneDiffractionMap(), CD= {MatID, PackedPosS, PackedPosE}.

[0058] The conversion process from CDto MD(e.g., decompression) is given in the MPEG-I Immersive Audio specification as described in Annex A.2. Techniques for Processing Audio Scene Information

[0059] The conversion process of 3D (and 2D) matrices MV(and MD) to the bitstream metadata format CV(and CD) supported by the MPEG-I Immersive Audio renderer interface is currently not specified in MPEG-I Immersive Audio. Several conversion methods may be feasible in this regard, however none of these is capable of real-time conversion at state-of-the-art rendering devices.

[0060] The present disclosure provides a fast (low complexity) conversion and (real-time capable) compression algorithm.

[0061] An application example of techniques according to embodiments of the disclosure to the interface between voxel-based scene geometry engine and audio renderer is schematically illustrated in Fig.3.

[0062] In this example, the scene engine 310 generates a (3D) matrix representation (voxel matrix) MVwhich is input to (encoder-side) conversion (or compression) process 320. Theresulting representation CV330 describing voxel scene elements stored in the compressed format can then be input to audio renderer 340 via the prescribed renderer interface.

[0063] Alternatively, conversion (e.g., compression) may be performed at the renderer-side, prior to input to a rendering process at the audio render 340 via the prescribed renderer interface.

[0064] An application example of techniques according to embodiments of the disclosure to the interface between two audio renderers for the diffraction metadata is schematically illustrated in Fig. 4.

[0065] In this example, the first audio renderer 410 generates a (2D) matrix representation (diffraction map) MDwhich is input to (first renderer-side) conversion (or compression) process 420. The resulting representation CD 430 describing voxel scene elements of the diffraction map stored in the compressed format can then be input to the second audio renderer 440 via the prescribed renderer interface.

[0066] Alternatively, conversion (e.g., compression) may be performed at the second renderer- side, prior to input to a rendering process at the second audio render 440 via the prescribed renderer interface. Conversion Algorithm

[0067] Conversion (e.g., compression) techniques according to the present disclosure are based on a single material voxel-block “expansion” algorithm, focused on low complexity performance (rather than high compression rate) to support real-time data communication.

[0068] The proposed methods may comprise the following three repetitive basic (encoding) steps: 1) Selection of one material type ID from all available material types (contained in the voxel matrix M). Essentially, one from a set of values relating to respective material identifiers of the voxels is selected. 2) Creation of a Boolean matrix (binary matrix) containing information about the selected material type ID entries. Entries of this Boolean may have value “1” if corresponding entries in the voxel matrix have the selected material type ID, and value “0” otherwise. 3) Lossless encoding of this Boolean matrix (binary matrix) in terms of the bitstream metadata format C. The bitstream metadata format C may for example describe voxelscene elements stored in the compressed format in the payload element voxSceneElements() (3-dimensional case) or describe voxel scene elements stored in the compressed format in the payload element voxSceneDiffractionMap() (2-dimensional case).

[0069] The third encoding step may be performed using an iterative procedure, such as the residual-estimation-free “voxel-block expansion” method.

[0070] An example of a method 500 of processing a voxel-based scene representation of an audio scene in line with the above is shown in the flowchart of Fig.5. The voxel-based scene representation is assumed, in line with the above, to comprise a voxel matrix M, where entries of the voxel matrix M are (uniquely) associated with respective voxels in the audio scene and take values from a set of values relating to respective material identifiers (material type identifiers) of the voxels. Thereby, each entry of the voxel matrix M indicates a material type ID of a respective associated voxel in the audio scene.

[0071] Method 500 has steps S510 through S550, which are performed for a given material type ID (i.e., for a given value among the aforementioned set of values).

[0072] At step S510, a binary matrix (Boolean matrix) with entries indicating whether a corresponding entry in the voxel matrix M has the given value is generated. This may be in line with step 2) described above.

[0073] At step S520, an entry of the binary matrix that indicates that the corresponding entry in the voxel matrix has the given value is selected. This entry of the binary matrix may be selected as the first entry in a predetermined scan order along the entries of the binary matrix that indicates that the corresponding entry in the voxel matrix has the given value. An example for the selection is described in step 3.1) below. For example, the predetermined scan order may correspond to a scan order along the x-, y-, and z-directions, in that order.

[0074] At step S530, a set of entries of the binary matrix is initialized with the selected entry. Noting that each entry in the binary matrix is uniquely associated with a respective voxel it may equivalently be said that a set of voxels is initialized to (or with) a voxel associated with the selected entry.

[0075] At step S540, dimensions (dimension directions) are iteratively selected from previously unselected dimensions of the binary matrix and the set of entries is iteratively updated based on the selected dimensions. For a 3-dimensional voxel matrix, there are three dimensions, whichmay be indicated by x, y, and z. For a 2-dimensional voxel matrix (e.g., diffraction map), there are two dimensions, which may be indicated by x and y.

[0076] Noting that each entry in the binary matrix is uniquely associated with a respective voxel, it may equivalently be said that dimensions (dimension directions) are iteratively selected from previously unselected dimensions (dimension directions) of the binary matrix and the set of voxels is updated based on the selected dimensions (dimension directions).

[0077] At step S550, a representation of the resulting set of entries is generated, for example for encoding or rendering.

[0078] Noting that each entry in the binary matrix is uniquely associated with a respective voxel, it may equivalently be said that a representation of the resulting set of voxels is generated.

[0079] The generated representation may correspond to a vector (or line) in the set of vectors (or lines) CVor CDdescribed above, for example. That is, the representation may correspond to the compressed format in the payload element voxSceneElements() (3-dimensional case) or the compressed format in the payload element voxSceneDiffractionMap() (2-dimensional case). In any case, the representation of the resulting set of entries (or voxels) may be indicative of a start entry (or voxel) of the resulting set of entries (e.g., PackedPosS), and end entry (or voxel) of the resulting set of entries (e.g., PackedPosE), and the material identifier for the entries in the set of entries (e.g., MatID).

[0080] For example, the start voxel may be a corner voxel of the cuboid formed by the set of voxels and the end voxel may be the opposite corner voxel of the cuboid. The start voxel may be the voxel in the set with the smallest coordinate values in each of the dimension directions of the binary matrix and the end voxel may be the voxel in the set of voxels with the largest coordinate values in each of the dimension directions of the binary matrix.

[0081] The generated representation may be output for encoding or rendering. For example, the generated representation may be output to a bitstream.

[0082] As will be described below, processing the binary matrix typically will yield several representations of sets of entries (or voxels), so that an output of proposed method may comprise a set of representations of sets of entries for encoding or rendering. It is understood that the set of representations of sets of entries (or voxels) may allow for a full (i.e., lossless) reconstruction of the binary matrix. In some implementations (even though this is not necessary for lossless reconstruction), the set may be an ordered set.

[0083] Next, an example implementation of step 3) defined above, i.e., lossless encoding of the Boolean matrix (binary matrix) in terms of the bitstream metadata format C will be described. 3.1) Find the first voxel of the material type MatID (e.g., using the fastest array search method) and assign it to the initial block of voxels C(i=0)sharing the same material type MatID. An example is shown in Fig. 7, where voxel 710 is the first voxel of material type MatID. 3.2) (Virtually) expand this block of voxels C(i)individually in each available dimension direction, as long as the expanded block contains only voxels of the same material type MatID. This may amount to determining, for each available dimension direction, a maximum range (or maximum extension) along the dimensions direction relative to a given voxel, so that for each voxel in the block (or set) of voxels C(i), all voxels within the maximum range relative to the respective voxel have the material type ID for which the Boolean matrix has been generated. This determination may be based on the Boolean matrix alone. In the example of Fig.7, an expansion in a first dimension direction (e.g., x-direction) yields (virtually) expanded block 720, an expansion in a second dimension direction (e.g., y-direction) yields (virtually) expanded block 730, and an expansion in a third dimension direction (e.g., z-direction) yields (virtually) expanded block 740. 3.3) Select that (virtually) expanded block obtained at 3.2) that results in the largest number of covered voxels. In the example of Fig.7, this would be block 720. This may amount to selecting a dimension direction among the available dimension directions that yields the largest maximum range. 3.4) Expand the block of voxels C(i)to the corresponding selected expanded block to obtain block of voxels C(i+1). An example is shown in Fig. 8, where block 820 relates to the expanded block of voxels C(i+1). This may amount to expanding the block of voxels C(i)in the selected dimension direction, in accordance with the maximum range for that dimension direction. 3.5) Repeat steps 3.2) through 3.4) until all dimension directions (e.g., 3 for MVor 2 for MD) dimensions have been checked (i.e., used for expansion of the block of voxels). For instance, block 820 in Fig.8 relates to the block of voxels C(i=1)obtained by expanding in the first selected dimension direction. (Virtual) expansion in the other two dimension directions yields blocks 830, 840. Of these, block 830 results in more covered voxels thanblock 840, so that block 830 is selected for expansion of the block of voxels C(i=1)to obtain block of voxels C(i=2). A further example is shown in Fig.9, where block 930 relates to the expanded block of voxels C(i=2), obtained by expansion in the second selected dimension. Finally, in the present 3-dimensional example, only the third dimension direction remains for virtual expansion (block 940 in Fig.9) and expansion of the block of voxels C(i=2)in the third dimension direction yields block of voxels C(i=3), which is illustrated by block 1040 in the example of Fig.10.

[0084] The method may then loop back to step 3.1) after the voxels that are included in the final block of voxels C(i=N), where ^^ is the number of dimension directions of the voxel matrix M, have been removed from the Boolean matrix (binary matrix.)

[0085] In line with the above, an example implementation of step S540 of method 500 is given by method 600 shown in Fig. 6. Method 600 comprises steps S610 through S630, which are performed at each iteration in step S540. Here, the number of iterations corresponds to the number of dimensions (dimension directions) of the voxel matrix that is to be converted (compressed). While the aforementioned use cases and examples relate to 3-dimensional and 2- dimensional voxel matrices, the proposed methods are likewise applicable to higher-dimensional matrix representations.

[0086] At step S610, a respective maximum range is determined along each of the previously unselected dimensions (dimension directions) relative to a given entry in the binary matrix. For each of the previously unselected dimensions (dimension directions), this determination is performed such that for each entry in the set of entries, all entries within the maximum range relative to the respective entry indicate that the corresponding entries in the voxel matrix have the given value. For a given dimension (dimension direction), the maximum range may relate to an extension in a positive direction along the given dimension (dimension direction) and an extension in a negative direction along the given dimension (dimension direction), relative to the given entry.

[0087] Noting that each entry in the binary matrix is uniquely associated with a respective voxel in the binary matrix (or voxel matrix), it may be said that a respective maximum range along each of the previously unselected dimensions (dimension directions) relative to a given voxel is determined, so that for each voxel in the set of voxels, all voxels within the maximum rangerelative to the respective voxel have associated entries in the binary matrix that indicate that the corresponding entries in the voxel matrix have the given value.

[0088] At step S620, that dimension (dimension direction) among the previously unselected dimensions (dimension directions) is selected that yields the largest maximum range.

[0089] At step S630, the set of entries is updated by adding, for each entry in the set of entries, all entries of the binary matrix within the maximum range for the selected dimension (dimension direction) relative to the respective entry.

[0090] Noting that each entry in the binary matrix is uniquely associated with a respective voxel in the binary matrix (or voxel matrix), it may be said that the set of voxels is updated by adding, for each voxel in the set of voxels, all voxels within the maximum range for the selected dimension relative to the respective voxel.

[0091] After all dimensions (dimension directions) have been processed and used for expansion / updating of the set of entries (or voxels), all those entries in the binary matrix that are included in the resulting set of entries for the given value in the set of values are set to a value indicating that the corresponding entries in the voxel matrix do not have the given value. That is, these entries are removed from the binary matrix (and by extension, from the voxel-based representation) to avoid double-processing of entries and achieve overlap-free sets of entries.

[0092] Then, another entry in the binary matrix that indicates that the corresponding entry in the voxel matrix has the given value (e.g., is of the given MatID) may be selected for iterative processing. In particular, entries of the binary matrix may be selected for iterative processing (with subsequent removal of resulting set of entries from the binary matrix) until all entries of the binary matrix that indicate that the corresponding entries in the voxel matrix have the given value have been assigned to sets of entries. Put differently, successive selection of entries in the binary matrix for iterative processing may be repeated until the binary matrix does not include any more entries indicating that corresponding entries in the voxel matrix have the given value. Selection of the entries of the binary matrix may be performed in accordance with the aforementioned scan order. Noting that previously encoded entries have been removed from the binary matrix, this may amount to repeatedly selecting the first entry in a predetermined scan order along the entries of the (updated) binary matrix that indicates that the corresponding entry in the voxel matrix has the given value.

[0093] Representations of all determined sets of entries may be output for encoding or rendering.

[0094] The above processing relates to a single value in the set of values (e.g., to a single material type ID or MatID). It is understood that all values in the set of values (e.g., all MatIDs) may be processed, in line with the above, according to a fixed processing order (e.g., in numerical order, from small to large, excluding 0).

[0095] Thereby, the voxel matrix M will be decomposed into non-overlapping blocks (e.g., cuboids) of voxels, with all voxels in a given block having the same material type.

[0096] An example of pseudocode that implements steps 1) through 3) with sub-steps 3.1) through 3.5) (and correspondingly, method 500 with sub-steps as per method 600) is given below. Further, examples of conversion process results are given in Annex B and performance estimations are provided in Annex C. Example Pseudocode % ************************************************************* / % Compression iterations % ************************************************************* / function all_lines = vox_compress(vox, scene_size) all_lines = []; tmp = unique(vox)'; materials_ids = tmp(2:end); materials_encoded = zeros(length(materials_ids),1)'; num_dmentions = length(scene_size); for idx_material = 1:length(materials_encoded) mat_id = materials_ids(idx_material); vox_mat_id = (vox == mat_id); % material selection while any(any(any(vox_mat_id))) [Ps(1), Ps(2), Ps(3)] = ind2sub(size(vox_mat_id), find(vox_mat_id, 1, 'first')); Pe = Ps; % block initialization checked_dimensions = true(3,1)'; for i=1:num_dmentions checked_dimensions(i) = false; end for i=1:num_dmentions % voxel block enlagement loop for all dimensions [vox_mat_id, Ps, Pe, checked_dimensions] = vox_compress_one_dimension(vox_mat_id, Ps, Pe, checked_dimensions, scene_size); end new_line = [mat_id, Ps, Pe];all_lines = [all_lines; new_line]; end end% Block enlargement for one dimensionfunction [vox_mat_id, Ps, Pe, checked_dimensions] = vox_compress_one_dimension(vox_mat_id, Ps, Pe, checked_dimensions, scene_size) i_ex = 0; i_ey = 0; i_ez = 0; Px = [Ps(1):Pe(1)]; Py = [Ps(2):Pe(2)]; Pz = [Ps(3):Pe(3)]; if ~checked_dimensions(1) for i = Ps(1):scene_size(1) i_ex = i + 1; P = min(i_ex,scene_size(1)); if ~all(all(all(vox_mat_id(P, Py, Pz)))) break end end end if ~checked_dimensions(2) for i = Ps(2):scene_size(2) i_ey = i + 1; P = min(i_ey,scene_size(2)); if ~all(all(all(vox_mat_id(Px, P, Pz)))) break end end end if ~checked_dimensions(3) for i = Ps(3):scene_size(3) i_ez = i + 1; P = min(i_ez,scene_size(3)); if ~all(all(all(vox_mat_id(Px, Py, P)))) break end end end[~, selected_dimension] = max([i_ex-Ps(1), i_ey-Ps(2), i_ez- Ps(3)]); checked_dimensions(selected_dimension) = true; if selected_dimension == 1 Pe(1) = i_ex-1; Px = [Ps(1):Pe(1)]; elseif selected_dimension == 2 Pe(2) = i_ey-1; Py = [Ps(2):Pe(2)]; elseif selected_dimension == 3 Pe(3) = i_ez-1; Pz = [Ps(3):Pe(3)]; end vox_mat_id(Px, Py, Pz) = false; Technical Advantages

[0097] Conversion (e.g., compression) techniques according to embodiments of the disclosure support real-time metadata updates at the encoder side or the decoder / renderer side (e.g., L3- updates).

[0098] For use case (A) described above, the proposed methods allow to run the MPEG-I Immersive Audio renderer without need to encode any voxel-related geometry metadata. All voxel information can be converted to the compressed bitstream syntax format (e.g., during scene initialization) and sent into the audio renderer, for example via L3 updates. This can make the technology “encoder-free” and provide the “low bitrate” operation point for the MPEG-I Immersive Audio renderer.

[0099] For use case (B) described above, the proposed methods allow one MPEG-I Immersive Audio renderer to provide other renderer instances with data for the acoustic diffraction modelling tool, therefore acting as an encoder. This supports the “low complexity” operation point for the MPEG-I Immersive Audio renderer.

[0100] In both use cases, the proposed methods may also be used for conversion at an input-side of respective audio renderers.Apparatus

[0101] While methods and processing chains have been described above, it is understood that the present disclosure likewise relates to apparatus (e.g., computer apparatus or apparatus having processing capability in general) for implementing these methods and processing chains (or techniques in general).

[0102] An example of such apparatus 1100 is schematically illustrated in Fig.11. The apparatus 1100 comprises a processor 1110 and a memory 1120 coupled to the processor 1110. The memory 1120 may store instructions for execution by the processor 1110. The processor 1110 may be adapted to implement the devices (e.g., encoders or decoders / renderers) described throughout the disclosure and / or to perform methods (e.g., methods of processing voxel-based scene representations of audio scenes) described throughout the disclosure. The apparatus 1100 may receive inputs (e.g., voxel-based scene representations of audio scenes, etc.) and generate outputs (e.g., representations of sets of entries / voxels, etc.) as described throughout the disclosure.

[0103] In one implementation, for example in line with use case (A) described above, the apparatus may relate to an audio encoder apparatus that performs conversion (e.g., compression) of the voxel-based representation (e.g., voxel matrix M) at its output, for providing the converted output to an audio rendering apparatus. This processing may be performed offline, for example.

[0104] In another implementation, for example in line with use case (A) described above, the apparatus may relate to an audio rendering apparatus that performs conversion (e.g., compression) of the voxel-based representation (e.g., voxel matrix M) at its input, before consuming the converted representation in an audio rendering process. This process may be performed in real time.

[0105] In both the above implementations for use case (A), the voxel matrix M may be a 3- dimensional matrix, relating to a 3-dimensional representation of the audio scene, for example.

[0106] In one implementation, for example in line with use case (B) described above, the apparatus may relate to an audio rendering apparatus that performs conversion (e.g., compression) of the voxel-based representation (e.g., voxel matrix M) at its output, for providing the converted output to another audio rendering apparatus. This processing may be performed offline, for example.

[0107] In one implementation, for example in line with use case (B) described above, the apparatus may relate to an audio rendering apparatus that performs conversion (e.g., compression) of the voxel-based representation (e.g., voxel matrix M) at its input, before consuming the converted representation in an audio rendering process. This process may be performed in real time, for example.

[0108] In both the above implementations for use case (B), the voxel matrix may be a 2- dimensional matrix, relating to an audio diffraction map. Interpretation

[0109] Aspects of the systems described herein may be implemented in an appropriate computer-based sound processing network environment (e.g., server or cloud environment) for processing digital or digitized audio files. Portions of these systems may include one or more networks that comprise any desired number of individual machines, including one or more routers (not shown) that serve to buffer and route the data transmitted among the computers. Such a network may be built on various different network protocols, and may be the Internet, a Wide Area Network (WAN), a Local Area Network (LAN), or any combination thereof.

[0110] One or more of the components, blocks, processes or other functional components may be implemented through a computer program that controls execution of a processor-based computing device of the system. It should also be noted that the various functions disclosed herein may be described using any number of combinations of hardware, firmware, and / or as data and / or instructions embodied in various machine-readable or computer-readable media, in terms of their behavioral, register transfer, logic component, and / or other characteristics. Computer-readable media in which such formatted data and / or instructions may be embodied include, but are not limited to, physical (non-transitory), non-volatile storage media in various forms, such as optical, magnetic or semiconductor storage media.

[0111] Specifically, it should be understood that embodiments may include hardware, software, and electronic components or modules that, for purposes of discussion, may be illustrated and described as if the majority of the components were implemented solely in hardware. However, one of ordinary skill in the art, and based on a reading of this detailed description, would recognize that, in at least one embodiment, the electronic-based aspects may be implemented insoftware (e.g., stored on non-transitory computer-readable medium) executable by one or more electronic processors, such as a microprocessor and / or application specific integrated circuits (“ASICs”). As such, it should be noted that a plurality of hardware and software-based devices, as well as a plurality of different structural components, may be utilized to implement the embodiments. For example, computer-implemented neural networks described herein can include one or more electronic processors, one or more computer-readable medium modules, one or more input / output interfaces, and various connections (e.g., a system bus) connecting the various components.

[0112] While one or more implementations have been described by way of example and in terms of the specific embodiments, it is to be understood that one or more implementations are not limited to the disclosed embodiments. To the contrary, it is intended to cover various modifications and similar arrangements as would be apparent to those skilled in the art. Therefore, the scope of the appended claims should be accorded the broadest interpretation so as to encompass all such modifications and similar arrangements.

[0113] Also, it is to be understood that the phraseology and terminology used herein are for the purpose of description and should not be regarded as limiting. The use of “including,” “comprising,” or “having” and variations thereof are meant to encompass the items listed thereafter and equivalents thereof as well as additional items. Unless specified or limited otherwise, the terms “mounted,” “connected,” “supported,” and “coupled” and variations thereof are used broadly and encompass both direct and indirect mountings, connections, supports, and couplings. Enumerated Example Embodiments

[0114] Various Aspects and implementations of the invention may also be appreciated from the following enumerated example embodiments (EEEs), which are not claims. EEE 1. A method of processing a voxel-based scene representation of an audio scene, wherein the voxel-based scene representation comprises a voxel matrix and wherein entries of the voxel matrix are associated with respective voxels in the audio scene and take values from a set of values relating to respective material identifiers of the voxels, the method comprising, for a given value in the set of values:generating a binary matrix with entries indicating whether a corresponding entry in the voxel matrix has the given value; selecting an entry of the binary matrix that indicates that the corresponding entry in the voxel matrix has the given value; initializing a set of entries of the binary matrix with the selected entry; iteratively selecting dimensions from previously unselected dimensions of the binary matrix and updating the set of entries based on the selected dimensions; and generating a representation of the resulting set of entries for encoding or rendering, wherein the method further comprises, at each iteration: determining a respective maximum range along each of the previously unselected dimensions relative to a given entry in the binary matrix, so that for each entry in the set of entries, all entries within the maximum range relative to the respective entry indicate that the corresponding entries in the voxel matrix have the given value; selecting that dimension among the previously unselected dimensions that yields the largest maximum range; and updating the set of entries by adding, for each entry in the set of entries, all entries of the binary matrix within the maximum range for the selected dimension relative to the respective entry. EEE 2. The method according to EEE 1, wherein the representation of the resulting set of entries is indicative of a start entry of the resulting set of entries, and end entry of the resulting set of entries, and the material identifier for the entries in the set of entries. EEE 3. The method according to EEE 1 or EEE 2, further comprising: outputting the generated representation for encoding or rendering. EEE 4. The method according to any one of EEEs 1 to 3, wherein the entry of the binary matrix is selected as the first entry in a predetermined scan order along the entries of the binary matrix that indicates that the corresponding entry in the voxel matrix has the given value. EEE 5. The method according to any one of EEEs 1 to 4, further comprising: setting entries in the binary matrix that are included in the resulting set of entries for the given value in the set of values to a value indicating that the corresponding entries in the voxel matrix do not have the given value.EEE 6. The method according to EEE 5, further comprising: selecting another entry in the binary matrix that indicates that the corresponding entry in the voxel matrix has the given value. EEE 7. The method according to EEE 6, wherein entries of the binary matrix are selected for iterative processing until all entries of the binary matrix that indicate that the corresponding entries in the voxel matrix have the given value have been assigned to sets of entries. EEE 8. The method according to EEE 6 or EEE 7, further comprising: outputting a set of representations of sets of entries for encoding or rendering. EEE 9. The method according to any one of EEEs 1 to 8, further comprising: processing all values in the set of values according to a fixed processing order. EEE 10. The method according to any one of EEEs 1 to 9, wherein the method is performed at an audio rendering device and the output of the method is used for input to an audio rendering process. EEE 11. The method according to any one of EEEs 1 to 10, wherein the voxel matrix is a 2-dimensional matrix. EEE 12. The method according to any one of EEEs 1 to 11, wherein the voxel matrix relates to an audio diffraction map. EEE 13. The method according to any one of EEEs 1 to 10, wherein the voxel matrix is a 3-dimensional matrix. EEE 14. The method according to any one of EEEs 1 to 10 or 13, wherein the voxel matrix relates to 3-dimensional representation of the audio scene. EEE 15. An audio rendering apparatus comprising a processor and a memory coupled to the processor and storing instructions for the processor, wherein the processor is configured to perform the method according to any one of EEEs 1 to 14. EEE 16. An audio encoder apparatus comprising a processor and a memory coupled to the processor and storing instructions for the processor,wherein the processor is configured to perform the method according to any one of EEEs 1 to 10, 13, or 14. EEE 17. A program comprising instructions that, when executed by a processor, cause the processor to carry out the method according to any one of EEEs 1 to 14. EEE 18. A computer-readable storage medium storing the program of EEE 17.Annex A.1 The conversion process from CVto MV(e.g., decompression) is given in the MPEG-I Immersive Audio specification as described below. Syntax No. of bits Mnemonic voxSceneElements() { numberOfVoxSceneElements = escapedValue(8, 16, 32); for (int i = 0; i < numberOfVoxSceneElements; i++) { voxSceneElementMatID[i] = escapedValue(4, 8, 32); voxSceneElementPosPackedS[i]; NbitsVoxauimsbf voxSceneElementPosPackedE[i]; NbitsVoxauimsbf } } Table 1: Syntax of voxSceneElements() Syntax elements in Table 1 may be defined as follows. voxSceneElements() provides a compact representation of a 3D voxel grid for a 3-dimensional audio scene. A SceneElement is defined by 2 points (x,y,z-indices) on the voxel grid and a corresponding value. The two points span a cuboid and all covered grid cells are assigned the value voxSceneElementMatID. The bitstream element numberOfVoxSceneElements signifies the number of SceneElements. The bitstream element voxSceneElementMatID signifies the integer value indicating the material type of the grid cells of a SceneElement. The bitstream element voxSceneElementPosPackedS signifies a packed representation of 3 indices of the start grid cell of a SceneElement. It may be an array that illustrates a collection of all start grid cells. The bitstream element voxSceneElementPosPackedE signifies a packed representation of 3 indices of the end grid cell of a SceneElement. It may be an array that illustrates a collection of all end grid cells.Both the voxSceneElementPosPackedS and voxSceneElementPosPackedE are useful because they allow for a compact representation of the data where a single voxSceneElementMatID is used for all grid cells between these two variables. The uncompressed voxel scene data is represented by the 3D matrix variable VoxDataMatrix is defined by the payload element voxSceneElements() using the following pseudo code: for (int i = 0; i < numberOfVoxSceneElements; i++) { PackedPosS[i] = voxSceneElementPosPackedS[i] PackedPosE[i] = voxSceneElementPosPackedS[i] for (int j = 0; j < 3; j++) { (voxSceneElementPosS[i][j], PackedPosS[i]) = FunctionUnpackPos(PackedPosS[i], voxSceneDimensions[j]) (voxSceneElementPosE[i][j], PackedPosE[i]) = FunctionUnpackPos(PackedPosE[i], voxSceneDimensions[j]) start[j] = voxSceneElementPosS[i][j] end[j] = voxSceneElementPosE[i][j] } for (int x = start[0]; x <= end[0]; x++) { for (int y = start[1]; y <= end[1]; y++) { for (int z = start[2]; z <= end[2]; z++) { VoxDataMatrix[x][y][z] = voxSceneElementMatID[i] } } } } where (UnpackedPos, PackedData) = FunctionUnpackPos(PackedData, Dimension) { UnpackedPos = mod(PackedData, Dimension) + 1 PackedData = floor(PackedData / Dimension)Annex A.2 The conversion process from CDto MD(e.g., decompression) is given in the MPEG-I Immersive Audio specification as described below. Syntax No. of bits Mnemonic voxSceneDiffractionMap() { numberOfVoxDiffractionMapElements = escapedValue(8, 16, 32); for (int i = 0; i < numberOfVoxDiffractionMapElements; i++) { voxDiffractionMapValue[i]; 1 bslbf voxDiffractionMapPosPackedS[i]; NbitsMapauimsbf voxDiffractionMapPosPackedE[i]; NbitsMapauimsbf } voxSceneDiffractionPreComputedPathData(); } Table 2: Syntax of voxSceneDiffractionMap() Syntax elements in Table 2 may be defined as follows. voxSceneDiffractionMap() provides a compact representation of a 2D diffraction map (VoxDataDiffractionMap). This 2D representation is similar to the 3D representation used for the voxel-based 3D audio scene. A MapElement is defined by 2 points (x,y-indices) on the diffraction map and a corresponding value. The two points span a rectangle and all covered grid cells are assigned the value voxDiffractionMapValue. The bitstream element numberOfVoxDiffractionMapElements signifies the number of MapElements. The bitstream element voxDiffractionMapValue signifies the binary value controlling the path finding algorithm. It is useful because the value indicates whether a path can go through the grid cell or not. This value is defined for all entries on the diffraction map. The bitstream element voxDiffractionMapPosPackedS signifies a packed representation of 2 indices of the start grid cell of a MapElement. It may be an array that illustrates a collection of all start grid cells.The bitstream element voxDiffractionMapPosPackedE signifies a packed representation of 2 indices of the end grid cell of a MapElement. It may be an array that illustrates a collection of all end grid cells. Both the voxDiffractionMapPosPackedS and voxDiffractionMapPosPackedE are useful because they allow for a compact representation of the data where a single voxDiffractionMapValue is used for all grid cells between these two variables. The same approach as described in Annex A.1 can be used for uncompressing of the 2D marix variableVoxDataDiffractionMap used for the diffraction modelling processing stage. If the 2D marix variable VoxDataDiffractionMap is specified by the encoder (i.e., voxSceneDiffractionMapPresentFlag == 1), it is defined by the bitstream payload element voxSceneDiffractionMap(); otherwise (i.e., voxSceneDiffractionMapPresentFlag == 0), it is defined by the bitstream payload element voxSceneElements() using the following pseudo code: if (voxSceneDiffractionMapPresentFlag == 1) { for (int i = 0; i < numberOfVoxDiffractionMapElements; i++) { PackedPosS[i] = voxDiffractionMapPosPackedS[i] PackedPosE[i] = voxDiffractionMapPosPackedE[i] for (int j = 0; j < 2; j++) { (voxDiffractionMapPosS[i][j], PackedPosS[i]) = FunctionUnpackPos(PackedPosS[i], voxSceneDimensions[j]) (voxDiffractionMapPosE[i][j], PackedPosE[i]) = FunctionUnpackPos(PackedPosE[i], voxSceneDimensions[j]) start[j] = voxDiffractionMapPosS[i][j] end[j] = voxDiffractionMapPosE[i][j] } for (int x = start[0]; x <= end[0]; x++) { for (int y = start[1]; y <= end[1]; y++) { VoxDataDiffractionMap[x][y] = voxDiffractionMapValue[i] } } } } else { if (voxSceneDiffractionMapHeightPresentFlag) { H = voxSceneDiffractionMapHeight } else {H = round(0.5*voxSceneDimensions[2]) } for (int x = 0; x < voxSceneDimensions[0]; x++) { for (int y = 0; y < voxSceneDimensions[1]; y++) { VoxDataDiffractionMap[x][y] = VoxDataMatrix[x][y][H] } } }Annex B Examples of C ↔M conversion process results are shown below for a 3-dimensional voxel matrix MVand a 2-dimensional diffraction map MD. M_V(:,:,1) = M_V(:,:,2) = M_V(:,:,3) = M_V(:,:,4) = 0 1 1 1 0 1 1 1 0 1 1 1 0 1 1 1 0 1 1 1 0 1 1 1 0 1 1 1 0 1 1 1 0 1 1 1 1 1 1 1 0 1 1 1 0 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 C_V = 1 1 2 1 3 4 4 1 3 1 2 3 1 2 M_D(:,:) = 0 1 1 1 0 1 1 1 0 1 1 1 0 0 0 0 C_D = 1 1 2 3 4Annex C Techniques according to embodiments of the present disclosure can convert a voxel matrix M to metadata C comprising a number of lines (or vectors), each line corresponding to a representation of a set of entries (or voxels) as defined in the disclosure. Simulation results show that the metadata C is much smaller in size than the initial voxel matrix M. Examples of simulation results are shown in Table 3 below, where the first column relates to a preconfigured example scene, the second column relates to a time for conversion using a state-of-the-art computing device with typical computation capabilities, the third column indicates a number of lines in the metadata C, and the fourth column indicates a number of voxels of the respective example scene. Scene Time (sec) Number of lines Number of voxels Hospital 0.002 20 33264 Maze 0.006 73 65340 Battle 0.02 356 74250 ParkingLot 0.06 506 112903 Cathedral (1.0m) 0.2 1361 818928 Cathedral (0.4m) 3.0 3037 6.55142e+06 Worst-Case-Scene 9.6 2528 1.67772e+07 size = 256^3 blocks = 1024 materials = 8 Table 3: Simulation results

Claims

CLAIMS 1. A method of processing a voxel-based scene representation of an audio scene, wherein the voxel-based scene representation comprises a voxel matrix and wherein entries of the voxel matrix are associated with respective voxels in the audio scene and take values from a set of values relating to respective material identifiers of the voxels, the method comprising, for a given value in the set of values: generating a binary matrix with entries indicating whether a corresponding entry in the voxel matrix has the given value; selecting an entry of the binary matrix that indicates that the corresponding entry in the voxel matrix has the given value; initializing a set of entries of the binary matrix with the selected entry; iteratively selecting dimensions from previously unselected dimensions of the binary matrix and updating the set of entries based on the selected dimensions; and generating a representation of the resulting set of entries for encoding or rendering, wherein the method further comprises, at each iteration: determining a respective maximum range along each of the previously unselected dimensions relative to a given entry in the binary matrix, so that for each entry in the set of entries, all entries within the maximum range relative to the respective entry indicate that the corresponding entries in the voxel matrix have the given value; selecting that dimension among the previously unselected dimensions that yields the largest maximum range; and updating the set of entries by adding, for each entry in the set of entries, all entries of the binary matrix within the maximum range for the selected dimension relative to the respective entry.

2. The method according to claim 1, wherein the representation of the resulting set of entries is indicative of a start entry of the resulting set of entries, and end entry of the resulting set of entries, and the material identifier for the entries in the set of entries.

3. The method according to claim 1 or 2, further comprising:outputting the generated representation for encoding or rendering.

4. The method according to any one of claims 1 to 3, wherein the entry of the binary matrix is selected as the first entry in a predetermined scan order along the entries of the binary matrix that indicates that the corresponding entry in the voxel matrix has the given value.

5. The method according to any one of claims 1 to 4, further comprising: setting entries in the binary matrix that are included in the resulting set of entries for the given value in the set of values to a value indicating that the corresponding entries in the voxel matrix do not have the given value.

6. The method according to claim 5, further comprising: selecting another entry in the binary matrix that indicates that the corresponding entry in the voxel matrix has the given value.

7. The method according to claim 6, wherein entries of the binary matrix are selected for iterative processing until all entries of the binary matrix that indicate that the corresponding entries in the voxel matrix have the given value have been assigned to sets of entries.

8. The method according to claim 6 or 7, further comprising: outputting a set of representations of sets of entries for encoding or rendering.

9. The method according to any one of claims 1 to 8, further comprising: processing all values in the set of values according to a fixed processing order.

10. The method according to any one of claims 1 to 9, wherein the method is performed at an audio rendering device and the output of the method is used for input to an audio rendering process.

11. The method according to any one of claims 1 to 10, wherein the voxel matrix is a 2- dimensional matrix.

12. The method according to any one of claims 1 to 11, wherein the voxel matrix relates to an audio diffraction map.

13. The method according to any one of claims 1 to 10, wherein the voxel matrix is a 3- dimensional matrix.

14. The method according to any one of claims 1 to 10 or 13, wherein the voxel matrix relates to a 3-dimensional representation of the audio scene.

15. An audio rendering apparatus comprising a processor and a memory coupled to the processor and storing instructions for the processor, wherein the processor is configured to perform the method according to any one of claims 1 to 14.

16. An audio encoder apparatus comprising a processor and a memory coupled to the processor and storing instructions for the processor, wherein the processor is configured to perform the method according to any one of claims 1 to 10, 13, or 14.

17. A program comprising instructions that, when executed by a processor, cause the processor to carry out the method according to any one of claims 1 to 14.

18. A computer-readable storage medium storing the program of claim 17.

Images