Methods, systems and apparatus for acoustic 3D extent modeling for voxel-based geometry representations

Abstract

Described herein is a method of rendering audio in an audio scene. The method comprises receiving a voxel-based audio scene representation of the audio scene, the audio scene representation including an indication of extent voxels representing a 3D extent together with a plurality of audio source signals for audio sources associated with the 3D extent; obtaining coordinates of an intersection point inside the 3D extent; determining one or more line- segments running through the intersection point and extending along respective coordinate directions of the audio scene representation, wherein end points of each line segment are determined based on coordinates of one or more of the extent voxels; and allocating audio sources among the plurality of audio sources to audio source locations within the audio scene based on the one or more line-segments. Further described are a respective apparatus and computer program product.

Description

a set of 3D grid indices VOX (representing set of 3D extent) a set of M audio signals (mono, stereo, etc.) Modelling settings: minimal distance Lmin between two ‘joint’ point audio sources mapping matrix F to assign audio signals to obtained point source position (and gains) reference distance Find number of point audio sources N = [1, ..., 6] and their coordinates P1>->N map audio signals S1,.,M to positions P1,.,N and gains 20 Solution Determine (step S102) the 3D extent center representation coordinates Cx,y,z of the voxelbased 3D extent representations VOXx,y,z, e.g., using the “min / max (geometric center)” approach: Cx,y,z = round(|Vmaxx,y,z - Vmmx,y,z| / 2) 25                    where    Vmaxx,y,z = max(VOXx,y,z),    Vminx,y,z = min(VOXx,y,z) (Note: it is also possible to use different center representation such as the “center of mass” point (centroid)). See the example of Figure 2. Center representation is needed to extract three characteristic dimensions from threedimensional 3D extent representation. 2023290448   04 May 2026 Determine (step S103) 3D extent characteristic dimension representation by the following extreme points Dmaxx,y,z and Dmin x,y,z: Dmaxx = [Vm xx, Cy, Cz], Dminx = [Vmi nx, Cy, Cz], Dmaxy = [Cx, Vmaxy, Cz], Dminy = [Cx, Vminy, Cz], Dmaxz = [Cx, Cy, Vmaxz], Dminz = [Cx, Cy, Vminz]. 15 (Note: it is also possible to use different characteristic dimension representation such as the offsets from the center representation). Determine (step S104) 3D extent joint point source coordinates Pmaxx,y,z and Pminx,y,z: where Pmaxx = [pmaxx, Cy, Cz] is the voxel closest to Dmaxx on the line [Dmaxx, Dminx) which is not an “occluder” voxel for this audio object where pminx = [pminx, Cy, Cz] is the voxel closest to Dminx on the line (Dmaxx, Dminx] and which is not an “occluder” voxel for this audio object The same procedure can be applied to get Pmaxy,z and Pminy,z . A “not an occluder” voxel can be defined to be either Pminx,y,z G VOXx,y,z (Figure 4) or Pmaxx,y,z £ VOXx,y,z || AIRx,y,z (Figure 3), where AIRx,y,z is a sound transmission media such as air and / or water. Calculate the number N = [1, ..., 6] of point audio sources by considering 3 variables: Wx,y,z = |Pmaxx,y,z- Pminx,y,z| The ‘x-‘, ‘y-‘, ‘z-’ pair of audio sources Pmaxx,y,z and Pminx,y,z is considered for 3D extent modelling, if Wx,y,z > &min. This is done to prevent phasing audio artifact caused by two 20 correlated audio signals rendered too close to each other. If all Wx,y,z < ^mn, then 3D extent is not modelled and the audio object is represented by a single audio point source positioned in the voxel Cx,y,z. Obtain, based on the bitstream payload, a mapping of M audio signals S1’"^ to position coordinates P1,",6, such as the example mapping given in Table 1 above. For example, the 25 mapping M may be read or extracted from the bitstream in some implementations. Assign appropriate signal gains based on the number of selected audio sources (to ensure energy preservation). Render selected subset of point audio sources {Pmaxx,y,z, Pminx,y,z} applying voxel-based occlusion and diffraction modelling (see Fig. 10 as an example) 2023290448   04 May 2026 Apply reference distance handling, i.e., subtract from the listener-to-object distance (L, P) the distance from the listener L to the closest point R of extent VOX, where |L - R| == min(|L -VOX|) Render selected subset of point audio sources {Pmaxx,y,z, Pminx,y,z} applying voxel-based 5 occlusion and diffraction modelling. The rendering may be performed by a renderer capable of simulating acoustic occlusion and diffraction modelling. Figure 10 shows a 3D extent representation of an object, i.e., tram, occluded by an obstacle as an example of how the diffraction processing may be applied to the methods described herein. The points in the middle are occluded and, as a result, a set of virtual points is generated by means of diffraction modeling. 513 and 514 are the view lines which are not obstructed by the occluder 512. 515 is the direction (azimuth) from which the objects 518 are going to be perceived. These objects are then perceived (modelled) as 516. Consequently, those coordinates 504a, 504b, 506a, 506b (missing in the Fig) belonging to 518 are modelled 15 by 516a, 516d, 516b, 516c belonging to 516. The 3D extent modeling method described assumes the application of diffraction modeling. However, the method can also be used without the application of diffraction modeling. In this case, the method is applied to the 3D extent subset visible to the listener. The following methods can be used to obtain the subset “visible” (visible implies the absence of acoustic occluder between the listener and the 20 corresponding point) to the listener: ray tracing based methods or by checking occlusion on the line between the listener and a subset of 3D extent representation. This subset can be determined by a Monte Carlo or any other sub-sampling methods. Referring to the example of Figure 11, an apparatus 1100 including one or more processors 1101, 1102 according to embodiments of the disclosure is illustrated. The one or more 25 processors 1101, 1102 may be configured to carry out the methods described herein. A computing device implementing the techniques described above can have the following example architecture. Other architectures are possible, including architectures with more or fewer components. In some implementations, the example architecture includes one or more processors (e.g., dual-core Intel® Processors), one or more output devices (e.g., LCD), one or 30 more network interfaces, one or more input devices (e.g., mouse, keyboard, touch-sensitive display) and one or more computer-readable mediums (e.g., RAM, ROM, SDRAM, hard 2023290448   04 May 2026 disk, optical disk, flash memory, etc.). These components can exchange communications and data over one or more communication channels (e.g., buses), which can utilize various hardware and software for facilitating the transfer of data and control signals between components. 5   The term “computer-readable medium” refers to a medium that participates in providing instructions to processor for execution, including without limitation, non-volatile media (e.g., optical or magnetic disks), volatile media (e.g., memory) and transmission media. Transmission media includes, without limitation, coaxial cables, copper wire and fiber optics. Computer-readable medium can further include operating system (e.g., a Linux® operating 0 system), network communication module, audio interface manager, audio processing manager and live content distributor. Operating system can be multi-user, multiprocessing, multitasking, multithreading, real time, etc. Operating system performs basic tasks, including but not limited to: recognizing input from and providing output to network interfaces and / or devices; keeping track and managing files and directories on computer-readable mediums 15   (e.g., memory or a storage device); controlling peripheral devices; and managing traffic on the one or more communication channels. Network communications module includes various components for establishing and maintaining network connections (e.g., software for implementing communication protocols, such as TCP / IP, HTTP, etc.). Architecture can be implemented in a parallel processing or peer-to-peer infrastructure or on 20 a single device with one or more processors. Software can include multiple software components or can be a single body of code. The described features can be implemented advantageously in one or more computer programs that are executable on a programmable system including at least one programmable processor coupled to receive data and instructions from, and to transmit data and instructions 25 to, a data storage system, at least one input device, and at least one output device. A computer program is a set of instructions that can be used, directly or indirectly, in a computer to perform a certain activity or bring about a certain result. A computer program can be written in any form of programming language (e.g., Objective-C, Java), including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program 30 or as a module, component, subroutine, a browser-based web application, or other unit suitable for use in a computing environment. Suitable processors for the execution of a program of instructions include, by way of 2023290448   04 May 2026 example, both general and special purpose microprocessors, and the sole processor or one of multiple processors or cores, of any kind of computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for executing instructions and one or more 5 memories for storing instructions and data. Generally, a computer will also include, or be operatively coupled to communicate with, one or more mass storage devices for storing data files; such devices include magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and optical disks. Storage devices suitable for tangibly embodying computer program instructions and data include all forms of non-volatile memory, including 0 by way of example semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks and removable disks; magnetooptical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, ASICs (application-specific integrated circuits). To provide for interaction with a user, the features can be implemented on a computer having 15 a display device such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor or a retina display device for displaying information to the user. The computer can have a touch surface input device (e.g., a touch screen) or a keyboard and a pointing device such as a mouse or a trackball by which the user can provide input to the computer. The computer can have a voice input device for receiving voice commands from the user. 20 The features can be implemented in a computer system that includes a back-end component, such as a data server, or that includes a middleware component, such as an application server or an Internet server, or that includes a front-end component, such as a client computer having a graphical user interface or an Internet browser, or any combination of them. The components of the system can be connected by any form or medium of digital data 25 communication such as a communication network. Examples of communication networks include, e.g., a LAN, a WAN, and the computers and networks forming the Internet. The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the 30 respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data (e.g., an HTML page) to a client device (e.g., for purposes of displaying data to and receiving user input from a user interacting with the client 2023290448   04 May 2026 device). Data generated at the client device (e.g., a result of the user interaction) can be received from the client device at the server. A system of one or more computers can be configured to perform particular actions by virtue of having software, firmware, hardware, or a combination of them installed on the system 5 that in operation causes or cause the system to perform the actions. One or more computer programs can be configured to perform particular actions by virtue of including instructions that, when executed by data processing apparatus, cause the apparatus to perform the actions. While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any inventions or of what may be claimed, but rather 0 as descriptions of features specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable sub-combination. Moreover, although features 15 may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination. Similarly, while operations are depicted in the drawings in a particular order, this should not 20 be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be 25 understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products. Unless specifically stated otherwise, as apparent from the following discussions, it is appreciated that throughout the present disclosure discussions utilizing terms such as “processing”, “computing”, “calculating”, “determining”, “analyzing” or the like, refer to the 30 action and / or processes of a computer or computing system, or similar electronic computing devices, that manipulate and / or transform data represented as physical, such as electronic, quantities into other data similarly represented as physical quantities. 2023290448   04 May 2026 Reference throughout this disclosure to “one example embodiment”, “some example embodiments” or “an example embodiment” means that a particular feature, structure or characteristic described in connection with the example embodiment is included in at least one example embodiment of the present disclosure. Thus, appearances of the phrases “in one 5 example embodiment”, “in some example embodiments” or “in an example embodiment” in various places throughout this disclosure are not necessarily all referring to the same example embodiment. Furthermore, the particular features, structures or characteristics may be combined in any suitable manner, as would be apparent to one of ordinary skill in the art from this disclosure, in one or more example embodiments. As used herein, unless otherwise specified the use of the ordinal adjectives “first”, “second”, “third”, etc., to describe a common object, merely indicate that different instances of like objects are being referred to and are not intended to imply that the objects so described must be in a given sequence, either temporally, spatially, in ranking, or in any other manner. Also, it is to be understood that the phraseology and terminology used herein are for the 15 purpose of description and should not be regarded as limiting. The use of “including,” “comprising,” or “having” and variations thereof are meant to encompass the items listed thereafter and equivalents thereof as well as additional items. Unless specified or limited otherwise, the terms “mounted”, “connected”, “supported”, and “coupled” and variations thereof are used broadly and encompass both direct and indirect mountings, connections, 20 supports, and couplings. In the claims below and the description herein, any one of the terms comprising, comprised of or which comprises is an open term that means including at least the elements / features that follow, but not excluding others. Thus, the term comprising, when used in the claims, should not be interpreted as being limitative to the means or elements or steps listed thereafter. For 25 example, the scope of the expression a device comprising A and B should not be limited to devices consisting only of elements A and B. Any one of the terms including or which includes or that includes as used herein is also an open term that also means including at least the elements / features that follow the term, but not excluding others. Thus, including is synonymous with and means comprising. 30 It should be appreciated that in the above description of example embodiments of the present disclosure, various features of the present disclosure are sometimes grouped together in a single example embodiment, Fig., or description thereof for the purpose of streamlining the 2023290448   04 May 2026 present disclosure and aiding in the understanding of one or more of the various inventive aspects. This method of disclosure, however, is not to be interpreted as reflecting an intention that the claims require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing 5 disclosed example embodiment. Thus, the claims following the Description are hereby expressly incorporated into this Description, with each claim standing on its own as a separate example embodiment of this disclosure. Furthermore, while some example embodiments described herein include some, but not other features included in other example embodiments, combinations of features of different example embodiments are meant to be within the scope of the present disclosure, and form different example embodiments, as would be understood by those skilled in the art. For example, in the following claims, any of the claimed example embodiments can be used in any combination. In the description provided herein, numerous specific details are set forth. However, it is 15 understood that example embodiments of the present disclosure may be practiced without these specific details. In other instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description. Thus, while there has been described what are believed to be the best modes of the present disclosure, those skilled in the art will recognize that other and further modifications may be 20 made thereto without departing from the spirit of the present disclosure, and it is intended to claim all such changes and modifications as fall within the scope of the present disclosure. For example, any formulas given above are merely representative of procedures that may be used. Functionality may be added or deleted from the block diagrams and operations may be interchanged among functional blocks. Steps may be added or deleted to methods described 25 within the scope of the present disclosure. Various aspects and implementations of the present disclosure may also be appreciated from the following enumerated example embodiments (EEEs), which are not claims. EEE1. A method of modelling extended audio objects for audio rendering in a virtual or augmented reality environment, the method comprising: 30          determining a 3D extent center representation of a voxel based 3D extent representation; determining 3D extent characteristic dimension representation based on the 3D extent 2023290448   04 May 2026 15 representation; determining 3D extent joint point source coordinates based on the 3D extent characteristic dimension representation or the 3D extent center representation; EEE2. The method of EEE1, further comprising receiving a mapping of M audio signals to position coordinates; and assigning signal gains of the M audio signals to the point sources. EEE3. The method of EEE1, further comprising rendering the point sources based on voxelbased occlusion and diffraction modeling. EEE4. The method of EEE1, wherein the center representation may be determined based on the geometric center of voxels or by the centroid approach. EEE5. The method of EEE1, wherein dimension representation may be based on extreme points or by calculating corresponding offsets from the center. EEE6. The method of EEEs 1-5, wherein the method is applied to a subset of the voxel based 3D extent representation. EEE7. The method of EEE6, wherein the subset of the voxel based 3D extent representation corresponds to acoustically non-occluded (visible) voxels. EEE8. A non-transitory computer program comprising instructions that, when executed by a processor, cause the processor to carry out the method according to any one of EEEs 1-7. EEE9. An apparatus configured to perform the method of EEEs 1-7.

Claims

1. A method of rendering audio in an audio scene, the method comprising:receiving a voxel-based audio scene representation of the audio scene, the audio scene representation including an indication of extent voxels representing a 3D extent together with a plurality of audio source signals for audio sources associated with the 3D extent;obtaining coordinates of an intersection point inside the 3D extent;determining one or more line-segments running through the intersection point and extending along respective coordinate directions of the audio scene representation, wherein end points of each line segment are determined based on coordinates of one or more of the extent voxels; andallocating audio sources among the plurality of audio source signals to audio source locations within the audio scene based on the one or more line-segments.

2. The method of claim 1, wherein the intersection point is one of the geometric center of the 3D extent and the center of gravity of the 3D extent.

3. The method of claim 1 or 2, wherein end points of each line segment are determined based on extremal coordinate values of the 3D extent along respective coordinate directions, such that lengths of the line segments correspond to maximum dimensions of projections of the 3D extent onto respective coordinate directions.

4. The method of any of claims 1 to 3, wherein the audio scene representation further indicates occluder voxels; andwherein allocating the audio sources includes allocating the audio sources to coordinates within voxels other than the occluder voxels.

5. The method of claim 4, wherein the audio scene representation further indicates unfilled voxels; andwherein allocating the audio sources includes allocating the audio sources to coordinates on respective line segments that are closest to the end points of the respective line segments and that are within extent voxels or unfilled voxels.

6. The method of any of claims 1 to 4, wherein allocating the audio sources further2023290448   04 May 2026includes determining one or more possible target locations for allocating the audio sources, based on the line segments.

7. The method of claim 6, wherein the audio scene representation further indicates unfilled voxels; andwherein determining the one or more possible target locations includes selecting coordinates for the one or more possible target locations that are closest to the end points of the respective line segments and that are within extent voxels or unfilled voxels.

8. The method of claim 6 or 7, wherein determining the one or more possible target locations includes selecting coordinates for the one or more possible target locations that are closest to the end points of the respective line segments and that are within extent voxels.

9. The method of any of claims 6 to 8, wherein the method further includes:selecting the audio source locations from the possible target locations based on a predefined minimum distance between audio sources; andallocating the audio sources among the plurality of audio source signals to the selected audio source locations.

10. The method of any of claims 1 to 9, further including obtaining a mapping indicating an assignment of the plurality of audio source signals to the audio source locations.

11. The method of claim 10 further including assigning gains to the audio source locations based at least in part on the mapping.

12. The method of any of claims 1 to 11, wherein the method further includes:obtaining coordinates of a listener location; andrendering the plurality of audio source signals of the allocated audio sources based on a reference distance between the listener position and the 3D extent.

13. The method of claim 12, wherein the rendering further includes rendering the plurality of audio source signals based on occlusion and diffraction modeling.

14. An apparatus for rendering audio in a voxel-based audio scene representation,2023290448   04 May 202615wherein the apparatus includes one or more processors configured to carry out a method including:receiving a voxel-based audio scene representation of the audio scene, the audio scene representation including an indication of extent voxels representing a 3D extent together with a plurality of audio source signals for audio sources associated with the 3D extent;obtaining coordinates of an intersection point inside the 3D extent;determining one or more line-segments running through the intersection point and extending along respective coordinate directions of the audio scene representation, wherein end points of each line segment are determined based on coordinates of one or more of the extent voxels; andallocating audio sources among the plurality of audio source signals to audio source locations within the audio scene based on the one or more line-segments.

15. A program comprising instructions that, when executed by a processor, cause the processor to carry out the method according to any one of claims 1 to 13.

16. A computer-readable storage medium storing the program according to claim 15.

Images