Method, system and apparatus for acoustic 3d range modeling of voxel-based geometric representations

Abstract

The present disclosure describes a method of rendering audio in an audio scene. The method comprises: receiving a voxel-based audio scene representation of an audio scene, the audio scene representation comprising an indication of range voxels representing a 3D range and a plurality of audio source signals of audio sources associated with the 3D range; obtaining coordinates of intersection points in the 3D range; determining one or more line segments that pass through the intersection and extend in respective coordinate directions of the audio scene representation, where an endpoint of each line segment is determined based on coordinates of one or more of the range voxels; and assigning an audio source of the plurality of audio sources to an audio source orientation within the audio scene based on the one or more line segments. A corresponding apparatus and computer program product are also described.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Application No. 63 / 352,360, filed on June 15, 2022, and U.S. Provisional Application No. 63 / 441,120, filed on January 25, 2023, all of which are incorporated herein by reference in their entirety. Technical Field

[0003] The present disclosure generally relates to a method for rendering audio in an audio scene, in particular to a method for rendering audio based on a voxel-based audio scene representation of the audio scene. The present disclosure also relates to a corresponding apparatus and a computer program product.

[0004] Although some embodiments will be described herein with particular reference to the present disclosure, it will be appreciated that the present disclosure is not limited to this field of use, but may be applied in a broader context. Background Art

[0005] Any discussion of the background art throughout the present disclosure should in no way be considered as an admission that such art is widely known or forms part of the common general knowledge in the art.

[0006] The Moving Picture Experts Group (MPEG) is a consortium of working groups jointly established by the International Organization for Standardization (ISO) and the International Electrotechnical Commission (IEC) to develop standards for media coding, including audio coding. MPEG is organized under ISO / IEC SC 29, and the audio group is currently identified as Working Group (WG) 6. WG 6 is currently working on the MPEG-I Audio standard.

[0007] The new MPEG-I standard enables auditory experiences from different viewpoints and / or perspectives or listening positions by supporting various motions of scenes and around such scenes, such as using motions of various degrees of freedom, such as three degrees of freedom (3DOF) or six degrees of freedom (6DoF) in virtual reality (VR), augmented reality (AR), mixed reality (MR) and / or extended reality (XR) applications. 6DoF interaction extends the 3DoF spherical video / audio experience, which is limited to head rotation (pitch, yaw and roll), to include translational motion (forward / backward, up / down and left / right), to enable roaming within a virtual environment (e.g., physically walking around a room) in addition to head rotation.

[0008] For audio rendering in VR, AR, MR, and XR applications, object-based approaches have been widely adopted by representing complex auditory scenes as multiple individual audio objects, each of which is associated with parameters or metadata that define the object's position / location and trajectory in the scene. Alternatively, audio rendering in such environments also uses Higher Order Ambisonics (HOA).

[0009] Audio objects are typically represented as point sources (i.e., without extent). As used herein, an audio source with "extent" is an audio source waveform (or waveforms) associated with a spatial region (where the region is larger than a point). For example, a piano can be represented as an audio source (or sources) with a cubic extent (e.g., stereo or mono L / R), rather than just a point source. The use of extents improves the user's audio experience when, for example, the user is around a virtual piano object in a VR, AR, MR, or XR environment. In this example, the extent of the piano represented for audio rendering does not need to have the exact same physical details as a real piano.

[0010] To reflect the acoustic effects of audio objects with range, such audio objects can be represented by voxel-based geometry. Voxels used for audio rendering are relevant to media environments (such as video games and / or VR, AR, MR, and XR environments) implemented in both hardware and software.

[0011] However, there is still a need for improvements in the rendering of 3D-scale acoustic effects represented by voxel-based geometry, and in particular, there may be a great need to simplify the process and reduce the computational burden. Summary of the invention

[0012] In view of the above, the present disclosure provides a method, an apparatus, a program, and a computer-readable storage medium for rendering audio in an audio scene, which have the features of the corresponding independent claims.

[0013] According to a first aspect of the present disclosure, a method for rendering audio in an audio scene is provided. The method may include receiving a voxel-based audio scene representation of an audio scene. The audio scene representation may include an indication of a range voxel representing a 3D range and a plurality of audio source signals of an audio source associated with the 3D range. The method may further include obtaining (e.g., determining, calculating) the coordinates of an intersection within the 3D range. The method may further include determining one or more line segments that pass through the intersection and extend along the corresponding coordinate directions of the audio scene representation. The endpoints of each line segment may be determined based on the coordinates of one or more range voxels in the range voxels. And the method may include assigning an audio source from a plurality of audio sources to an audio source position within the audio scene based on the one or more line segments.

[0014] In some embodiments, the intersection point may be one of a geometric center of the 3D extent and a center of gravity of the 3D extent.

[0015] In some embodiments, the endpoints of each line segment may be determined based on the extreme coordinate values ​​of the 3D range along the corresponding coordinate direction, such that the length of the line segment corresponds to the maximum size of the projection of the 3D range in the corresponding coordinate direction.

[0016] In some embodiments, the audio scene representation may further indicate occluding voxels.Assigning audio sources may include assigning audio sources to coordinates within voxels other than occluding voxels.

[0017] In some embodiments, the audio scene representation may further indicate unfilled voxels (eg, air voxels).Assigning an audio source may include assigning an audio source to a coordinate on the corresponding line segment that is closest to an endpoint of the corresponding line segment and within a range voxel or an unfilled voxel.

[0018] In some embodiments, assigning the audio source may further include determining one or more possible target positions for assigning the audio source based on the line segment.

[0019] In some embodiments, the audio scene representation may further indicate unfilled voxels (eg, air voxels).Determining one or more possible target positions may include selecting coordinates for the one or more possible target positions that are closest to an endpoint of the corresponding line segment and that are within a range voxel or an unfilled voxel.

[0020] In some embodiments, determining one or more possible target positions may include selecting coordinates for the one or more possible target positions that are closest to an endpoint of a corresponding line segment and that are within a range voxel.

[0021] In some embodiments, the method may further include selecting an audio source position from possible target positions based on a predefined minimum distance between audio sources, and the method may include assigning an audio source from the plurality of audio sources to the selected audio source position.

[0022] In some embodiments, the method may further comprise obtaining a mapping indicating the assignment of audio source signals to audio source positions.

[0023] In some embodiments, the method may further include assigning gains to audio source locations based at least in part on the mapping.

[0024] In some embodiments, the method may further include obtaining coordinates of the listener position. And the method may include rendering the audio source signal of the assigned audio source based on a reference distance between the listener position and the 3D range.

[0025] In some embodiments, rendering may further include rendering the audio source signal based on occlusion and diffraction modeling.

[0026] According to a second aspect of the present disclosure, a device for rendering audio in a voxel-based audio scene representation is provided. The device may include one or more processors configured to perform a method, which may include receiving a voxel-based audio scene representation of an audio scene, the audio scene representation including an indication of a range voxel representing a 3D range and multiple audio source signals of an audio source associated with the 3D range. The method may further include obtaining the coordinates of an intersection within the 3D range. The method may further include determining one or more line segments that pass through the intersection and extend along the corresponding coordinate direction of the audio scene representation. The endpoints of each line segment may be determined based on the coordinates of one or more range voxels in the range voxel. And the method may include assigning an audio source from a plurality of audio sources to an audio source position within the audio scene based on the one or more line segments.

[0027] Various aspects of the present disclosure may be implemented via an apparatus. The apparatus may include a processor and a memory coupled to the processor. The processor may be suitable for executing the methods described in accordance with various aspects and embodiments of the present disclosure.

[0028] Various aspects of the present disclosure may be implemented via a program. When the instructions of the program are executed by a processor, the processor may execute various aspects and embodiments of the present disclosure. A computer-readable storage medium may store a program. Such a computer-readable storage medium may include a memory device as described herein, including but not limited to a random access memory (RAM) device, a read-only memory (ROM) device, etc. Therefore, some innovative aspects of the subject matter described in the present disclosure may be implemented via one or more computer-readable storage media having software stored thereon.

[0029] It should be understood that device features and method steps can be interchanged in a variety of ways. In particular, as will be appreciated by the skilled person, the details of the disclosed method(s) can be implemented by the corresponding device (or system), and vice versa. Furthermore, any of the above statements made with respect to method(s) is understood to apply equally to the corresponding device (or system), and vice versa. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Example embodiments of the present disclosure will now be described, by way of example only, with reference to the accompanying drawings, in which:

[0031] Figure 1 An example of a method for rendering audio in an audio scene according to an embodiment of the present disclosure is illustrated.

[0032] Figure 2 illustrates an example of a voxel-based audio scene representation of an audio scene according to an embodiment of the present disclosure,

[0033] Figure 3 An example of assigning audio sources to audio source positions within an audio scene according to an embodiment of the present disclosure is illustrated.

[0034] Figure 4 FIG. 4 illustrates another example of assigning audio sources to audio source positions within an audio scene according to an embodiment of the present disclosure.

[0035] Figures 5 to 9 An exemplary use case of an example of a method of rendering audio in an audio scene according to an embodiment of the present disclosure is illustrated,

[0036] Fig.10 illustrates an example of a reference distance between a listener position and a 3D range and an example of occlusion and diffraction modeling according to an embodiment of the present disclosure, and

[0037] Fig.11 An example of a device including one or more processors according to an embodiment of the present disclosure is illustrated.

[0038] In the accompanying drawings, when connecting elements such as solid or dotted lines or arrows are used to illustrate the connection, relationship or association between two or more other schematic elements, the absence of any such connecting elements does not mean that there can be no connection, relationship or association. In other words, some connections, relationships or associations between elements are not shown in the accompanying drawings to avoid blurring the present disclosure. In addition, for ease of illustration, a single connecting element is used to represent multiple connections, relationships or associations between elements. For example, in the case where a connecting element represents the communication of a signal, data or instruction, it will be understood by those skilled in the art that such elements represent one or more signal paths that may be needed to affect the communication. DETAILED DESCRIPTION

[0039] An audio source with a range is an audio source waveform (one or more) associated with a spatial region (larger than a point). The spatial region can be modeled by a geometric shape (2D or 3D). A voxel is a 3D volume representation, so it is possible to model such a geometric shape. The use of voxels for audio rendering is related to various media environments (such as video games and / or VR, AR, MR, and XR environments) implemented in both hardware and software. A voxel is a spatial volume with audio rendering instructions or acoustic properties assigned to it. The voxel size is an encoder configuration parameter, and it can be selected (manually or automatically) according to the geometric detail level of the scene (for example, in the range of 10cm-1m). Voxels for audio rendering can be obtained in the following ways:

[0040] Voxelize (or convert) a grid-based scene representation;

[0041] From the scene representation used for scene generation (or even video rendering), e.g. by downsampling to smaller sized voxels.

[0042] The methods and apparatus described herein relate to how to render acoustic effects of a 3D range when the 3D range is represented by voxel-based geometry. More specifically, the methods and apparatus described herein relate to how to obtain the coordinates of a "joint" (point) audio source.

[0043] Typically, more than one audio source is needed to model an audio source with a range to approximate the spatial region of the range. These (target) audio sources can be obtained from (one or more) given audio sources associated with the range specified by the scene creator using, for example, a scene description. As used herein, it can be said that the word "joint" implies that these target sources are related to each other because they represent the spatial region of the range in a certain dimension. Since there are three dimensions, at least one pair of audio sources is required for each dimension. As an example, the scene description specifies stereo channels with a cubic range to represent a virtual piano object. This can then be processed in the renderer to obtain three pairs of "joint" "target" audio sources placed at six different positions within the adjacent range.

[0044] That is, the method and apparatus described herein are intended to find (eg, select, determine) a corresponding number of (point) audio sources (eg, N = [1, ..., 6]) and their coordinates (orientations) P 1,…,N, and based on a given scene description (including, for example, the listener position coordinates L, a 3D extent material ID (representing a geometric approximation of the 3D extent of an audio object), a set of 3D mesh indices VOX (representing a set of 3D extents), and a set of M audio signals (mono, stereo, etc.)) and a modeling setup (including, for example, the minimum distance Δ between two “joint” (point) audio sources min , a mapping matrix F for assigning audio signals to the obtained point source positions (and gains), and a reference distance) to convert the audio signal S 1,…,M Map to the corresponding position P 1,…,N and gain.

[0045] The methods and apparatus described herein allow modeling of audio objects having extents represented by voxel-based geometry without explicitly signaling the audio source coordinates (e.g., without explicitly sending and receiving this information in the bitstream). That is, assuming that the extents are represented by voxel-based geometry, it can be said that the methods and apparatus described herein emphasize a way of determining "joint" audio source coordinates (positions) within a neighboring extent. The resulting orientation / coordinates are voxel coordinates. Since these coordinates are calculated on the renderer side, these coordinates do not need to be known in advance and do not need to be explicitly signaled / sent.

[0046] Advantageously, this allows automatic acquisition of signal audio source coordinates for complex voxel-based 3D extent geometry at the decoder side, particularly when the decoder operates in a manner compliant with audio standards such as those set by MPEG. Another advantage is that this allows support for 3D extent geometry modifications at the decoder (without requiring re-encoding of the modified scene).

[0047] The encoding of the geometry of the 3D range is done in the encoder and sent to the decoder to deliver the information of the geometry of the range to the decoder / renderer. Like many other objects in the scene, the range can also be modified at the encoder or decoder / renderer side. Modifications at the encoder require "re-encoding" of the range to be sent to the decoder. This does not apply to modifications on the decoder / renderer side. Since the method described in this article is implemented on the decoder / renderer side, that is, any modifications to the range are done on the decoder / renderer side, no "re-encoding" is required.

[0048] How to represent a voxel-based audio scene?

[0049] Any voxel-based representation of an audio scene may include an indication of voxels that are not transmission voxels (e.g., they are occluding voxels) (i.e., voxels in which sound cannot propagate or cannot propagate freely - a representation of the geometry of the occlusion). The indication may involve an indication of the coordinates of the corresponding voxels (e.g., center coordinates, angular coordinates, etc.). For example, the coordinates of these voxels may be represented by a grid index. In addition, the voxel-based representation may include an indication of material properties (such as absorption coefficients, reflection coefficients, etc.) of voxels that are not transmission voxels. In addition to occluding voxels, the voxel-based representation may also indicate transmission voxels or unfilled voxels (e.g., air voxels), i.e., voxels in which sound can propagate - a representation of the sound propagation medium. Accordingly, for each voxel in a predefined portion of the space (e.g., within a boundary surrounding the audio scene), some embodiments of the voxel-based representation of the audio scene may include an indication of the corresponding material properties.

[0050] Methods for rendering audio in audio scenes

[0051] refer to Figure 1 , which illustrates an example of a method for rendering audio in an audio scene. The method is performed on the decoder / renderer side and can be implemented by the corresponding decoder / renderer. For example, all method steps can be performed in real time in a single device, which can be a VR / AR / MR / XR device.

[0052] exist Step S101 In one embodiment, a voxel-based audio scene representation of an audio scene is received. The audio scene representation includes an indication of a range voxel representing a 3D range and a plurality of audio source signals for an audio source associated with the 3D range. In other words, the 3D range can be said to correspond to an audio object having a range having a geometric form represented by the range voxel.

[0053] Figure 2 An example of a voxel-based audio scene representation of an audio scene is schematically illustrated in . Figure 2 An example of is a 2D cross-section view of a voxel-based representation of a 3D audio scene including a 3D extent. Figure 2 A grid pattern representing a voxelization of an audio scene representation is shown. Figure 2 In the example of , range voxels 205 and unfilled voxels (e.g., air voxels) 206 are indicated. That is, in addition to the range voxels representing the 3D range, the audio scene representation can also indicate voxels representing the acoustic environment of a portion of the 3D range. It can be said that the unfilled voxels represent the sound transmission medium. For example, the sound transmission medium can be air and / or water.

[0054] Reference again Figure 1 For example, Step S102In the example, the coordinates of the intersection point in the 3D range are obtained (eg, determined, calculated). In an embodiment, the intersection point may be one of the geometric center of the 3D range and the center of gravity of the 3D range. Figure 2 In the example of FIG. 2 , a geometric center 201 of a 3D range and a mass center (centroid) 202 of the 3D range, which may be used alternatively, are schematically illustrated.

[0055] In a manner not intended to be limiting, the intersection point may be taken as the origin O of a Cartesian coordinate system. Then, in the context of the example of a Cartesian coordinate system, the voxel-based 3D range representation VOX x,y,z The intersection point (3D range center) C x,y,z This can be determined using the "min / max" method as follows:

[0056] C x，y，z =round(|V max x，y，z -V min x，y，z | / 2)

[0057] Among them, V max x，y，z =max(VOX x，y，z ), V min x，y，z =min(VOX x，y，z )

[0058] Here, it should be understood that the above equations apply to coordinates x, y and z separately, ie there is one such equation for each coordinate. Note that the "centre of gravity" method or other methods may also be used.

[0059] Reference again Figure 1 For example, Step S103 In the embodiment of the present invention, one or more line segments are determined, wherein each line segment passes through the intersection point and extends along a corresponding coordinate direction (e.g., along the x-coordinate axis, the y-coordinate axis, and the z-coordinate axis) of the audio scene representation. The endpoints of each line segment are determined based on the coordinates of one or more range voxels in the range voxels. For example, as described in detail below, the endpoints of each line segment can be determined based on the extreme coordinate values ​​of the 3D range along the corresponding coordinate direction. That is, for example, for a line segment extending along the x-coordinate axis, the endpoints can be determined based on the extreme coordinates of the 3D range along the x-coordinate axis.

[0060] exist Figure 2In the example of , in the 2D cross section, two such line segments 203, 204 are illustrated as passing through the geometric center 201 of the 3D range and having corresponding endpoints 203a, 203b, 204a, 204b. In the case of a Cartesian coordinate system, the lines may be the X-axis, Y-axis, and Z-axis (assuming that the intersection point is taken as the origin of the coordinate system), and the line segments may be the line segments of the X-axis, Y-axis, and Z-axis. Figure 2 In the 2D cross-section of FIG. 2 , line 203 may be the Y-axis and line 204 may be the X-axis.

[0061] Reference again Figure 1 For example, Step S104 In step S104, an audio source from a plurality of audio sources is allocated to an audio source position within an audio scene based on one or more line segments. As used herein, it can be said that "allocated" means that a target audio source is generated (e.g., based on a given / specified audio source in a range) and is linked / mapped to a calculated coordinate position. That is, in step S104, a set of target audio sources can be output, which are placed at a calculated position near the range. These target sources (rather than given / specified audio sources with a range) can be used to replace the task of rendering "audio sources with a range" by rendering a set of point sources. One or more line segments are constructed to help determine the position of the target audio source. Note that S103 outputs these line segments.

[0062] Reference now Figure 3 and Figure 4 , which schematically illustrates two examples of assigning audio sources to audio source positions within an audio scene. That is, Figure 3 and Figure 4 308a, 308b, 309a, 309b indicate two possible implementations of method step S104. These implementations differ in the manner in which the target source position indicated by 308a, 308b, 309a, 309b is determined. It is worth noting that Figure 3 and Figure 4 The corresponding 2D cross-section is also shown.

[0063] Figure 3 and Figure 4 Illustrated are examples of indications of range voxels 305, unfilled voxels 306, and occlusion voxels 307. Occlusion voxels may represent acoustic occluders that exist between, for example, a 3D range and a listener.

[0064] In some embodiments, in step S103 , the endpoints of each line segment may be determined based on the extreme coordinate values ​​of the 3D range along the corresponding coordinate direction, so that the length of the line segment may correspond to the maximum size of the projection of the 3D range in the corresponding coordinate direction.

[0065] As described above, the line segments of the X-axis, Y-axis, and Z-axis are represented by corresponding line segments, and the intersection point in the 3D range can be used as the origin O of the Cartesian coordinate system. Therefore, in the context of the example of the Cartesian coordinate system, the maximum size (3D range characteristic size extreme point) D of the projection of the 3D range in the corresponding coordinate direction max x,y,z and D min x,y,z It can be determined (extracted) as follows:

[0066] D max x =[V max x , C y , C z ], D min x =[V min x , C y , C z ],

[0067] D max y =[C x , V max y , C z ], D min y =[C x , V min y , C z ],

[0068] D max z =[C x , C y , V max z ], D min z =[C x , C y , V min z ].

[0069] However, different characteristic size representations may also be used, such as offsets from a center representation.

[0070] Figure 3 and Figure 4 The corresponding maximum dimensions 303a, 303b and 304a, 304b are illustrated in the example of FIG.

[0071] In such Figure 3In some embodiments, in step S104, assigning the audio source (to the audio source position within the audio scene) may include assigning the audio source to a coordinate (e.g., 308a) within a voxel other than the occlusion voxel 307. The occlusion voxel 307 may affect the sound perceived by the listener at the corresponding listener position, and thus, assigning the audio source to the coordinate within the voxel other than the occlusion voxel 307 enables the rendering of the corresponding audio source signal of the assigned audio source, thereby making the sound perceived by the listener appear realistic.

[0072] Furthermore, in Figure 4 In some embodiments, assigning audio sources (to audio source positions within the audio scene) may further include assigning the audio sources to coordinates on corresponding line segments (e.g., Figure 4 308a, 308b, 309a, 309b in FIG. 3 ), these coordinates are closest to the endpoint of the corresponding line segment and are within the range voxel or unfilled voxel.

[0073] As an alternative or supplement to the above embodiments, assigning an audio source may further include determining one or more possible target locations for assigning the audio source based on the line segment. Then, determining the one or more possible target locations may include selecting coordinates for the one or more possible target locations that are closest to the endpoint of the corresponding line segment and are within the range voxel or the unfilled voxel. In a further embodiment, determining the one or more possible target locations may include selecting coordinates for the one or more possible target locations that are closest to the endpoint of the corresponding line segment and are within the range voxel.

[0074] Depending on the use case, assigning an audio source to a coordinate within a range voxel or selecting the corresponding coordinates that are closest to the endpoint of the corresponding line segment and within the range voxel for one or more possible target positions can allow the corresponding audio source signal to be rendered so that the sound perceived by the listener can appear more natural compared to the coordinates within an unfilled voxel.

[0075] In the context of the example of a Cartesian coordinate system, the possible target positions (joint point source coordinates of the 3D range) P max x,y,z 308a, 309a and P min x,y,z 308b and 309b can be selected in the following ways:

[0076] P max x =[P max x , C y , C z ] is the line [D max x ,D minx ) is closest to D max x a voxel that is not an “occluded” voxel for the audio object;

[0077] P min x =[P min x , C y , C z ] is the line (D max x ,D min x ] is closest to D min x , and that voxel is not an "occluded" voxel for that audio object.

[0078] The same process can be applied to obtain P max y,z and P min y,z。 The “unoccluded” voxels can be defined as P min x，y，z ∈VOX x，y，z ( Figure 4 ) or P max x，y，z ∈VOX x，y，z ||AIR x，y，z ( Figure 3 ), where AIR x,y,z is a sound transmission medium such as air and / or water (unfilled voxels).

[0079] In some embodiments, the method may further include selecting an audio source position from possible target positions based on a predefined minimum distance between the audio sources. The method may then include assigning an audio source from a plurality of audio sources to the selected audio source position. For example, the number of (point) audio sources N = [1, ..., 6] may be calculated by considering 3 variables:

[0080] W x，y，z =|P max x，y，z -P min x，y，z |

[0081] If W x，y，z >Δ min , then consider the audio source P of "x", "y", and "z" pairs max x,y,z and P min x,y,z Used for 3D range modeling. mincan be the (desired) minimum distance between two "joint" (point) audio sources. This allows preventing phase audio artifacts caused by rendering two related audio signals too close to each other. If all W x，y，z ≤Δ min , the 3D extent may not be modeled, and the audio object may be located at voxel C x,y,z It is represented by a single audio point source (N=1) in .

[0082] In an embodiment, the method may further comprise obtaining a mapping indicating an (e.g., expected or desired) assignment of audio source signals to audio source locations (or possible target locations). For example, M audio signals S may be read from the bitstream payload. 1,…,M To position coordinates P 1,…,6 The following mapping:

[0083] # Location Audio Source 1 <![CDATA[P max x ]]> OBJ_ID_1 2 <![CDATA[P min x ]]> OBJ_ID_2 3 <![CDATA[P max y ]]> OBJ_ID_3 4 <![CDATA[P min y ]]> OBJ_ID_4 5 <![CDATA[P max z ]]> OBJ_ID_5 6 <![CDATA[P min z ]]> OBJ_ID_6

[0084] Table 1: Example mapping

[0085] In some embodiments, the method may further include assigning gains to audio source locations. Such assignments may be based at least in part on the above mapping. Appropriate signal gains may be further assigned based on the number of selected audio sources to ensure energy conservation.

[0086] Reference now Figures 5 to 9 An example of, in which a use case of an example of a method for rendering audio in an audio scene as described herein is illustrated. In this example use case, the 3D range to be rendered / modeled is exemplarily based on a tram 500. Figures 7 to 9 Illustrated are respective "visible" line segments 501, 502, 503 and respective assigned coordinates / target position coordinates 504a, 504b, 505a, 505b, 506a, 506b which have been determined according to the methods described herein.

[0087] In this example use case, the renderer is tasked with properly rendering the sounds of a virtual tram in the VR / AR / XR / MR scene. The tram can be seen traveling on a busy road. Since the audio originating from the tram comes from several parts distributed along the length of the tram, the tram cannot be modeled by a single point source. Initially, the "tram" object ( Figure 5 ) and the accompanying “audio source(s) with ranges” representing the “tram” sound may be specified by the scene creator as part of the “scene description” of the VR / AR / XR / MR scene. As an example, Figure 6 A designated “range” model representing a tram for audio rendering is shown in . Figure 7 , Figure 8 and Fig. 9Describes when the application Figure 1 The method shown in FIG. 1 is a possible embodiment of determining a set of target audio source positions 504a, 504b, 505a, 505b, 506a, 506b near the range. Subsequently, actual target audio sources corresponding to these positions are generated. In this context, the positions of the target audio sources are shown by cutting the tram range representation. Figure 7 Collect Figure 8 and Fig. 9 Thus, these target audio sources are rendered to obtain the tram sounds in the VR / AR / XR / MR scene.

[0088] Reference now Fig.10 In an example, in an embodiment, the method may further include obtaining the coordinates of the listener position 510, and rendering the audio source signal of the assigned audio source based on the reference distance 511 between the listener position 510 and the 3D range 500. For example, the distance from the listener L to the nearest point R of the range VOX may be subtracted from the listener-to-object distance (L, P), where |LR| == min(|L-VOX|).

[0089] In some embodiments, rendering may further include rendering a (point) audio source signal based on (voxel-based) occlusion and diffraction modeling. That is, for example, voxel-based occlusion and diffraction modeling may be applied to render a point audio source {P max x,y,z ,P min x,y,z} of the selected subset. Fig.10 The example of shows a 3D range representation of a tram 500 that is occluded by an obstacle 512. Therefore, coordinates 504a, 504b, 506a and 518 are occluded, and as a result, a set of virtual coordinates 516, 516a-d are generated through diffraction modeling.

[0090] The 3D range modeling method described herein assumes that diffraction modeling is applied. However, the method can also be used without applying diffraction modeling. In such cases, the method is applied to a subset of the 3D range that is visible to the listener 510. The following methods can be used to obtain the subset that is "visible" to the listener (visible means that there are no acoustic occluders between the listener and the corresponding point): a ray tracing-based method, or by checking for occlusion on the line between the listener and the subset of the 3D range representation. The subset can be determined by Monte Carlo or any other subsampling method.

[0091] Example Algorithm

[0092] In other words, the method of rendering audio in an audio scene can be described as follows. The following content means Figure 1This example implementation assumes that the decoder has received "scene description" information containing the aspects indicated in step S101.

[0093] Known

[0094] Scenario description:

[0095] Listener position coordinates L

[0096] 3D extent material ID (represents a geometric approximation of the 3D extent of the audio object)

[0097] A set of 3D grid index VOX (representing a set of 3D ranges)

[0098] A set of M audio signals (mono, stereo, etc.)

[0099] Modeling settings:

[0100] Minimum distance Δ between two "joint" point audio sources min

[0101] The mapping matrix F used to assign audio signals to the obtained point source positions (and gains)

[0102] Reference distance

[0103] Search

[0104] The number of point audio sources N = [1, ..., 6] and their coordinates P 1,…,N

[0105] The audio signal S 1,…,M Map to position P 1,…,N and gain

[0106] Solution

[0107] Determine (step S102) a voxel-based 3D range representation VOX x,y,z The center of the 3D range is represented by coordinate C x,y,z , for example using the "min / max (geometry center)" method:

[0108] C x，y，z =round(V max x，y，z -V min x，y，z | / 2)

[0109] Among them, V max x，y，z =max(VOX x，y，z ), V min x，y，z =min(VOXx，y，z )

[0110] (Note: Different representations of the center may also be used, such as the "center of mass" point (centroid). Figure 2 .

[0111] The center representation is needed to extract the three characteristic dimensions from the 3D range representation.

[0112] Through the following extreme point D max x,y,z and D min x,y,z To determine (step S103) the 3D range characteristic size representation:

[0113] D max x =[V max z , C y , C z ], D min x =[V min x , C y , C z ],

[0114] D max y =[C x , V max y , C z ], D min y =[C x , V min y , C z ],

[0115] D max z =[C x , C y , V max z ], D min z =[C x , C y , V min z ].

[0116] (Note: a different characteristic size representation may also be used, such as an offset from the center).

[0117] Determine (step S104) the 3D range joint point source coordinates P max x,y,z and P minx,y,z :

[0118] Among them, P max x =[p max x , C y , C z ] is the line [D max x ,D min x ) is closest to D max x voxel that is not an "occluded" voxel for this audio object

[0119] Among them, P min x =[p min x , C y , C z ] is the line (D max x ,D min x ] is closest to D min x , and the voxel is not an "occluded" voxel for the audio object

[0120] The same procedure can be applied to obtain P max y,z and P min y,z The “unoccluded” voxels can be defined as P min x，y，z ∈VOX x，y，z ( Figure 4 ) or P max x，y，z ∈VOX x，y，z ||AIR x，y，z ( Figure 3 ), where AIR x,y,z A sound transmission medium such as air and / or water.

[0121] The number of point audio sources N = [1, ..., 6] is calculated by considering 3 variables:

[0122] W x，y，z =|P max x，y，z -P min x，y，z |

[0123] If W x，y，z >Δ min , then consider the audio source P of "x", "y", and "z" pairs maxx,y,z and P min x,y,z Used for 3D range modeling. This is done to prevent phase audio artifacts caused by two related audio signals being rendered too close to each other.

[0124] If all W x，y，z ≤Δ min , then the 3D extent is not modeled and the audio object consists of x,y,z It is represented by a single audio point source in .

[0125] Based on the bit stream payload, M audio signals S are obtained 1,…,M To position coordinates P 1,…,6 A mapping such as the example mapping given above in Table 1. For example, in some embodiments, the mapping M can be read or extracted from the bitstream.

[0126] Appropriate signal gain is assigned based on the number of audio sources selected (to ensure energy conservation).

[0127] Apply voxel-based occlusion and diffraction modeling to render point audio sources {P max x,y,z ,P min x,y,z} of a selected subset (see, for example, Fig.10 )

[0128] A reference distance process is applied, i.e., the distance from the listener L to the closest point R of the range VOX is subtracted from the listener-to-object distance (L, P), where |LR| == min(|L-VOX|)

[0129] Apply voxel-based occlusion and diffraction modeling to render a selected subset of point audio sources {Pmaxx,y,z,Pminx,y,z}.

[0130] Rendering may be performed by a renderer capable of simulating acoustic occlusion and diffraction modeling.

[0131] Fig.10 A 3D range representation of an object (i.e., a tram) occluded by an obstacle is shown as an example of how diffraction processing can be applied to the methods described herein. The point in the middle is occluded, and therefore, a set of virtual points is generated by diffraction modeling. 513 and 514 are the lines of sight that are not blocked by the occluder 512. 515 is the direction (azimuth) in which the object 518 will be perceived. These objects are then perceived (modeled) as 516. Therefore, these coordinates 504a, 504b, 506a, 506b (missing in the figure) belonging to 518 are modeled by 516a, 516d, 516b, 516c belonging to 516.

[0132] The described 3D range modeling approach assumes the application of diffraction modeling.

[0133] However, the method can also be used without applying diffraction modeling. In such cases, the method is applied to a subset of the 3D range that is visible to the listener. The following methods can be used to obtain the subset that is "visible" to the listener (visible means that there are no acoustic occluders between the listener and the corresponding point): ray tracing-based methods, or by checking for occlusion on the line between the listener and the subset of the 3D range representation. The subset can be determined by Monte Carlo or any other subsampling method.

[0134] refer to Fig.11 , illustrating an apparatus 1100 including one or more processors 1101 and 1102 according to an embodiment of the present disclosure. The one or more processors 1101 and 1102 may be configured to execute the methods described herein.

[0135] A computing device implementing the techniques described above may have the following example architecture. Other architectures are also possible, including architectures with more or fewer components. In some embodiments, the example architecture includes one or more processors (e.g., a dual-core The components may include a processor, one or more output devices (e.g., an LCD), one or more network interfaces, one or more input devices (e.g., a mouse, a keyboard, a touch-sensitive display), and one or more computer-readable media (e.g., RAM, ROM, SDRAM, a hard disk, an optical disk, flash memory, etc.). These components may exchange communications and data via one or more communication channels (e.g., buses), which may utilize various hardware and software to facilitate the transfer of data and control signals between the components.

[0136] The term "computer-readable medium" refers to media that participate in providing instructions to a processor for execution, including but not limited to non-volatile media (e.g., optical or magnetic disks), volatile media (e.g., memory), and transmission media. Transmission media include but are not limited to coaxial cables, copper wire, and fiber optics.

[0137] The computer readable medium may further include an operating system (e.g., The operating system may include an operating system, an audio interface manager, an audio processing manager, and a real-time content distributor. The operating system may be multi-user, multi-processing, multi-tasking, multi-threaded, real-time, etc. The operating system performs basic tasks including, but not limited to: identifying input from a network interface and / or device and providing output to a network interface and / or device; recording and managing files and directories on a computer-readable medium (e.g., a memory or storage device); controlling peripheral devices; and managing services on one or more communication channels. The network communication module includes various components for establishing and maintaining a network connection (e.g., software for implementing communication protocols such as TCP / IP, HTTP, etc.).

[0138] The architecture may be implemented in a parallel processing or peer-to-peer infrastructure, or on a single device having one or more processors.The software may include multiple software components or may be a single body of code.

[0139] The described features may advantageously be implemented in one or more computer programs executable on a programmable system including at least one programmable processor coupled to receive data and instructions from and send data and instructions to a data storage system, at least one input device, and at least one output device. A computer program is a set of instructions that can be used directly or indirectly in a computer to perform a certain activity or bring about a certain result. A computer program may be written in any form of programming language (e.g., Objective-C, Java), including compiled or interpreted languages, and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, browser-based web application, or other unit suitable for use in a computing environment.

[0140] Suitable processors for executing a program of instructions include, by way of example, both general-purpose and special-purpose microprocessors, and a single processor or one of multiple processors or cores of any kind of computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more memories for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data files, or be operably coupled to communicate with them; such devices include magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and optical disks. Storage devices suitable for tangibly embodying computer program instructions and data include all forms of non-volatile memory, which include, by way of example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and memory may be supplemented by or incorporated into an ASIC (Application Specific Integrated Circuit).

[0141] To provide interaction with a user, features may be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor or retina display device. The computer may have a touch surface input device (e.g., a touch screen) or a keyboard and a pointing device such as a mouse or trackball, through which the user can provide input to the computer. The computer may have a voice input device for receiving voice commands from the user.

[0142] Features may be implemented in a computer system that includes back-end components, such as a data server, or includes middleware components, such as an application server or an Internet server, or includes front-end components, such as a client computer with a graphical user interface or an Internet browser, or any combination thereof. The components of the system may be connected by any form of medium for digital data communication, such as a communication network. Examples of communication networks include, for example, LANs, WANs, and the computers and networks that form the Internet.

[0143] A computing system may include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The relationship between the client and the server is due to the computer programs running on the respective computers and having a client-server relationship relative to each other. In some embodiments, the server sends data (e.g., an HTML page) to the client device (e.g., in order to display data to a user interacting with the client device and receive user input from the user). Data generated at the client device (e.g., the result of a user interaction) can be received from the client device at the server.

[0144] A system of one or more computers may be configured to perform specific actions by virtue of having software, firmware, hardware, or a combination thereof installed on the system that causes the system to perform the actions in operation. One or more computer programs may be configured to perform specific actions by virtue of including instructions that, when executed by a data processing device, cause the device to perform the actions.

[0145] Although this specification includes many specific implementation details, these details should not be interpreted as limiting the scope of any invention or the content that can be claimed. On the contrary, it should be regarded as a description of the characteristics peculiar to the specific embodiment of a specific invention. The specific features described in the context of each embodiment in this specification can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in the form of any suitable sub-combination. In addition, although the features can be described above as working with a specific combination and even initially required, one or more features from the required combination can be deleted from the combination in some cases, and the required combination can be for a variant of a sub-combination or a sub-combination.

[0146] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring such operations to be performed in the particular order shown or in a sequential order, or performing all illustrated operations to achieve desired results. In some cases, multitasking and parallel processing can be advantageous. In addition, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can be generally integrated together in a single software product or encapsulated in multiple software products.

[0147] Unless otherwise indicated, it will be apparent from the following discussion that the use of terms such as "processing", "computing", "calculating", "determining", "analyzing", etc. throughout the disclosed discussion should be understood to refer to the action and / or process of a computer or computing system or similar electronic computing device to manipulate and / or transform data represented as physical (such as electronic) quantities into other data similarly represented as physical quantities.

[0148] References throughout the disclosure to "an example embodiment," "some example embodiments," or "example embodiments" mean that a particular feature, structure, or characteristic described in connection with the example embodiment is included in at least one example embodiment of the disclosure. Thus, the phrases "in an example embodiment," "in some example embodiments," or "in an example embodiment" appearing in various places throughout the disclosure are not necessarily all referring to the same example embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more example embodiments, as will be apparent to one of ordinary skill in the art in light of the present disclosure.

[0149] As used herein, unless otherwise specified, the use of ordinal adjectives "first," "second," "third," etc. to describe common objects merely indicates that different instances of similar objects are being referred to and is not intended to imply that the objects described must be in a given order in time, space, rank, or in any other manner.

[0150] Likewise, it should be understood that the phraseology and terminology used herein is for descriptive purposes and should not be regarded as limiting. The use of "including," "comprising," or "having" and variations thereof are meant to encompass the items listed thereafter and equivalents thereof as well as additional items. Unless otherwise specified or limited, the terms "mounted," "connected," "supported," and "coupled" and variations thereof are used broadly and encompass direct and indirect mountings, connections, supports, and couplings.

[0151] In the claims below and the description herein, any of the terms comprising, including, or which comprises is an open term, which means including at least the elements / features that follow, but not excluding other elements / features. Therefore, when the term comprising is used in a claim, the term should not be interpreted as being limited to the devices or elements or steps listed thereafter. For example, the scope of the expression a device comprising A and B should not be limited to a device comprising only elements A and B. As used herein, the term including, or any of which includes, or that includes is also an open term, which also means including at least the elements / features that follow the term, but not excluding other elements / features. Therefore, including is synonymous with comprising and means comprising.

[0152] It should be understood that in the above description of the example embodiments of the present disclosure, various features of the present disclosure are sometimes combined together in a single example embodiment, figure, or description thereof to simplify the present disclosure and to help understand one or more of the various creative aspects. However, the method of the present disclosure should not be interpreted as reflecting the intention that the claims require more features than the features explicitly recited in each claim. On the contrary, as reflected in the following claims, each creative aspect lies in less than all the features of a single previously disclosed example embodiment. Therefore, the claims following the specification are hereby expressly incorporated into this specification, wherein each claim independently serves as a separate example embodiment of the present disclosure.

[0153] In addition, although some example embodiments described herein include some features included in other example embodiments and do not include other features included in other example embodiments, as will be understood by those skilled in the art, the combination of features of different example embodiments is intended to be within the scope of the present disclosure and form different example embodiments. For example, in the appended claims, any of the example embodiments claimed for protection may be used in any combination.

[0154] In the description provided herein, many specific details are set forth. However, it should be understood that the exemplary embodiments of the present disclosure can be practiced without these specific details. In other instances, well-known methods, structures, and techniques are not shown in detail in order to avoid obscuring the understanding of this specification.

[0155] Therefore, although the mode that is considered to be the best mode of the present disclosure has been described, it will be appreciated by those skilled in the art that other and further modifications may be made thereto without departing from the spirit of the present disclosure, and all such changes and modifications falling within the scope of the present disclosure are intended to be protected. For example, any formula given above merely represents a program that may be used. Functions may be added or deleted from the block diagram, and operations may be interchanged among functional blocks. Steps may be added or deleted to the methods described within the scope of the present disclosure.

[0156] Various aspects and embodiments of the present disclosure may also be understood from the following enumerated example embodiments (EEE), which are not claims.

[0157] EEE 1. A method for modeling an extended audio object for audio rendering in a virtual reality or augmented reality environment, the method comprising:

[0158] determining a 3D range center representation of the voxel-based 3D range representation;

[0159] determining a 3D range characteristic size representation based on the 3D range representation;

[0160] The 3D range joint point source coordinates are determined based on the 3D range characteristic size representation or the 3D range center representation.

[0161] EEE 2. The method as described in EEE 1 further includes receiving a mapping of M audio signals to position coordinates; and assigning signal gains of the M audio signals to point sources.

[0162] EEE 3. The method as described in EEE 1 further includes rendering point sources based on voxel-based occlusion and diffraction modeling.

[0163] EEE 4. The method as described in EEE 1, wherein the center representation can be determined based on the geometric center of the voxel or by a center of mass method.

[0164] EEE 5. The method as described in EEE 1, wherein the size representation can be based on extreme points or by calculating the corresponding offset from the center.

[0165] EEE 6. The method as described in EEE 1 to 5, wherein the method is applied to a subset of the voxel-based 3D range representation.

[0166] EEE 7. The method of EEE 6, wherein the subset of the voxel-based 3D range representation corresponds to acoustically unoccluded (visible) voxels. ...

[0167] EEE 8. A non-transitory computer program comprising instructions which, when executed by a processor, cause the processor to perform a method according to any one of EEE 1 to 7. EEE 8.

[0168] EEE 9. A device configured to perform the method as described in EEE 1 to 7.

Claims

1. A method for rendering audio in an audio scene, the method comprising: Receiving (S101) a voxel-based audio scene representation of the audio scene, the audio scene representation comprising an indication of range voxels (205; 305) representing a 3D range and a plurality of audio source signals for audio sources associated with the 3D range; Obtaining (S102) the coordinates of the intersection point (201, 202; 301) within the 3D range; determining (S103) one or more line segments (203, 204; 303, 304) passing through the intersection point (201, 202; 301) and extending along corresponding coordinate directions of the audio scene representation, wherein endpoints (203a, 203b, 204a, 204b; 303a, 303b, 304a, 304b) of each line segment (203, 204; 303, 304) are determined based on coordinates of one or more range voxels among the range voxels (205; 305); and An audio source of the plurality of audio sources is assigned (S104) to an audio source position (308a, 308b, 309a, 309b) within the audio scene based on the one or more line segments (203, 204; 303, 304).

2. The method of claim 1, wherein: The intersection point is one of a geometric center of the 3D range and a centroid of the 3D range.

3. The method according to claim 1 or 2, wherein: The endpoints of each line segment are determined based on extreme coordinate values ​​of the 3D range along the corresponding coordinate direction, so that the length of the line segment corresponds to the maximum size of the projection of the 3D range in the corresponding coordinate direction.

4. The method according to any one of claims 1 to 3, wherein: The audio scene representation further indicates occlusion voxels; and Herein, assigning the audio source comprises assigning the audio source to coordinates within a voxel other than the occluding voxel.

5. The method of claim 4, wherein: The audio scene representation further indicates unfilled voxels; and Herein, assigning the audio source comprises assigning the audio source to a coordinate on the corresponding line segment that is closest to an endpoint of the corresponding line segment and is within a range voxel or an unfilled voxel.

6. The method according to any one of claims 1 to 4, wherein: Assigning the audio source further includes determining one or more possible target locations for assigning the audio source based on the line segment.

7. The method of claim 6, wherein: The audio scene representation further indicates unfilled voxels; and Wherein, determining the one or more possible target positions comprises selecting coordinates for the one or more possible target positions that are closest to the endpoint of the corresponding line segment and are within a range voxel or an unfilled voxel.

8. The method according to claim 6 or 7, wherein: Determining the one or more possible target positions includes selecting coordinates for the one or more possible target positions that are closest to the endpoint of the corresponding line segment and that are within a range voxel.

9. The method according to any one of claims 6 to 8, wherein: The method further comprises: selecting the audio source position from the possible target positions based on a predefined minimum distance between audio sources; and An audio source of the plurality of audio sources is assigned to the selected audio source position.

10. The method of any one of claims 1 to 9, further comprising obtaining a mapping indicating the assignment of the audio source signals to the audio source positions.

11. The method of claim 10, further comprising assigning gains to the audio source locations based at least in part on the mapping.

12. The method according to any one of claims 1 to 11, wherein: The method further comprises: obtaining coordinates of the listener's location; and An audio source signal of the assigned audio source is rendered based on a reference distance between the listener position and the 3D range.

13. The method of claim 12, wherein: The rendering further includes rendering the audio source signal based on occlusion and diffraction modeling.

14. An apparatus (1100) for rendering audio in a voxel-based audio scene representation, wherein: The apparatus comprises one or more processors (1101, 1102), wherein the one or more processors are configured to perform a method comprising the following operations: Receiving (S101) a voxel-based audio scene representation of the audio scene, the audio scene representation comprising an indication of range voxels (205; 305) representing a 3D range and a plurality of audio source signals for audio sources associated with the 3D range; Obtaining (S102) the coordinates of the intersection point (201, 202; 301) within the 3D range; determining (S103) one or more line segments (203, 204; 303, 304) passing through the intersection point (201, 202; 301) and extending along corresponding coordinate directions of the audio scene representation, wherein endpoints (203a, 203b, 204a, 204b; 303a, 303b, 304a, 304b) of each line segment (203, 204; 303, 304) are determined based on coordinates of one or more range voxels among the range voxels (205; 305); and An audio source of the plurality of audio sources is assigned (S104) to an audio source position (308a, 308b, 309a, 309b) within the audio scene based on the one or more line segments (203, 204; 303, 304).

15. A program comprising instructions which, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 13.

16. A computer-readable storage medium storing the program according to claim 15.

Images