Diffraction modeling based on grid path discovery

Abstract

[0003] Described herein is a method for processing audio content for rendering in a three-dimensional audio scene, the audio content including a sound source at a source location, the method including: obtaining a voxelized representation of the three-dimensional audio scene, the voxelized representation indicating volume elements through which sound can propagate and volume elements that are occluded by sound; generating a two-dimensional projection map for the audio scene based on the voxelized representation by applying a projection operation that projects onto a horizontal plane to the voxelized representation; and determining parameters indicative of a virtual source position based on the source position, the listener position, and the projection map to simulate the effect of acoustic diffraction by the three-dimensional audio scene on a source signal of the sound source at the source position by rendering a virtual source signal from the virtual source position of the virtual sound source. Corresponding apparatuses and corresponding computer program products are also described.

Description

[Technical Field]

[0001] [Cross-references to related applications] This application claims priority to the following priority applications: U.S. Provisional Application No. 63 / 004,539 (cf. D20016USP1), filed on April 3, 2020, and European Patent Application No. 20167907.3 (cf. D20016EP), filed on April 3, 2020, which are incorporated herein by reference.

[0002] [Technology] This disclosure relates, in general, to a method for processing audio content for audio rendering in a three-dimensional audio scene, taking into account the diffraction effects caused by elements of the three-dimensional audio scene. In particular, this disclosure relates to a method of (acoustic) diffraction modeling based on grid path discovery. This disclosure further relates to corresponding apparatus and computer program products.

[0003] While some embodiments are described herein with reference to their disclosures, it will be understood that this disclosure is not limited to such domains of use and is applicable in a broader context. [Background technology]

[0004] Nothing discussed in this disclosure regarding background technology should be construed as acknowledging that such technology is widely known or forms part of the common general knowledge in the art.

[0005] Acoustic diffraction refers to the various phenomena that occur when waves hit an obstacle or an occluder opening. The psychoacoustic perception of acoustic diffraction is that sound "bends" around an obstacle or "spreads out" as it passes through a small opening in a soundproof wall. This is why we can still perceive sound well in a concert hall even when it is blocked by a pillar or pole, or hear sound coming from a room with a slightly ajar door while standing in a corridor.

[0006] Acoustic diffraction is a naturally occurring phenomenon that plays a crucial role in achieving a correct interpretation of the acoustic environment. It is particularly perceptually relevant when sound barriers and / or small openings are present within them. Without simulation of acoustic diffraction effects, a realistic reproduction of the sound scene would inevitably be compromised.

[0007] Despite the importance of acoustic diffraction in the physical world, diffraction modeling in virtual environments (e.g., virtual reality or game worlds) is often overlooked. Modeling acoustic diffraction effects is frequently abandoned or replaced by direct signal propagation approaches. Even state-of-the-art audio rendering solutions have yet to reach a point where they can accurately reproduce realistic acoustic effects in real time in 3D virtual environments. One reason for this is that physically accurate modeling of diffraction effects is a computationally intensive task. This is due to the complexity of representing occlusion / diffraction-related object geometry (e.g., walls and holes), the number of spatial dimensions for audio rendering (e.g., 3D virtual reality), and the requirements for realism and content creator intent regarding the effects being modeled (e.g., audible range).

[0008] Therefore, there is a need for realistic yet computationally feasible acoustic diffraction modeling. In other words, improved methods and apparatus are needed for processing audio content for rendering in (virtual) 3D audio scenes. [Overview of the Initiative]

[0009] According to a first aspect of this disclosure, a method is provided for processing audio content for rendering in a three-dimensional audio scene. The three-dimensional audio scene may be a virtual audio scene. The audio content may include sound sources at source locations. The method may include obtaining a voxelized representation of the three-dimensional audio scene. The voxelized representation may represent volume elements (e.g., voxels) through which sound can propagate and volume elements (e.g., voxels) through which sound is blocked. Thus, the voxelized representation can be said to include a description of the occlusion geometry of the three-dimensional audio scene. In some embodiments, the volume elements (e.g., voxels) in the voxelized representation may be cubes with a predetermined side length, e.g., about 10 centimeters. In other embodiments, the volume elements (e.g., voxels) may have an adaptive size, i.e., an adaptive side length. Volume elements through which sound can propagate (e.g., freely propagate) may be thought to be filled with air. In some implementations, volume elements may have associated occlusion coefficients ranging from coefficient values ​​indicating free sound propagation (e.g., 0) to coefficient values ​​indicating complete occlusion (e.g., 1). The method may further include obtaining a two-dimensional projection map of the audio scene. The two-dimensional projection map may be associated with the voxelized representation by a projection operation that projects onto a horizontal plane. This projection map may include a grid, where each element of the grid represents a horizontal position in the audio scene and indicates whether at least one volume element at that horizontal position in the voxelized representation (e.g., above or below it) is a volume element on which sound can propagate. The projection map is sometimes called an occlusion map or propagation map. The method may further include determining parameters indicating the virtual source position based on the source position, listener position, and projection map in order to simulate the effect of acoustic diffraction by the three-dimensional audio scene on the source signal of the sound source at the source position by rendering a virtual source signal from the virtual source position of the virtual sound source.

[0010] By considering voxelized representation of 3D audio scenes, the complexity of representation can be significantly reduced. This can be achieved without perceptible degradation of the listening experience through appropriate selection of voxel size (e.g., side lengths of approximately 10 centimeters, or a few centimeters). Further projection onto a 2D projection map can reduce complexity even more, thereby enabling the use of a 2D pathfinding algorithm. The paths output by the pathfinding algorithm contain sufficient information to generate virtual sound sources at virtual source locations that realistically simulate the effects of sound diffraction in the original 3D audio scene. Thanks to the complexity reduction achieved by the proposed method, a realistic listening experience in 3D audio scenes can be provided with reasonable computational effort. In particular, this enables realistic sound rendering in 3D audio scenes even for real-time applications such as virtual reality applications or computer / console games.

[0011] In some embodiments, obtaining a two-dimensional projection map may involve generating a two-dimensional projection map based on a voxelized representation by applying a projection operation to project onto a horizontal plane to the voxelized representation. If the volume element has associated occlusion coefficients, generating the projection map may involve a truncation step that maps any of these coefficient values ​​to one of two extreme values ​​(e.g., 0 or 1). Depending on the situation (e.g., how often the geometry is expected to change), the voxelized representation and the two-dimensional projection map may be pre-calculated (e.g., for volume elements of multiple sizes) and accessed as needed. For example, pre-calculation may be performed on the encoder side, and the voxelized representation and the two-dimensional projection map may be provided to the decoder side as part of the bitstream. This may further reduce the computational load on the decoder side.

[0012] In some embodiments, determining parameters indicating virtual source locations may involve applying a path-finding algorithm to a projection map to determine paths in the projection map between a two-dimensional projection of source locations (e.g., a grid element containing source locations) and a two-dimensional projection of listener locations (e.g., a grid element containing listener locations), along grid elements indicating that at least one volume element at each horizontal position in the voxelized representation is a volume element through which sound can propagate. These grid elements are sometimes referred to as unoccluded grid elements.

[0013] A wide range of reliable and efficient pathfinding algorithms are readily available and can be selected according to the specific requirements of the rendering environment. Thus, the proposed method can be tailored specifically to the computing power available on the rendering side by appropriately selecting the pathfinding algorithm.

[0014] In some embodiments, determining parameters indicating a virtual source location may include calculating a line of sight between a source location and a listener location in a three-dimensional audio scene. The determination may further include applying a path-finding algorithm to determine a path in the projection map between the two-dimensional projection of the source location and the two-dimensional projection of the listener location, along grid elements indicating that at least one volume element at each horizontal location in the voxelized representation is a volume element through which sound can propagate, if the calculated line of sight intersects a volume element in the voxelized representation where sound is obstructed, and the two-dimensional projection of the line of sight onto the horizontal plane intersects a grid element in the projection map indicating that at least one volume element at each horizontal location in the voxelized representation is a volume element through which sound can propagate. Under the above conditions, the determination may further include determining parameters indicating a virtual source location based on the determined path. It is understood that the above substeps of the determination step may imply the respective steps of determining whether there is an intersection in the voxelized representation and / or projection map.

[0015] By checking whether direct line-of-sight lines intersect occluding voxels in the voxelized representation or occluding grid elements in the projection map, we can identify special cases that are easier to handle and perform special processing, thereby further reducing the overall computational load.

[0016] In some embodiments, the method may further include the step of obtaining an elaborated voxelized representation and an elaborated 2D projection map if the pathfinding algorithm is unable to identify a path in the projection map between the 2D projection of the source location and the 2D projection of the listener location. Under the above conditions, the method may further include applying the pathfinding algorithm to determine a path in the elaborated projection map between the 2D projection of the source location and the 2D projection of the listener location, along grid elements indicating that at least one volume element at each horizontal location in the elaborated voxelized representation is a volume element on which sound can propagate. The elaborated 2D projection map may be associated with the elaborated voxelized representation by a projection operation that projects onto a horizontal plane. Furthermore, the volume elements of the elaborated voxelized representation may have shorter side lengths than the volume elements of the voxelized representation. For example, the side lengths of the volume elements of the elaborated voxelized representation may be half the side lengths of the volume elements of the (initial / previous) voxelized representation. Whether refined voxelized representations and refined 2D projection maps are obtained may depend on whether there is still time (e.g., CPU time) available within the current cycle (e.g., duration) of the rendering procedure. For example, the cycle length may be related to the desired (e.g., predetermined) refresh rate (update rate) of the audio rendering. If CPU time is available, the representation may be refined; otherwise, if no path is found, the sound from the sound source may not be rendered at all.

[0017] This allows for a sequential improvement in the accuracy of the representation. Even a "coarse" representation can be maintained if it yields an acceptable result in the sense that a path can be found and the virtual source position of the virtual sound source can be determined. On the other hand, if no such path is found (and CPU time is available), the granularity of the voxelized representation is refined to check whether there are openings within the occlusion geometry that are not identified at the coarser granularity but allow sound from the sound source to be perceived at the listener's position. As a result, the proposed conditional refinement can achieve an optimal trade-off between the accuracy of the representation and the reduction of computational load.

[0018] In some embodiments, determining parameters indicating a virtual source location may include determining the distance and azimuth angle between the listener location and the virtual source location based on a determined path, where the distance may be determined based on the path length of the determined path. For example, the distance may be determined as the path length of the (complete) determined path (e.g., the actual length of the path, or an estimate / approximation of the path, such as the "Manhattan" distance or a weighted number of voxels the path intersects). Alternatively, the path length of the determined path may be adjusted based on the difference between the vertical coordinates of the listener location and the vertical coordinates of the source location, or based on extrapolation to a voxelized representation. Determining the azimuth angle may include identifying the nearest directional change in the determined path, starting from a two-dimensional projection of the listener location. Determining the azimuth angle may further include determining the azimuth angle as the azimuth angle between the two-dimensional projection of the listener location and the identified nearest directional change, where the (nearest) directional change is understood to be caused by a volume element that obstructs the sound.

[0019] This leverages the fact that the determined paths within the projection map contain enough information to efficiently determine the 0.0 distance and azimuth angle for the virtual source location, creating a realistic listening experience during rendering.

[0020] In some embodiments, determining a parameter indicating a virtual source location may include determining an elevation angle between the listener location and the virtual source location based on a determined path and voxelized representation. Here, determining the elevation angle may include identifying the nearest directional change in the determined path, starting from a two-dimensional projection of the listener location. Determining the elevation angle may further include determining a volume element in the voxelized representation through which sound can propagate at the horizontal position of the identified directional change. Here, a volume element through which sound can propagate at the horizontal position of the determined directional change may be determined that has the closest vertical coordinate to the vertical coordinate of the listener location, or the shortest distance to the direct line of sight between the listener location and the sound source location, or is within the largest continuous subset of voxels through which sound can propagate. Determining the elevation angle may further include determining the elevation angle as the elevation angle between the listener location and the determined volume element.

[0021] This leverages the fact that the determined path within the projection map, along with the voxelized representation, contains enough information to efficiently determine the elevation angle of the virtual source position, which generates a realistic listening experience during rendering.

[0022] In some embodiments, determining the parameter indicating the virtual source position may include calculating the line of sight between the source position and the listener position in the three-dimensional audio scene. Determining the parameter indicating the virtual source position may further include, when the calculated line of sight intersects a volume element within the voxelized representation where sound is blocked and the two-dimensional projection of the line of sight onto the horizontal plane does not intersect a grid element within the projection map indicating that none of the volume elements at that horizontal position within the voxelized representation are volume elements through which sound can propagate, determining the distance and azimuth angle between the listener position and the virtual source position based on the two-dimensional projection of the line of sight onto the horizontal plane. It is understood that when the calculated line of sight does not intersect a volume element within the voxelized representation where sound is blocked (in which case the original source position can be used), or when no path is found within the projection map (in which case no sound rendering is necessary), there is no need to determine the virtual source position.

[0023] Thereby, specific trivial cases can be identified and processed in a simple way, further reducing the overall computational load incurred by the proposed method while still being able to obtain a realistic listening experience.

[0024] In some embodiments, determining the parameter indicating the virtual source position may further include determining the elevation angle between the listener position and the virtual source position based on the line of sight and the voxelized representation. Here, determining the elevation angle may include starting from the listener position and identifying the nearest volume element that the calculated line of sight intersects and where sound is blocked. Determining the elevation angle may further include determining, at the horizontal position of the identified volume element, the volume elements in the voxelized representation through which sound can propagate. Determining the elevation angle may further include determining the elevation angle as the elevation angle between the listener position and the determined volume element.

[0025] Thereby, in the identified special case, the fact that the direct line of sight includes sufficient information to efficiently determine the elevation angle of the virtual source position that creates a realistic listening experience, together with the voxelized representation, is utilized.

[0026] According to another aspect of the present disclosure, an apparatus is provided that includes a processor coupled to a memory storing instructions for the processor. The processor may be adapted to cause the apparatus to execute a method according to any of the above aspects and their embodiments.

[0027] According to another aspect of the present disclosure, a computer program is provided that includes instructions for causing a processor executing the instructions to execute a method according to any of the above first aspects and their embodiments.

[0028] According to yet another aspect of the present disclosure, a computer-readable storage medium is provided that stores a computer program according to the above aspects.

Brief Description of the Drawings

[0029] Next, exemplary embodiments of the present disclosure will be described by way of example only with reference to the accompanying drawings. [Figure 1] FIGS. 1A and 1B respectively show an example of an acoustic diffraction scenario caused by a shielding element in a three-dimensional audio scene and its two-dimensional horizontal projection on a two-dimensional voxelized grid. [Figure 2] A flowchart of an example of a method for processing audio content for rendering in a three-dimensional audio scene is shown. [Figure 3] FIGS. 3A and 3B show examples of paths in a two-dimensional projection map. [Figure 4] A flowchart of another example of a method for processing audio content for rendering in a three-dimensional audio scene is shown. [Figure 5]A schematic example of a device that implements a method for processing audio content for rendering in a 3D audio scene is shown. [Modes for carrying out the invention]

[0030] Figure 1A provides an example of a three-dimensional audio scene 100, and Figure 1B shows its horizontal projection onto a two-dimensional projection plane. This audio scene illustrates the effect of acoustic diffraction when sound emitted at a source location 110 is perceived at a listener location 120. The direct path 130 between the source location 110 and the listener location 120 is blocked by an occlusion element 140 in the audio scene, such as a wall or other extension between the source location 110 and the listener location 120. Nevertheless, the sound emitted at the source location 110 is perceived at the listener location 120. This is due to acoustic diffraction, which allows the sound to reach the listener location 120 along all paths around the occlusion element 140. The shortest "curved" path 150 can be considered the dominant path from a psychoacoustic standpoint. This effect needs to be appropriately modeled to enable realistic audio rendering in a (virtual) three-dimensional audio scene that includes occlusion elements.

[0031] Broadly speaking, this disclosure proposes a computationally efficient method for acoustic diffraction modeling based on a path-finding algorithm for an equidistant grid representing voxelized occlusion geometry of a three-dimensional (audio) scene. To achieve this objective, this disclosure utilizes a simplified (but sufficiently accurate) geometric representation using a voxelization method (e.g., an adaptive voxelization method). Furthermore, this disclosure uses a two-dimensional space for diffraction modeling by contraction of the associated geometric representation, along with means for controlling acoustic effects that approximate acoustic occlusion / diffraction phenomena for content creators and encoder operators.

[0032] Therefore, the methods and apparatus described herein enable perceptually realistic acoustic occlusion / diffraction effect simulations for dynamic and interactive three-dimensional virtual environments, aiming to improve the overall user experience and facilitate the broader deployment of virtual reality (VR) applications. Hereafter, “simulation of occlusion / diffraction” may be understood to mean modeling the perceptual effects of occlusion / diffraction, and not necessarily to mean an accurate simulation of the physical effects.

[0033] As a starting point, this disclosure assumes that a (virtual) three-dimensional space (e.g., a three-dimensional audio scene) includes an occlusion geometry description (i.e., information about occlusion elements in the audio scene, expressed in terms of, for example, meshes or geometric primitives and, optionally, occlusion coefficients obtained for or assigned to those faces). This disclosure further assumes the locations of listeners / users and audio sources, as well as audio signals (e.g., waveforms) emitted by the audio sources. These elements can be considered inputs to methods proposed by embodiments of this disclosure. However, it should be noted that these methods can also receive processed versions of the occlusion geometry description. For example, a method may receive a voxelized representation of the three-dimensional space (e.g., a three-dimensional audio scene) as input.

[0034] Based on the above assumptions (i.e., the above inputs), methods and apparatus according to embodiments of the present disclosure attempt to simulate the perceptual effect of acoustic occlusion / diffraction by providing (e.g., determining or generating) a virtual audio source (virtual sound source) at a virtual source location, possibly along with a virtual source signal. Here, the virtual source location is defined by its coordinates, including distance (e.g., radius) and direction (e.g., azimuth and elevation). However, it should be noted that the virtual source location may also be defined using other coordinate representations (e.g., Cartesian coordinate system), and the present disclosure is not limited to spherical coordinates. If the location is represented in a coordinate system different from the coordinate system used for sound rendering, appropriate conversion steps between coordinate systems may be performed, for example, during audio rendering. The virtual source signal may be determined based on the original waveform of the source signal, and may be modified (e.g., gain, filters, etc.).

[0035] Figure 2 shows a flowchart of an example of a method 200 for processing audio content for rendering in a 3D audio scene. The audio content includes sound sources at source locations that emit (sound) source signals, i.e., sound (e.g., source waveform) based on the source signal.

[0036] In 202, scene geometry is received as input. Scene geometry may relate to a meshed representation of a 3D audio scene (3D audio environment). The meshed representation includes a mesh or set of meshes representing the faces of objects (elements of the scene), such as local occluders (e.g., obstacles) or global occluders (e.g., walls) in a 3D audio scene. In some implementations, each mesh or mesh face may have associated occlusion coefficients.

[0037] In step 204, it is determined whether the scene geometry and / or the corresponding occlusion properties have been updated. If so (Yes), the method proceeds to step 206 to determine the voxelized representation of the 3D audio scene, or (for example, if a voxelized representation is already available and the changes to the scene geometry are relatively small) to update at least a subpart of the voxelized representation based on the updated subpart of the scene geometry, and then in step 208 to generate a 2D projection map of the voxelized representation. If not so (No), the method proceeds to step 214.

[0038] In step 206, the selection of acoustically relevant geometry is performed. Thus, a subset of acoustically relevant data representing objects (with dimensions and types that can / should cause occlusion / diffraction effects) is selected from all available scene geometry information. This step may be optional.

[0039] In 208, voxelization (i.e., a voxelization method / algorithm) is applied to scene geometry or acoustically relevant geometry (e.g., occlusion / diffraction-related geometry) to generate a voxelized representation of the 3D audio scene. The voxelized representation contains multiple voxels (volume elements) of regular size arranged in a regular 3D grid. Therefore, the voxels may be cubes with a predetermined side length, for example, about 10 centimeters.

[0040] In one exemplary implementation, a voxelization algorithm can be applied to a meshed representation of a 3D audio scene to generate a voxelized representation. This step involves reducing the complexity of the geometric representation by a 3D voxelization method / algorithm (e.g., an adaptive 3D voxelization method / algorithm) that divides the 3D space into two (abstract) classes of voxels (volume elements): occluder object material (OM) voxels, such as concrete or wood, and sound propagation medium (SM) voxels, such as air or water. Thus, a complete set of voxels, VOX, is given by the sets of OM voxels and SM voxels:

number

[0041] In step 210, voxels for diffraction modeling are extracted / selected. For this purpose, the OM voxels are divided into two groups: locally occluding (LO) voxels (corresponding to locally occluding obstacles, e.g., furniture) and globally occluding (GO) voxels (corresponding to globally occluding barriers, e.g., walls). Locally occluding obstacles are assumed not to affect the large-scale directivity of sound propagation, and as a result, sound from a sound source behind a locally occluding obstacle propagates in the same direction as the original sound, and possibly with some attenuation, at least over a sufficient distance from the locally occluding obstacle. Thus, the set of OM voxels is divided as follows:

number

[0042] Subsequently, the subsequent steps of method 200 may consider only GO voxels. Thus, in other words, step 210 involves removing any voxels corresponding to local occluders from the generated voxelized representation, where local occluders represent objects in a 3D audio scene that are not expected to affect the large-scale directivity of sound propagation. For example, the boundaries of a 3D audio scene (e.g., walls) are global occluders that should be retained at this point. Local occluders can be identified, for example, by applying image processing techniques (e.g., filtering) to the generated voxelized representation, assuming that local occluders are removed by appropriate spatial filtering. Local occluders can also be identified based on the spatial characteristics of OM voxels (e.g., the number and / or size of voxels representing occluding obstacles) and / or their relationship to surrounding SM voxels (e.g., free volume for sound propagation). Similarly, local occluders may be directly specified by corresponding metadata, reflecting the intentions of the audio scene creator. In particular, step 210 may be optional.

[0043] Steps 206, 208, and 210 aim to controllably reduce the complexity of the geometric representation of a 3D audio scene by determining the importance of geometric elements in relation to the intensity of occlusion / diffraction effects caused by those geometric elements.

[0044] In some implementations, steps 206, 208, and 210 may be replaced by obtaining a voxelized representation of the 3D audio scene in a different way, for example, by receiving a voxelized representation from an external source. For example, the voxelized representation may be read from storage memory or extracted (e.g., decoded) from a bitstream. In these cases, step 202 may also be omitted. Generally, method 200 can be said to include the step of obtaining a voxelized representation of the 3D audio scene, where the voxelized representation represents volume elements through which sound can propagate (e.g., SM voxels) and volume elements through which sound is blocked (e.g., OM voxels). Voxels through which sound can propagate (e.g., freely propagate) can be thought of as being filled with air. In some implementations, as described above, this step may be carried out by steps 206, 208, and 210. Obtaining a voxelized representation of the 3D audio scene may depend on whether the 3D audio scene has been updated.

[0045] The effect of any voxels removed from the voxelized representation in step 210 may be considered in an optional step 236 by applying local attenuation modeling, depending on the listener position, to simulate the attenuation of sound by the removed voxels corresponding to the local occluder. Any attenuation gain or filter determined in this step may be determined in 230 and applied to the virtual source signal described below.

[0046] In 212, a two-dimensional projection map (or projection matrix) for diffraction modeling is generated (e.g., computed) from a voxelized representation (e.g., from GO voxels). For example, a two-dimensional projection map for an audio scene can be generated based on a voxelized representation by applying a projection operation to the voxelized representation. The projection operation projects onto a horizontal plane (e.g., horizontal from the listener's viewpoint). As a result, the projection map contains a grid, where each element of the grid represents a horizontal position in the audio scene and indicates whether at least one voxel (volume element) at that horizontal position in the voxelized representation (e.g., above, in the middle, or below the horizontal plane) is a voxel from which sound can propagate (e.g., it is an SM voxel, or equivalently, not a GO voxel). Thus, the projection map is sometimes called a propagation map or occlusion map.

[0047] In some implementations, a 2D projection map (projection matrix) can be obtained from a 3D voxelized representation (3D voxel group matrix), or a portion of it representing a globally occluding barrier, using the following reduction method:

number

[0048] Here, the values ​​of PM and GO may each fall within a range between extreme values ​​representing free tone propagation (no occlusion; e.g., value 0) and complete occlusion (e.g., value 1). For example, the values ​​of PM and GO may fall within the range [0,1] and correspond to the following: 0 - No occlusion 1 - Complete occlusion

[0049] Needless to say, it is also possible to assign the values ​​in reverse, in which case equation (3) would need to be adapted by replacing min with max.

[0050] For PM and GO values ​​within the range [0,1], an optional truncation step may be performed in the context of generating the projection map in order to render the projection map entries as Boolean types, as follows:

number

[0051] On the other hand, it is also possible to have a case where the occlusion coefficient value of the voxel in the voxelized representation (e.g., the GO value) is already of type Boolean. In this case, the PM matrix can be obtained as follows:

number

[0052] Step 212 can be said to aim at reducing the spatial dimension for diffraction modeling by introducing a PM matrix to represent the concepts of "walls and holes." In fact, if sound propagation is possible for any of the voxels at a given horizontal position (i.e., within a "row" of voxels at that horizontal position), the projection map will show that sound propagation is possible for that horizontal position (i.e., that grid element), and therefore will show at least a "hole" at that horizontal position.

[0053] Generating a 2D projection map may depend on whether or not the 3D audio scene has been updated.

[0054] In some implementations, step 212 may be replaced by obtaining the 2D projection map in a different way, for example, by receiving the 2D projection map from an external source. For example, the 2D projection map may be read from storage memory or extracted (e.g., decoded) from a bitstream. Generally, method 200 can be said to include the step of obtaining a 2D projection map of an audio scene, which is associated with a voxelized representation by a projection operation that projects onto a horizontal plane. The projection map includes a grid, where each element of the grid represents a horizontal position in the audio scene and indicates whether at least one volume element at that horizontal position in the voxelized representation is a volume element on which sound can propagate. In some implementations, as described above, this step may be performed by step 212, for example, in the sense that obtaining the 2D projection map includes applying a projection operation to the voxelized representation. Nevertheless, obtaining the 2D projection map may depend on whether an update of the 3D audio scene has been performed.

[0055] In step 214, it is determined whether the listener position and / or source position have been updated. If so (Yes), the method proceeds to step 216 to determine the parameter indicating the virtual source position of the virtual sound source. If not so (No), the method proceeds to step 230 using the previously determined virtual source position.

[0056] Generally, determining the parameters that indicate the virtual source position of a virtual sound source is performed based on the (original) source position, listener position, and projection map. In some cases, the determination may also be based on a voxelized representation. The purpose of determining the virtual source position of a virtual sound source is to simulate the effect of acoustic diffraction by the 3D audio scene on the source signal of the sound source at the (original) source position by rendering the virtual source signal from the virtual source position.

[0057] Determining the parameters indicating the virtual source location can be carried out by some or all of steps 216, 218, 220, 222, 224, 226, 228, and 234 described below. Generally, this step can be said to involve applying a path-finding algorithm to a projection map to determine the path in the projection map between the two-dimensional projection of the source location (e.g., the grid element containing that location) and the two-dimensional projection of the listener location (e.g., the grid element containing that location), along grid elements indicating that at least one volume element at each horizontal position in the voxelized representation is a volume element (e.g., an unoccluded grid element) through which sound can propagate.

[0058] As described above, determining the parameter indicating the virtual source position of a virtual sound source may depend on whether the listener position and / or source position have been updated.

[0059] In step 216, the line of sight between the source and listener positions in a three-dimensional audio scene is calculated. This line of sight can also be called the listener-to-source (user-to-source) direct line of sight. Its calculation may involve general geometric calculations. In some implementations, the listener and / or source positions may be quantized to the center of the respective voxels in which they are located. The line of sight can be a line of sight in three-dimensional space.

[0060] In step 218, it is checked whether the calculated line of sight intersects with a voxel in the voxelized representation where sound is blocked (e.g., a GO voxel). If it intersects (Yes), the method proceeds to step 220. If it does not intersect (No), the method proceeds to step 238. In the latter case, a direct line of sight exists between the sound source and the listener, so diffraction modeling is not necessary.

[0061] In 220, the two-dimensional projection of the line of sight onto the horizontal plane (of the projection map) is such that any of the volume elements at their horizontal position within the voxelized representation are volume elements (e.g., PM) through which sound can propagate. x,yThe system checks whether the point intersects with a grid element in the projection map that does not indicate occlusion (such as a PM element that indicates occlusion, e.g., =1). If it intersects (Yes), the process proceeds to step 222. If it does not intersect (No), the process proceeds to step 234. In the latter case, a direct line (straight line) exists between the 2D projections of the source and listener positions, so pathfinding in the projection map is not necessary.

[0062] Steps 218 and 220 aim to reduce the computational complexity required to determine the parameters indicating the virtual source location. This is achieved through special handling of simple or trivial cases. Note that steps 218 and 220 may be optional in some implementations and yield reasonable results even when the general handling is simple or trivial.

[0063] In 222, a pathfinding algorithm is applied to determine the path in the projection map between the two-dimensional projection of the source position and the two-dimensional projection of the listener position, along grid elements indicating that at least one volume element at each horizontal position in the voxelized representation is a volume element (e.g., an unoccluded grid element) on which sound can propagate. In some implementations, the listener position and / or source position may be quantized to the center of the projection of the voxel (e.g., grid element) on which they each lie. Any pathfinding algorithm, including A* and JPS, can be applied in this step. In this sense, the proposed model is modular with respect to pathfinding algorithms.

[0064] In one exemplary implementation, step 222 may include the following substeps: • Calculation of voxel indices (voxels) corresponding to listener (user) position and source (object) position: VOX listener and VOX source . • Optionally, VOX listener ∈SM, VOX source∈SM (i.e., whether the listener and source positions are within the sound propagation material), VOX listener =VOX source (In this case, spatial audio rendering can be skipped), ||VOX object -VOX user || > Δ (In this case, assuming the source is too far from the listener to be perceived, spatial audio rendering can also be skipped similarly), etc. are checked and the corresponding rules are applied. · Execute a path discovery algorithm (e.g., A*, JPS, etc.) to obtain a path (e.g., approximate the shortest path from the source to the listener). The following parameters can be applied for its control (e.g., acceleration and termination of the search loop). - Heuristic parameters for the path discovery algorithm - Previously detected paths (e.g., in the case of a dynamic scenario) - Threshold(s) for calculation time and / or number of resources - Threshold for the maximum path length expressed in terms of, for example, the number of 2D grid elements (2D projection of voxels), distance units, sound attenuation, resulting source object sound level, etc.

[0065] As described above, step 222 is executed when the calculated line of sight intersects a volume element within the voxelized representation where the sound is blocked, and the 2D projection of the line of sight onto the horizontal plane intersects a grid element within the projection map indicating that none of the volume elements at that horizontal position within the voxelized representation are volume elements through which sound can propagate (e.g., are shielding grid elements).

[0066] In step 224, it is checked whether a path was found in step 222. If a path is found (Yes), the method proceeds to 226. If a path is not found (No), the method proceeds to 238 and will not determine the parameter indicating the virtual source position of the virtual sound source. The method may then completely skip diffraction modeling and / or audio rendering for the current listener position and the current source position. Alternative processing when no path is found is described below with reference to Figure 4.

[0067] Steps 226 and 228 generally relate to determining parameters that indicate the virtual source location based on the path determined in step 222.

[0068] More specifically, in 226, the distance (i.e., radius) and azimuth angle (i.e., horizontal component) between the listener position and the virtual source position are determined based on the determined path.

[0069] The distance can be determined based on the path length of the chosen route. For example, the distance can be determined as follows: - The length of the complete path on the projection map determined in step 222, - The length of the complete path on the projection map, adjusted according to the difference between the listener position and the source position in vertical coordinates (e.g., z-dimension) (i.e., height relative to the horizontal plane), or - The length of the complete path between the listener and source, extrapolated to a voxelized representation of the 3D audio scene (e.g., GO voxel).

[0070] Incidentally, if the determined distance between the virtual source position and the listener position exceeds the maximum renderer distance of the renderer intended for use, the virtual source position may be set to the maximum rendering distance, and any remaining distance not yet considered may be encapsulated in additional attenuation gain for the virtual sound source.

[0071] The azimuth angle can first be determined by identifying the nearest change of direction along the determined path, starting from a two-dimensional projection of the listener's position. Then, the azimuth angle can be determined as the azimuth angle between the two-dimensional projection of the listener's position (on a projection map) and the identified nearest change of direction. For example, the azimuth angle can be determined based on the relative position of each grid element in the projection map containing the two-dimensional projection of the listener's position and the identified nearest change of direction. In other words, the azimuth angle can be determined based on the direction of the determined path, starting from the listener's position. Further paraphrasing, the azimuth angle can be determined from the listener's position based on the direction in the projection map to the first occluding grid element relative to the determined path, and then the path changes direction. It is understood that the azimuth angle can be determined by referring to a predetermined reference direction, such as the x-dimension or y-dimension in the coordinate system of a three-dimensional audio scene.

[0072] Figure 3A schematically illustrates an example of determining the azimuth angle of a virtual source position. The projection map 30 shown therein includes occlusion grid elements 50 (shaded areas, i.e., grid elements representing each horizontal position in the audio scene and indicating that none of the voxels at that horizontal position in the voxelized representation are voxels from which sound can propagate) and unocclusion grid elements 40 (i.e., grid elements representing each horizontal position in the audio scene and indicating that at least one of the voxels at that horizontal position in the voxelized representation is a voxel from which sound can propagate). For illustrative purposes, horizontal projections of occlusion elements 140 of the audio scene are also shown, but these projections are not part of the projection map 30. The two-dimensional projection 60 of the line of sight between the source position 10 (two-dimensional projection) and the listener position 20 (two-dimensional projection) onto the projection map 30 is occluded by at least one of the occlusion grid elements 50 of the projection map 30. Thus, the path 70 between the source position 10 and the listener position 20 is determined using a path-finding algorithm. The path 70 changes direction behind the occluding grid element 90. Thus, the location of the nearest direction change 80 of the path 70 is determined. Then, an azimuth angle is determined to indicate the direction 95 of the path 70 to the identified nearest direction change 80. This direction 95 is typically different from the direction 60 of the direct line of sight (in its two-dimensional projection).

[0073] In step 228, the elevation angle (i.e., the vertical component) between the listener position 120 and the virtual source position is determined based on the determined path and voxelized representation. In particular, the elevation angle may be determined by first identifying the nearest directional change in the determined path, starting from a two-dimensional projection of the listener position. This may proceed as in step 226, or the nearest directional change identified in step 226 may be reused. Next, the volume element in the voxelized representation on which sound can propagate is determined at the horizontal position of the identified directional change. For example, such an (unshielded) voxel may be determined in a column of voxels at the horizontal position of the grid element containing the identified nearest directional change. In particular, that (unshielded) voxel may be determined at the horizontal position of the identified nearest directional change having the vertical coordinate closest to the vertical coordinate of the listener position. Finally, the elevation angle may be determined as the elevation angle between the listener position and the determined voxel. In other words, the elevation angle can be determined based on the direction of the determined path, starting from the listener position, when extrapolated to a voxelized representation of a three-dimensional audio scene. It is understood that the elevation angle can be determined by referring to a predetermined reference plane, such as a horizontal plane onto which the voxelized representation is projected to generate a projection map.

[0074] Figure 3B schematically illustrates an example of determining the elevation angle of a virtual source position. In particular, Figure 3B shows a (vertical) cross-section through the voxelized representation along a vertical cross-section plane that intersects with the location (e.g., voxel) of the nearest directional change 80 (closest to the 2D projection of listener position 20) of the determined path 70 between listener position 20 and source position 10, aligned with one of two directions defined by the grid of the 2D projection map. Either choice of direction defined by the grid of the 2D projection map is appropriate, unless the source position 10 and listener position have a common horizontal voxel index (i.e., are located in the same horizontal row of the voxelized representation). If the two positions share a horizontal index, the vertical cross-section plane should be chosen to intersect with the direct line of sight between source position 10 and listener position 20. The vertical cross-section includes occluded vertical grid elements 55 and unoccluded vertical grid elements 45.

[0075] In this example, the vertical section plane is orthogonal to the two-dimensional projection map in Figure 3A and is the plane containing the bottom horizontal row of the voxels in Figure 3A (i.e., the horizontal parallel of the voxel containing the nearest direction change 80 voxel). The source position 10, listener position 20, and direct line of sight are projected onto the vertical section plane to their respective vertical projections 15, 25, and 85. In this example, the vertical section contains four vertical shading grid elements 55, which are arranged in vertical columns and correspond to the voxels representing the bearing out of the shading element 140 to the right in Figure 1A.

[0076] To determine the elevation angle, an unoccluded vertical grid element 45 in the vertical section that is at the same horizontal position as the nearest direction change 80 is identified. In this example, the unoccluded vertical grid element immediately below the column of occluded vertical grid elements 55 is identified. Generally, an unoccluded vertical grid element can be identified that is at the horizontal position of the nearest direction change 80 and corresponds to, for example, a) the vertical projection 85 of the line of sight, b) the vertical coordinates of the listener position 20, or c) a voxel that is part of the largest continuous subset of unoccluded voxels. The identified unoccluded vertical grid element is used to determine the elevation angle. For example, the vertical projection of the elevation angle between the vertical projection 5 of the virtual source position and the vertical projection 25 of the listener position can be determined based on the horizontal and vertical distances between the identified unoccluded vertical grid element (e.g., its center) and the vertical projection 25 of the listener position. Next, based on the vertical projection of the azimuth and elevation angles, the actual elevation angle can be determined by applying, for example, a triangulation transformation based on the difference angle between the azimuth direction and the direction of the vertical cross-sectional plane to the vertical projection of the elevation angle.

[0077] While the determination of the elevation angle, as shown in Figure 3B, may use the actual position coordinates (as indicated by the hatched and dashed circles) of the vertical projection 15 of the source position and the vertical projection 25 of the listener position, it should be noted that the path discovery, as shown in Figure 3A, may only refer to the respective voxels (voxel indices) that contain these positions.

[0078] Furthermore, it should be noted that Figure 3B shows the vertical projection 5 of a virtual source position that is further away from the vertical projection 25 of the listener position than from the vertical projection 15 of the source position, which may be due to the fact that the length of the determined path 70 exceeds the length of the direct line of sight 60.

[0079] In summary, an unshielded (i.e., non-GO) voxel is identified at the horizontal position of the nearest identified direction change 80. This unshielded voxel may be, for example, the unshielded voxel closest to the listener position 20 in vertical coordinates (e.g., height), or the unshielded voxel closest to the direct line of sight, or an unshielded voxel that is part of the largest continuous subset of unshielded voxels (e.g., belonging to the largest opening in the wall). The elevation angle is then determined based on the identified unshielded voxel at the horizontal position of the nearest identified direction change 80.

[0080] Importantly, the above determination of the azimuth and elevation angles requires only knowledge of the last segment of the determined path between the nearest direction change 80 and the listener position 20. Knowledge of any further sections of the path is not required (and irrelevant) to determine these angles. If any, these further sections are referenced to determine the radius (or distance) to the virtual source position.

[0081] As described above, if the calculated line of sight does not intersect with voxels in the voxelized representation where sound is obstructed (e.g., GO voxels), and the two-dimensional projection of the line of sight onto the horizontal plane does not intersect with grid elements in the projection map indicating that none of the volume elements at their horizontal positions in the voxelized representation are volume elements through which sound can propagate, the method proceeds to 234. In 234, the distance (i.e., radius) and azimuth angle (i.e., horizontal component) between the listener position and the virtual source position are determined based on the two-dimensional projection of the line of sight onto the horizontal plane. The distance may be determined in the same way as in step 226, but the determined path is replaced with the two-dimensional projection of the line of sight. The azimuth angle may be determined based on the direction of the two-dimensional projection of the line of sight as seen from the listener position. The method then proceeds to 228 where the elevation angle is determined. At this point, the elevation angle is determined based on the calculated line of sight and the voxelized representation. This can be done in the following way: First, the occluding voxels in the voxelized representation that intersect the line of sight (e.g., GO voxels) are identified. If there are two or more intersecting occluding voxels, the voxel closest to the listener's position is selected. In other words, starting from the listener's position, this corresponds to identifying the nearest voxel (volume element) that is intersected by the calculated line of sight and where sound is occluded. Then, for that occluding voxel, the nearest unoccluding voxel at the same horizontal position (e.g., above or below the occluding voxel) is determined. This can correspond to determining the volume element in the voxelized representation through which sound can propagate at the horizontal position of the identified voxel. Finally, the elevation angle is determined as the elevation angle between the listener's position and the determined unoccluding volume element.

[0082] In step 230, the virtual source signal (e.g., waveform) for the virtual source at the determined virtual source position is determined based on the source signal (e.g., waveform) of the sound source. For example, the virtual source signal may be the original waveform (i.e., source signal) to which modifications (e.g., gain, filter, etc.) have been applied. These modifications can be selected from the following options: -none - Gain / filter affected by the local environment of the virtual source (e.g., slow reverberation) - Gain / filter affected by all environmental factors

[0083] In other words, the original source signal can be used as a virtual source signal, or a virtual source signal can be generated by applying gain and / or filters to the source signal. Applying filters to the source signal may include, for example, applying reverb corresponding to the area of ​​the (original) source location, applying reverb corresponding to the area of ​​the virtual source location, or generally applying reverb corresponding to one or more areas along a determined path.

[0084] In step 232, rendering (e.g., 3DoF rendering) is applied to the virtual source at the virtual source location. Generally, this step involves rendering the virtual source signal from the virtual source location to the listener location. Furthermore, any conventional processing steps for the audio source, including any post-processing steps, may be performed at this point.

[0085] In step 238, diffraction modeling is skipped if the calculated line of sight does not intersect with a voxel in the voxelized representation where sound is obstructed (e.g., a GO voxel), or if no path is found in the projection map. When there is a direct line of sight between the source and listener positions that is not obstructed by any occluding voxels (e.g., a GO voxel), it is not necessary to consider sound diffraction by the 3D audio scene when attempting to provide a realistic listening experience and lifelike perception of sound propagation in a 3D audio scene. If no path is found, the sound from the source does not need to be rendered at all.

[0086] It is understood that the process flow of steps 214 to 232 may be performed for each of multiple source locations and / or listener locations (for example, for each of multiple pairs of source locations and listener locations) together with steps 234, 236, and 238 of method 200.

[0087] In method 200 described above, if no path is found (No in step 224), the parameter indicating the virtual source position of the virtual sound source does not need to be determined. Method 200 can then completely skip diffraction modeling and / or audio rendering for the current listener position and the current source position. Method 400, which provides an alternative process for when no path is found, will be described with reference to Figure 4.

[0088] Steps 402, 404, 406, 408, 410, 412, 414, 416, 418, 420, 422, and 434 of Method 400 can be carried out in the same manner as steps 202, 204, 206, 208, 210, 212, 214, 216, 218, 220, 222, and 234 of Method 200 described above, and the respective descriptions made above can also be applied here.

[0089] In step 424, it is checked whether a route was found in step 422. If a route is found (Yes), the method proceeds to 426. If a route is not found (No), the method proceeds to 440.

[0090] Next, steps 426, 428, 430, and 432 of method 400 can be carried out in the same manner as steps 226, 228, 230, and 232 of method 200 described above, and the respective descriptions described above can also be applied here. Similarly, step 436 can be carried out in the same manner as step 236 described above.

[0091] In step 440, it is checked whether CPU time is available. This step may be performed, for example, by a real-time resource management application (real-time resource manager). It may be assumed that a specific cycle (period) is available to determine the virtual source location and perform the actual rendering. This cycle or period of time may be related to the rendering update rate or refresh rate (for example, it may be derivable therefrom). Then, in each cycle, it may be checked whether there is enough time (CPU time) available in that cycle to perform another iteration of voxelization and subsequent steps. This may be based, for example, on an estimate of the time that would be required for such iterations, based on the characteristics of the 3D audio scene and / or the amount of time previously required at a given granularity of voxelization. If it is found that CPU time is not available (No), the method proceeds to 438, which may be performed in the same manner as in step 238 above. If it is found that CPU time is available (Yes), the method proceeds to 442.

[0092] In step 442, the granularity of voxelization is refined. This corresponds to reducing the size of the voxels (e.g., side length). For example, the side length of a voxel may be halved at this point so that each voxel is divided into eight smaller voxels for the next iteration. In general, the size (e.g., side length) may be reduced according to a predetermined ratio. The method then returns to step 408 to perform voxelization of the 3D audio scene using the refined granularity. As above, this step can be replaced with obtaining a voxelized representation of the 3D audio scene with refined granularity. For example, such a refined voxelized representation may be read from storage memory or extracted from a bitstream (in some cases, after requesting a refined voxelized representation with an applicable refined granularity from a source device, such as an encoder).

[0093] The remaining steps then proceed as described above. It is understood here that the refined 2D projection map can also be obtained, for example, by reading it from storage memory, or (in some cases, after requesting a refined 2D projection map with an applicable refined granularity from a source device such as an encoder) by extracting it from a bitstream.

[0094] In particular, the refinement of voxelization granularity can be repeated (possibly multiple times) as long as it is determined at 424 that no path was found and at 440 that CPU time is available. A further criterion that may be applied at this stage is to check whether a predetermined minimum granularity of voxelization has been reached. If the minimum granularity has been reached, the method may proceed as if no CPU time had been found to be available at step 440.

[0095] In yet another implementation, the granularity refinement of the voxelized representation can be repeated (i.e., performed iteratively) as long as CPU time is available, regardless of whether a path has been found. This can be done by changing the order of steps 440 and 424, that is, by first checking in 440 whether CPU time is available, and only after it is found that no CPU time is available (Yes) checking in 424 whether a path has been found. In this case, it is understood that if it is found that no CPU time is available in step 440 (No), the method will proceed to step 424. Then, in step 424, if no path is found (No), the method will proceed to step 438. A further criterion that may be applied in step 440 is to check whether a predetermined minimum granularity of voxelization has been reached.

[0096] In addition to refining the granularity of voxelization in step 442, this implementation can modify the parameters of the path discovery algorithm (e.g., the starting position and / or target position) based on the path discovered earlier at a coarser granularity of voxelization.

[0097] The voxelization granularity (e.g., voxel size or side length) may be reset to a predetermined (e.g., default) value in the above implementation whenever a geometry update is detected in step 404. In some implementations, this may be the only instance in which the voxelization granularity is reset (or generally made coarser).

[0098] While methods for processing audio content have been described above, it will be understood that this disclosure also relates to apparatus and devices adapted to perform the described methods (e.g., encoders, decoders, renderers, playback devices, etc.), computer programs containing instructions suitable for causing a device having processing capabilities to perform the described methods, and computer-readable storage media for storing such computer programs.

[0099] Figure 5 shows an example of a device 500 adapted to perform the method described. The device 500 comprises a processor 510 and a memory 520 coupled to the processor 510 and storing instructions for the processor 510. The processor 510 is adapted to perform the method(s) described above. The device 500 may receive an input 530 including, for example, audio content (including source signals and source location indications) and a three-dimensional audio scene indication, and may generate an output 540 including, for example, a virtual source signal and virtual source location indications, or a rendered audio signal.

[0100] It should be noted that the proposed method can be performed by an encoder or decoder, or distributed between an encoder and a decoder. In the first case, the encoder may produce a representation of the rendered (virtual) sound signal as an output. In the second case, the decoder may generate the voxelized representation and the 2D projection map itself, if its computing power allows. Alternatively, in the third case, the voxelized representation and the 2D projection map may be generated by the encoder (possibly with voxelization of different granularities) and provided to the decoder, for example, as part of a bitstream. The decoder then obtains the voxelized representation and the 2D projection map, for example, by extracting them from the bitstream, and proceeds to further steps of the method(s) described above. A mixed case is also possible in which the voxelized representation is obtained (for example, from the encoder side) and the 2D projection map is generated by the decoder based on the obtained voxelized representation.

[0101] interpretation Unless otherwise specified, as is evident from the following description, throughout this disclosure, any use of terms such as “processing,” “computing,” “calculating,” “determining,” and “analyzing” is understood to refer to the operation and / or process of a computer or computing system or similar electronic computing device that manipulates and / or converts data expressed as physical quantities, such as electron quantities, into other data similarly expressed as physical quantities.

[0102] Similarly, the term “processor” can refer to any device or part of a device that processes electronic data from, for example, registers and / or memory and converts that electronic data into other electronic data that can be stored, for example, in registers and / or memory. A “computer,” “computing machine,” or “computing platform” may include one or more processors.

[0103] An exemplary embodiment of each method described herein is in the form of an instruction set, for example, a computer-readable carrier medium that carries a computer program for execution on one or more processors, for example, one or more processors that are part of a web server configuration. Thus, as will be understood by those skilled in the art, exemplary embodiments of the Disclosure may be embodied as a method, an apparatus such as a dedicated device, an apparatus such as a data processing system, or a computer-readable carrier medium, for example, a computer program product. The computer-readable carrier medium carries computer-readable code that includes a set of instructions that cause one or more processors to perform the method when executed on one or more processors. Thus, embodiments of the Disclosure may be in the form of a method, an exemplary embodiment that is entirely hardware, an exemplary embodiment that is entirely software, or an exemplary embodiment that combines software and hardware embodiments. Furthermore, the Disclosure may be in the form of a carrier medium (for example, a computer program product on a computer-readable storage medium) that carries computer-readable program code embodied on the carrier medium.

[0104] Throughout this disclosure, any reference to “an exemplary embodiment,” “several exemplary embodiments,” or “exemplary embodiments” means that certain features, structures, or characteristics described in relation to an exemplary embodiment are included in at least one exemplary embodiment of this disclosure. Therefore, occurrences of the phrases “in an exemplary embodiment,” “several exemplary embodiments,” or “exemplary embodiments” in various places throughout this disclosure do not necessarily all refer to the same exemplary embodiment. Furthermore, certain features, structures, or characteristics may be combined in any suitable manner in one or more exemplary embodiments, as will be apparent to those skilled in the art from this disclosure.

[0105] In the above description of exemplary embodiments of the Disclosure, it should be understood that various features of the Disclosure may be grouped together in a single exemplary embodiment, figure, or description thereof for the purpose of streamlining the Disclosure and aiding in the understanding of one or more of the various aspects of the Invention. However, this method of disclosure should not be construed as reflecting an intention that the claims require more features than are expressly described in each claim. Rather, as reflected in the following claims, aspects of the Invention are fewer in features than all the features of a single, aforementioned exemplary embodiment. Accordingly, the claims following the specification are expressly incorporated herein, and each claim stands independently as a separate exemplary embodiment of the Disclosure.

[0106] Furthermore, while some exemplary embodiments described herein include some features included in other exemplary embodiments and others not, combinations of features of different exemplary embodiments are intended to be within the scope of this disclosure and form different exemplary embodiments, as will be understood by those skilled in the art. For example, any of the claimed exemplary embodiments can be used in any combination within the following claims.

[0107] While we have described what we believe to be the best form of this disclosure, those skilled in the art will recognize that other and further modifications can be made thereto without departing from the spirit of this disclosure, and that all such changes and modifications that fall within the scope of this disclosure are intended to be claimed. For example, any formula given above is merely representative of the procedures that may be used. Functions may be added to or removed from the block diagram, and actions may be interchangeable between function blocks. Steps may be added to or removed from the methods described within the scope of this disclosure.

Claims

1. A method for processing audio content for rendering in a three-dimensional audio scene, wherein the audio content includes a sound source at the source position, and the method is The voxelized representation of the aforementioned three-dimensional audio scene is obtained, which shows volume elements through which sound can propagate and volume elements that block sound. A two-dimensional projection map of the three-dimensional audio scene, obtained by a projection operation projecting onto a horizontal plane, wherein the two-dimensional projection map includes a grid, and each element of the grid represents a horizontal position within the three-dimensional audio scene and indicates whether at least one volume element at that horizontal position within the voxelized representation is a volume element on which sound can propagate. To simulate the effect of acoustic diffraction by the 3D audio scene on the source signal of the sound source at the source location by rendering a virtual source signal from the virtual source location of the virtual sound source, parameters indicating the virtual source location are determined based on the source location, listener location, and the 2D projection map. A method that includes this.

2. The method according to claim 1, wherein obtaining the two-dimensional projection map includes applying the projection operation to the voxelized representation.

3. Determining the parameter indicating the virtual source location is, Applying a pathfinding algorithm to the two-dimensional projection map to determine the path in the two-dimensional projection map between the two-dimensional projection of the source position and the two-dimensional projection of the listener position, along grid elements indicating that at least one volume element at each horizontal position in the voxelized representation is a volume element on which sound can propagate. The method according to claim 1.

4. Determining the parameter indicating the virtual source location is, Calculating the line of sight between the source position and the listener position in the three-dimensional audio scene, If the calculated line of sight intersects with a volume element in the voxelized representation where sound is blocked, and the two-dimensional projection of the line of sight onto the horizontal plane intersects with a grid element in the two-dimensional projection map that indicates that none of the volume elements at their horizontal positions in the voxelized representation are volume elements through which sound can propagate, Applying a path discovery algorithm to determine the path in the two-dimensional projection map between the two-dimensional projection of the source position and the two-dimensional projection of the listener position, along grid elements indicating that at least one volume element at each horizontal position in the voxelized representation is a volume element on which sound can propagate, Based on the determined path, the parameter indicating the virtual source location is determined. The method according to claim 1, including the method described in claim 1.

5. If the path discovery algorithm fails to identify a path in the two-dimensional projection map between the two-dimensional projection of the source location and the two-dimensional projection of the listener location, To obtain a refined voxelized representation and a refined 2D projection map, Applying the path discovery algorithm to determine the path in the refined two-dimensional projection map between the two-dimensional projection of the source position and the two-dimensional projection of the listener position, along grid elements indicating that at least one volume element at each horizontal position in the refined voxelized representation is a volume element on which sound can propagate. It further includes, The refined two-dimensional projection map is associated with the refined voxelized representation by the projection operation that projects it onto the horizontal plane. The volume element of the refined voxelized representation has a shorter side length than the volume element of the voxelized representation. The method according to claim 3 or 4.

6. Determining the parameter indicating the virtual source location is, Based on the determined path, the distance and azimuth angle between the listener position and the virtual source position are determined. The method according to any one of claims 3 to 5, including the method described in any one of claims 3 to 5.

7. The distance is determined based on the path length of the determined path, and / or Determining the aforementioned azimuth angle means Starting from the two-dimensional projection of the listener position, the nearest directional change in the determined path is identified, The aforementioned azimuth angle is determined as the azimuth angle between the two-dimensional projection of the listener position and the identified nearest change in direction. including, The method according to claim 6.

8. Determining the parameter indicating the virtual source location is, Based on the determined path and the voxelized representation, the elevation angle between the listener position and the virtual source position is determined. The method according to any one of claims 3 to 7, including the method described in any one of claims 3 to 7.

9. Determining the aforementioned elevation angle means Starting from the two-dimensional projection of the listener position, the nearest directional change in the determined path is identified, In the horizontal position of the identified directional change, determine the volume element in the voxelized representation from which sound can propagate. The aforementioned elevation angle is determined as the elevation angle between the listener position and the determined volume element. The method according to claim 8, including the method described in claim 8.

10. Determining the parameter indicating the virtual source location is, Calculating the line of sight between the source position and the listener position in the three-dimensional audio scene, If the calculated line of sight intersects with volume elements in the voxelized representation where sound is blocked, and the two-dimensional projection of the line of sight onto the horizontal plane does not intersect with grid elements in the two-dimensional projection map that indicate that none of the volume elements at their horizontal positions in the voxelized representation are volume elements through which sound can propagate, Based on the two-dimensional projection of the line of sight onto the horizontal plane, the distance and azimuth angle between the listener position and the virtual source position are determined. The method according to claim 1, including the method described in claim 1.

11. Determining the parameter indicating the virtual source location is, Based on the line of sight and the voxelized representation, the elevation angle between the listener position and the virtual source position is determined. Furthermore, optionally, Determining the aforementioned elevation angle means Starting from the listener position, the nearest volume element where the calculated line of sight intersects and the sound is blocked is identified. In the horizontal position of the identified volume element, determine the volume element through which sound can propagate in the voxelized representation. The aforementioned elevation angle is determined as the elevation angle between the listener position and the determined volume element. The method according to claim 10, including the method described in claim 10.

12. Obtaining the voxelized representation of the three-dimensional audio scene is: Receiving the mesh representation of the aforementioned three-dimensional audio scene, The voxelization algorithm is applied to the mesh representation of the three-dimensional audio scene to generate the voxelized representation. Includes, optional Obtaining the voxelized representation of the three-dimensional audio scene is: The process further includes removing any voxels corresponding to local occluders from the generated voxelized representation, where local occluders represent objects in the three-dimensional audio scene that are not expected to affect the large-scale directivity of sound propagation. The method according to any one of claims 1 to 11.

13. Determining the virtual source signal from the source signal of the sound source, Rendering the virtual source signal from the virtual source position to the listener position Furthermore, optionally, Determining the aforementioned virtual source signal means Using the aforementioned source signal as the virtual source signal, or A filter is applied to the source signal to generate the virtual source signal. The method according to any one of claims 1 to 12, comprising one of the following.

14. The volume element of the voxelized representation has an occlusion coefficient having a value between a first value indicating free sound propagation and a second value indicating complete occlusion. The generation of the two-dimensional projection map includes a truncation operation such that the grid elements of the two-dimensional projection map represent either free sound propagation or complete occlusion. The method according to any one of claims 1 to 13.

15. Obtaining the two-dimensional projection map after obtaining the voxelized representation is performed each time the three-dimensional audio scene is updated, and / or Determining the parameter indicating the virtual source location is performed each time an update occurs to the source location or the listener location. The method according to any one of claims 1 to 14.

16. An apparatus comprising a processor coupled to a memory for storing instructions for the processor, wherein the processor is adapted to cause the apparatus to perform the method according to any one of claims 1 to 15.

17. A computer program comprising instructions causing a processor that executes instructions to perform the method according to any one of claims 1 to 15.

18. A computer-readable storage medium storing the computer program described in claim 17.

Images