Apparatus and method for rendering an audio scene using effective intermediate diffraction paths - Patents.com

By precomputing intermediate diffraction paths and using filter information independent of listener and audio source positions, the method addresses inefficiencies in rendering audio scenes with complex, dynamic environments, achieving efficient and accurate sound simulation.

JP7579626B2Active Publication Date: 2024-11-08FRAUNHOFER GESELLSCHAFT ZUR FORDERUNG DER ANGEWANDTEN FORSCHUNG EV
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
JP2022555051
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2020-03-13
Filing Date
2021-03-12
Publication Date
2024-11-08
Estimated Expiration
2041-03-12

AI Technical Summary

Technical Problem

Current methods for rendering audio scenes in virtual or augmented reality are inefficient and computationally expensive, particularly when dealing with complex, dynamic environments involving multiple diffractive objects, due to the high processing requirements for modeling sound diffraction effects.

Method used

The method employs precomputation of intermediate diffraction paths between edges in a scene, providing complete filter information that is independent of listener and audio source positions, reducing runtime computations by calculating these paths during initialization and using them for real-time rendering.

Benefits of technology

This approach significantly enhances processing efficiency by minimizing runtime computations, allowing for accurate and efficient simulation of diffracted sound in complex virtual reality scenes with both static and dynamic objects, while reducing computational complexity.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007579626000002
    Figure 0007579626000002
  • Figure 0007579626000003
    Figure 0007579626000003
  • Figure 0007579626000004
    Figure 0007579626000004
Patent Text Reader

Abstract

An apparatus for rendering an audio scene (50) including an audio source at an audio source position and a plurality of diffractive objects comprises: a diffraction path provider (100) for providing a plurality of intermediate diffraction paths (300, 400) through the plurality of diffractive objects, the intermediate diffraction paths having starting points and output edges of the plurality of diffractive objects, and associated filter information for the intermediate diffraction paths; and a renderer (200) for rendering the audio source to a listener position, wherein the renderer (200) is configured to determine (216) one or more effective intermediate diffraction paths from the audio source position to the listener position based on the output edges of the intermediate diffraction paths and the listener position; determine (218) a filter representation of a full diffraction path for each effective intermediate diffraction path of the one or more effective intermediate diffraction paths; and calculate (220) an audio output signal of the audio scene (50) using an audio signal associated with the audio source and the filter representation of each full diffraction path.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to audio signal processing, in particular in the context of geometrical acoustics, which may be used, for example, in virtual or augmented reality applications. [Background technology]

[0002] The term virtual acoustics is often applied when sound signals are processed to include the characteristics of a simulated acoustic space and the sound is reproduced spatially using either binaural or multichannel techniques. Virtual acoustics thus consists of spatial sound reproduction and room acoustics modeling [1].

[0003] Concerning room modeling techniques, the most accurate method for propagation modeling is to solve a theoretical wave equation subject to a set of boundary conditions. However, most of the methods based on numerical solutions are limited to precalculating relevant acoustic features such as parametric models that approximate the impulse response due to the computational complexity, i.e., when the frequency of interest and / or the size (volume / surface) of the scene space increase and even when dynamically moving objects are present. Given the fact that modern virtual scenes are becoming larger and more sophisticated to allow highly detailed and delicate interactions between players and objects in the scene or between players, current numerical methods are not adequate to handle interactive dynamic and large-scale virtual scenes. Several algorithms have shown their rendering capabilities by precalculating relevant acoustic features using parametric directional coding for precalculated sound propagation [2, 3] and efficient GPU-based time-domain solvers for the acoustic wave equation [4]. However, these methods require high-quality system resources such as graphics cards and multi-core computing systems.

[0004] Geometric acoustics (GA) techniques are practically reliable methods for interactive sound propagation environments. Commonly used GA techniques include image source method (ISM) and ray tracing (RTM) [5, 6], and modified methods using beam tracing and frustum tracing have been developed for interactive environments [7, 8]. For diffracted sound modeling, Kouryoumjian [9] proposed the Uniform Theory of Diffraction (UTD) and Svensson

[10] proposed the Biot-Tolstoy-Medwin (BTM) model for a better approximation of diffracted sound in a numerical sense. However, current interactive algorithms are limited to static scenes

[11] or first-order diffraction in dynamic scenes

[12] .

[0005] By combining these two categories, i.e., numerical methods for low frequencies and GA methods for high frequencies, hybrid methods are possible

[13] .

[0006] In particular, complex sound scenes with several diffractive objects have high processing requirements for modeling sound diffraction around edges, and therefore very powerful computational resources are needed to adequately model sound diffraction effects in audio scenes with multiple diffractive objects. Summary of the Invention [Problem to be solved by the invention]

[0007] It is an object of the present invention to provide an improved concept for rendering audio scenes. [Means for solving the problem]

[0008] This object is achieved by an apparatus for rendering an audio scene according to claim 1, or a method for rendering an audio scene according to claim 19, or a computer program according to claim 20.

[0009] The invention is based on the finding that the processing of sound wave diffraction can be significantly improved by using intermediate diffraction paths between the start or input edge and the final or output edge of a sound scene, which already have the relevant filter information. This relevant filter information already covers the entire path between the start and final edge, regardless of whether there is a single or multiple diffraction between the start and final edges. The procedure relies on the fact that the path between the start and final edges, i.e. the path that the sound wave must travel due to the diffraction effect, does not depend on the listener position, which is usually variable, nor on the audio source position. Even when the audio source also has a variable position, only the variable source position or the variable listener position changes from time to time, but any intermediate diffraction path between the start and final edges of the diffractive object does not depend on anything other than the geometry. This diffraction path is constant, since it is only defined by the diffractive object provided by the audio scene geometry. This kind of path is only variable over time when one of the multiple diffractive objects changes its shape, which means that this kind of path does not change for a moving rigid geometry. Furthermore, the multiple objects in the audio scene are static, i.e. not movable. Providing complete filter information for the entire intermediate diffraction path improves processing efficiency, especially at run-time. Filter information for intermediate diffraction paths that are not ultimately used because it has not been explicitly verified also needs to be calculated, but this calculation can be performed in the initialization / encoding step and does not need to be performed at run-time. In other words, any run-time processing on the filter information or on the intermediate diffraction paths needs to be done only for dynamic objects, which typically occur rarely, while for static objects, which typically occur, the filter information associated with a particular intermediate diffraction path always remains the same regardless of any moving audio source for any moving listener.

[0010] An apparatus for rendering an audio scene including an audio source at an audio source location and multiple diffractive objects comprises a diffraction path provider for providing multiple intermediate diffraction paths through the multiple diffractive objects, the intermediate diffraction paths having start points or starting edges and output or final edges of the multiple diffractive objects and associated filter information of the intermediate diffraction paths describing the entire sound propagation by diffraction from the start points or starting edges to the output edges or output or final points. Typically, the multiple intermediate diffraction paths are provided by a pre-processor in an initialization or pre-computation step that occurs before the actual run-time processing, for example in a virtual reality environment. The diffraction path provider does not need to calculate all this information at run-time, and can provide this information, for example, as a list of intermediate diffraction paths that can be accessed by a renderer during run-time processing.

[0011] The renderer is configured to render the audio source to the listener position, the renderer configured to determine one or more effective intermediate diffraction paths from the audio source position to the listener position based on the output edge of the intermediate diffraction path and the listener position. The renderer is configured to determine, for each effective intermediate diffraction path of the one or more effective intermediate diffraction paths, a filter representation for a full diffraction path from the audio source position to the listener position, corresponding to an effective intermediate diffraction path of the one or more effective intermediate diffraction paths, using a combination of associated filter information for the effective intermediate diffraction path and filter information describing audio signal propagation from an output edge or from a final edge of the effective intermediate diffraction path to the listener position. An audio output signal of the audio scene can be calculated using the audio signal associated with the audio source and the complete filter representation of each full diffraction path.

[0012] Depending on the application, the audio source position is fixed, and then the diffraction path provider determines each effective intermediate diffraction path such that the starting point of each effective intermediate diffraction path corresponds to the fixed audio source position. Alternatively, when the audio source position is variable, the diffraction path provider determines the input or starting edge of the multiple diffracting objects as the starting point of the intermediate diffraction path. The renderer is configured to determine one or more effective intermediate diffraction paths based further on the input edge of one or more intermediate diffraction paths and the audio source position of the audio source, i.e., to determine the path that can belong to a specific audio source position to determine the final filter representation of the full diffraction path based further on another filter information from the source to the input edge, in this case the full filter representation is configured to be determined by three parts. The first part is the filter information for the sound propagation from the sound source position to the input edge. The second part is the relevant information belonging to the effective intermediate diffraction path, and the third part is the sound propagation from the output or final edge to the actual listener position.

[0013] The present invention is advantageous because it provides an efficient method and system for simulating diffracted sound in complex virtual reality scenes. The present invention is advantageous because it enables modeling of sound propagation through static and dynamic geometric objects. In particular, the present invention is advantageous in that it provides a method and system on how to calculate and store diffraction path information based on a set of a priori known geometric primitives. In particular, the diffracted sound path includes a set of attributes such as potential diffraction edges, diffraction angles, and groups of geometric primitives of intermediate diffraction edges.

[0014] The present invention is advantageous because it allows the analysis of the geometric information of a given primitive and the extraction of a useful database via a preprocessor in order to increase the speed at which the present invention renders sound in real time. In particular, the procedure disclosed in, for example, US Patent Application Publication No. 2015 / 0378019, or other procedures described below, allows precomputing a visibility graph between edges that minimizes the number of diffractive edges whose structure needs to be considered at run time. Visibility between two edges does not necessarily mean that an exact path from the source to the listener is specified, since in the precomputation stage the locations of the source and the listener are usually not known. Instead, the visibility graph between all possible pairs of edges is a map that navigates from the set of visible edges from the source to the set of visible edges from the listener. Preferred embodiments of the present invention are discussed below with reference to the accompanying drawings. [Brief description of the drawings]

[0015] [Figure 1] FIG. 1 is a top view of an example scene with four static objects. [Figure 2a] FIG. 1 is a top view of an example scene with four static objects and a single dynamic object. [Figure 2b] 1 is a list illustrating the diffraction paths without and with a dynamic object (DO). [Diagram 3] FIG. 1 is a top view of an example scene with six static objects. [Figure 4] FIG. 13 is a top view of an example scene with six static objects illustrating how to calculate higher order diffraction paths from a first or input edge. [Diagram 5] FIG. 1 is a block diagram of an algorithm for pre-calculating intermediate diffraction paths (including higher-order paths) and rendering the diffracted sound in real-time. [Figure 6] FIG. 13 is a block diagram of an algorithm according to a third preferred embodiment for pre-calculating intermediate diffraction paths (including higher order paths) and rendering diffracted sound in real-time taking into account dynamic objects. [Figure 7]FIG. 2 illustrates an apparatus for rendering a sound scene according to a preferred embodiment. [Figure 8] FIG. 5 illustrates an exemplary path list having two intermediate diffraction paths shown in FIGS. 4 and 3. [Figure 9] FIG. 1 illustrates a procedure for computing a filter representation of a full diffraction path. [Figure 10] FIG. 13 illustrates a preferred embodiment for retrieving filter information related to effective intermediate diffraction paths. [Figure 11] FIG. 1 illustrates a procedure for validating one or more potentially valid intermediate diffraction paths to obtain an effective intermediate diffraction path. [Figure 12] FIG. 2 illustrates a rotation performed to a pre-rotation or original source position to enhance the audio quality of a rendered audio scene. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Fig. 7 shows an apparatus for rendering an audio scene including an audio source with an audio source signal at an audio source position and a number of diffractive objects. The diffraction path provider 100 comprises a storage device filled by a pre-processor performing calculations of intermediate diffraction paths, for example in an initialization step, i.e. before the runtime processing operations performed by the renderer 200. Depending on the information of the list of intermediate diffraction paths obtained by the diffraction path provider 100, the renderer is configured to calculate an audio output signal in a desired output format, such as a binaural format, a stereo format, a 5.1 format or any other output format, to a headphone speaker, or to a loudspeaker, or simply for storage or transmission. For this purpose, the renderer 200 receives the list of intermediate diffraction paths, as well as the listener positions and the audio source signal on the one hand and the audio source positions on the other hand.

[0017] In particular, the renderer 200 is configured to render an audio source to a listener position, and thus a sound signal that arrives at the listener position is calculated. This sound signal exists because the audio source is located at the audio source position. For this purpose, the renderer is configured to determine one or more effective intermediate diffraction paths from the audio source position to the listener position based on the output edge of the intermediate diffraction path and the actual listener position. The renderer is also configured to determine, for each effective intermediate diffraction path of the one or more effective intermediate diffraction paths, a filter representation for a full diffraction path from the audio source position to the listener position, corresponding to a certain effective intermediate diffraction path of the one or more effective intermediate diffraction paths, using a combination of the associated filter information for the effective intermediate diffraction path and the filter information describing the audio signal propagation from the output edge of the effective intermediate diffraction path to the listener position.

[0018] The renderer uses the audio signals associated with the audio sources and the filter representation of each full diffraction path to calculate an audio output signal for the audio scene. Depending on the implementation, the renderer may be configured to further calculate first order, second order or higher order reflections in addition to the diffraction calculation, and further the renderer may be configured to calculate contributions from one or more additional audio sources, if present in the sound scene, as well as direct sound propagation contributions from sources that have a direct sound propagation path that is not obstructed by diffractive objects.

[0019] Subsequently, preferred embodiments of the invention will be described in more detail. In particular, any first order diffraction path can be practically calculated in real time if desired, which is even more problematic for complex scenes with several diffractive objects.

[0020] Especially for higher order diffraction paths, the real-time calculation of such diffraction paths is problematic due to a significant amount of redundant information in the visibility map as shown in US2015 / 0378019. For example, in a scene as in FIG. 1, when the source is located to the right of the first edge and the listener is located to the left of the fifth edge, the diffracted sound can be imagined to go from the source to the listener via the first and fifth edges. However, building diffraction paths edge by edge in real time based on the visibility graph becomes computationally complex, especially as the average number of visible edges increases and the diffraction order becomes higher. Furthermore, no runtime edge-to-edge visibility checks are performed, which limits the diffraction effects between dynamic objects and between one static object and one dynamic object. It is only possible to take care of the diffraction effects of static objects or a single dynamic object. The only way to combine the diffraction effects due to dynamic object(s) with the effects related to static objects is to update all visibility graphs with the repositioned primitives of the dynamic objects. However, this is almost impossible at runtime.

[0021] The method of the present invention aims to reduce the runtime calculations required to specify possible (first / higher order) diffraction paths through edges of static and dynamic objects from a source to a listener. As a result, a set of multiple diffracted sound / audio streams is rendered with appropriate delays. An embodiment of the preferred concept is applied to multiple visible appropriately oriented edges with a newly designed system hierarchy using the UTD model. As a result, the embodiment can render high order diffraction effects by static geometry, by dynamic objects, by a combination of static geometry and dynamic objects, or even by a combination of multiple dynamic objects. More detailed information on the preferred concept is presented in the following subsections.

[0022] The main idea that started this embodiment started from the question: "Do we need to keep calculating intermediate diffraction paths?" For example, as shown in FIG. 3, we may say that there are three possible diffraction paths from the source to the listener, namely (source)-(1)-(5)-(listener), (source)-(9)-(13)-(listener), and (source)-(1)-(7)-(11)-(listener). For simplicity of explanation, the last path through three intermediate edges including (1), (7), and (11) is a good example. In an interactive environment, the source can move and the listener can also move. However, in either situation, the intermediate paths including (1)-(7), (7)-(11), and (11)-(13) do not change unless there is a dynamic object that occludes these intermediate paths. (Note that a method for handling / combining the diffraction effect due to dynamic objects will be presented at the end of this section.)

[0023] Therefore, being able to pre-compute intermediate paths from first order to any allowed higher order depending on edges, adjacent polygons (e.g., triangular meshes), and diffraction angles in the intermediate paths minimizes the required computations at run-time.

[0024] For example, Figure 4 shows an example scene with six static objects to demonstrate how to calculate high-order diffraction paths from an edge. In this case, the diffraction path starts from the first edge, and the edge can contain some correlation information such as: struct DiffractionEdge { const EAR::Geomtery* parentGeometry; int meshID; int edgeID; std::pair < ​​EAR::Vector3, EAR::Vector3 > vtxCoords; std::pair < ​​EAR::Mesh::Triangle*, EAR::Mesh::Triangle* > adjTris; float internalAngle; std::vector < DiffractionEdge* > visibleEdgeList; };

[0025] For example, parentGeometry or meshID indicate the geometry to which the selected edge belongs. Furthermore, an edge can be physically defined as a line of two vertices (by the coordinates or vertex ids of those vertices), and the neighboring triangles help to calculate the angle from the edge, source or listener. internalAngle is the angle between two neighboring triangles that indicates the maximum possible diffraction angle around this edge. It is also an indicator that can determine if this edge is a potential diffraction edge or not.

[0026] From a selected edge (in this case, the first edge shown in Figure 4), we can image two possible diffraction orientations: from one of the triangle mesh into open space and from the other. These orientations are visualized by the normal vectors of the neighboring triangles shown as red and blue arrows. For example, along the red surface normal (in a counterclockwise direction), edges no. 2, no. 4, no. 5, and no. 7 become the next edges for diffraction by checking if there is any margin (i.e., dark zone) for the wave to be diffracted. For example, at edge no. 6, the sound wave cannot diffract from edge no. 1 to edge no. 6 because both sides of edge no. 6 are visible from edge no. 1, which means there is no dark zone due to edge no. 6 for the sound wave coming from edge no. 1. And as a next step, from edge no. 7, we can find the next possible diffraction edges as no. 10 and no. 11. For example, if we navigate to edge no. 11, we can calculate the intermediate angle from edges no. 1, no. 7 to no. 11. This intermediate angle is defined as the angle between an inward wave and an outward wave, where the inward wave to edge number 7 is the vector from edge number 1 to number 7, and the outward wave is the vector from number 7 to number 11. Then, we define this angle as TIFF0007579626000001.tif22 1-7-11 However, there is no such intermediate angle at the beginning or end of the path. Instead, a maximum acceptable angle (MAAS) of the source and a minimum acceptable angle (MAAL) of the listener can be assigned. This means that if the source angle with respect to the relevant surface normal (in this case Red at edge number 1) is greater than MAAS, the source can see the second edge (e.g. number 7). With the same concept, if the listener has an angle smaller than a given MAAL, the listener can see the edge before the last edge in the path. In real time, the source and listener angles with respect to the relevant surface normal can be calculated based on the MAAL and MAAS values, and then this path can be verified. Thus, the pre-computed quaternary path 400 in the scene of FIG. 4 can be defined as a vector of edges, triangles, and angles, as shown in the top part of FIG. 8.

[0027] The full procedure of the preferred pre-computation of intermediate paths and associated real-time rendering algorithm is in Figure 5. Pre-computing all possible diffraction paths in the scene, we only need to find the visible edge list from the source location as the starting point of diffraction, and from the listener location as the final point, only if the direct path between the source and the listener is obstructed. Then, we need to calculate the angle of the source to the associated triangle (e.g., 1-R in Table 1) and the angle of the listener to the triangle (e.g., 12-B in Table 1). If the source angle is smaller than MAAS and the listener angle is larger than MAAL, it becomes a valid path along which the sound source signal will be propagated. Then, the edge vertex information and the associated angle can be used to update the source location and the diffraction filter. The binaural rendering module synthesizes the diffracted source information (rotation and filtering) with appropriate directional filters such as head-related transfer functions (HRTFs). It is possible to add more features to the diffraction framework, such as directional effects and distance effects.

[0028] FIG. 1 shows an audio scene with four diffractive objects, and there is a first-order diffraction path between edge 1 and edge 5. The top of FIG. 2b shows the first-order diffraction path from edge 1 to edge 5 with an angle criterion that the source angle with respect to the starting or input edge 1 must be smaller than the maximum acceptable angle of the source. This is the case in FIG. 1. The same is true for the minimum acceptable angle of the listener (MAAL). In particular, the angle of the listener position with respect to the output or final edge, calculated from the edge between vertex 5 and vertex 6 in FIG. 1, is larger than the minimum acceptable angle of the listener (MAAL). For the current source position in FIG. 1 and the current listener position in FIG. 1, the diffraction path as shown in the top of FIG. 2b is active, and the diffraction characteristics between edge 1 and edge 5 are relevant filter information that can be pre-stored in relation to the diffraction path without dynamic objects, or can be easily calculated using the edge list in FIG. 2b.

[0029] FIG. 4 shows an intermediate diffraction path 400 going from an input or starting edge to an output or final edge 12. FIG. 3 shows another intermediate diffraction path 300 going from a starting edge 1 to a final edge 13. The audio scenes in FIG. 3 and FIG. 4 have an additional diffraction path from the source to edge 9, then to edge 13, then to the listener, or from the source to edge 1, from there to edge 5, and from this edge to the listener. However, the path 300 in FIG. 3 is only determined to be a valid intermediate diffraction path for the listener position shown in FIG. 3, which satisfies the minimum acceptable angle of the listener, i.e. the angle criterion MAAL. However, the listener position shown in FIG. 4 does not satisfy this MAAL criterion for path 300.

[0030] On the other hand, the listener position in Fig. 3 does not meet the MAAL criteria for the path 400 shown in Fig. 4. Therefore, the diffraction path provider 100 or pre-processor forwards a number of intermediate diffraction paths shown in Fig. 8, including the intermediate diffraction path 400 shown in Fig. 4 as a first list entry and providing the other intermediate diffraction paths 300 shown in Fig. 3. This list of intermediate diffraction paths is provided to a renderer, which determines one or more effective intermediate diffraction paths from the audio source position to the listener position. For the listener position shown in Fig. 3, only the path 300 in the list in Fig. 8 is determined as an effective intermediate diffraction path, and for the listener position shown in Fig. 4, only the path 400 is determined as an effective intermediate diffraction path.

[0031] Referring to the scenario of Figure 3 or Figure 4, any intermediate diffraction paths with final or output edges 15, 10, 7 or 2 are not determined to be valid intermediate diffraction paths because these edges are not visible to the listener at all. The renderer 200 of Figure 7 selects only those intermediate diffraction paths with final or output edges visible to the listener, i.e. edges 4, 5, 12 and 13 in this example, from all intermediate diffraction paths theoretically possible according to the audio scene of Figure 4. This corresponds to an edge list (lis).

[0032] Similarly, for the source position, none of the pre-computed intermediate diffraction paths provided in the list of intermediate diffraction paths coming from the diffraction path provider 100 of Fig. 7 having as starting edge, for example, edges 3, 6, 11 or 14, are selected at all. Only the diffraction paths having starting edges 1, 8, 9, 16 are selected for specific validation using the MAAS angle criteria. These edges are in the edge list (src).

[0033] In summary, the determination of the actual effective intermediate diffraction paths, from which the filtered representation of the sound propagation from the source to the listener is finally determined, is selected in a three-stage procedure. In the first stage, only pre-stored diffraction paths with a starting edge that coincides with the source position are selected. In the second stage, only such intermediate diffraction paths with an output edge that coincides with the listener position are selected, and in the third stage, each of those selected paths is verified using the angular criteria of the source on the one hand and the listener on the other hand. Then, only the intermediate diffraction paths that survive all three stages are used by the renderer to compute the audio output signal.

[0034] FIG. 10 shows a preferred embodiment of the selection information. In step 102, a potential start edge for a particular source position is determined using geometry information of the audio scene, using the source position, in particular using a number of intermediate diffraction paths already precalculated by a preprocessor. In step 104, a potential end edge for a particular listener position is determined. In step 102, potential intermediate diffraction paths are determined based on the result of step 102 and the result of block 104, which corresponds to the first and second stages described above. In step 108, the potential intermediate diffraction paths are verified using the angle conditions MAAS or MAAL, or in general by a visual decision to determine whether an edge is a diffraction edge or not. Step 108 shows the third stage described above. The input data MAAS and MAALS are obtained, for example, from the list of intermediate diffraction paths shown in FIG. 8.

[0035] FIG. 11 illustrates a further sequence of the set of steps performed in block 108 for validating potential intermediate diffraction paths. In step 112, a source position angle is calculated relative to the starting edge, which corresponds, for example, to the calculation of angle 113 in FIG. 1. In step 114, a listener position angle is calculated relative to the ending edge, which corresponds, for example, to the calculation of angle 115 in FIG. 1. In step 116, the source position angle is compared to the maximum allowable angle of the source, MAAS, and if it is determined that the comparison results in a situation where the angle is greater than MAAS, the test fails, as shown in block 120. However, if it is determined that angle 113 is less than MAAS, the initial validity test passes.

[0036] Nevertheless, an intermediate diffraction path is only a valid intermediate diffraction path if it also passes the second verification. This is obtained by the result of block 118, i.e., MAAL is compared with the listener position angle 115. When the angle 115 is greater than MAAL, a second contribution to pass the validity test is obtained as shown in block 122, and filter information is retrieved from the intermediate diffraction path list as shown in step 126, or, for example, the filter information is calculated according to the data in the list in case of a parametric representation according to the intermediate angle as shown in the list of FIG.

[0037] As soon as the filter information related to the effective intermediate diffraction path is obtained, as is the case after step 126 of FIG. 11, the audio renderer 200 of FIG. 7 must calculate the final filter information as shown in FIG. 9. In particular, step 126 of FIG. 9 corresponds to step 126 of FIG. 11. In step 128, the starting filter information from the source position to the starting edge of the effective intermediate diffraction path is determined. In particular, this is the filter information describing the audio propagation of the source to the vertex of the edge (1) in FIG. 1, for example. This propagation information does not only refer to the attenuation due to distance, but also depends on the angle. As known from the geometric theory of diffraction (GTD) or the uniform theory of diffraction (UTD) or other models of sound diffraction possible to the present invention, the frequency characteristics of the diffracted sound depend on the diffraction angle. When the source angle 113 is very small, generally only the low frequency part of the sound is diffracted, and the high frequency part is more strongly attenuated compared to the situation when the source angle 113 approaches the MAAS angle of FIG. 1. In such a situation, the high frequency attenuation is reduced compared to the case when the source angle is approaching 0 or is very small.

[0038] Similarly, the final filter information from the final or output edge 5 to the listener position is determined, again based on the listener angle 115 relative to the MAAL. Then, as soon as those three filter information items or filter contributions are determined, they are combined in step 132 to obtain a filter representation of the full diffraction path, which includes the path from the source to the start edge, the intermediate diffraction paths, and the path from the output or final edge to the listener position. The combination can be done in many ways, one effective way is to convert each of the three filter representations obtained in steps 128, 126 and 130 into a spectral representation to obtain the corresponding transfer functions, and then multiply the three transfer functions in the spectral domain to obtain a final filter representation that can be used as is if the audio renderer operates in the frequency domain. In an alternative method, the frequency domain filter information can be converted to the time domain if the audio renderer operates in the time domain. Alternatively, the three filter items can be convolved with time domain filter impulse responses representing the individual filter contributions, and the resulting time domain filter impulse responses can be used by the audio renderer for rendering. In this case, the renderer performs a convolution operation between the audio source signal on the one hand and the full filtered representation on the other hand.

[0039] Subsequently, FIG. 5 is shown to provide a flow chart of a preferred embodiment of the rendering of a static differential object. The procedure starts at block 202. Then, a pre-calculation step is provided to generate the list provided by the diffraction path provider 100 of FIG. 7. In step 204, the tracer is set with a mesh for occlusion testing. This step determines all the different diffraction paths between edges, and a diffraction path can only occur, by definition, when a direct path between two non-adjacent edges is blocked. For example, considering FIG. 3, the path between edge 1 and edge 11 is blocked by edge 7, and therefore diffraction can occur. Such a situation is determined, for example, by block 204. In block 206, a potential diffraction edge list is calculated using the interior angles between two adjacent triangles. This procedure determines the intermediate or interior angles of such diffraction path portions, i.e., the portion between edge 1 and edge 11, for example. This step 206 also determines further diffraction path portions between edge 7 and edge 12 via edge 11, or between edge 7 and edge 13 via edge 11. The corresponding diffraction paths are pre-computed, for example as shown in FIG. 8, so that for example path 300 in FIG. 3 or for example path 400 in FIG. 4 are pre-computed with associated filter information describing the entire sound propagation from edge 1 to edge 13 in path 300, or from edge 1 to edge 12 in path 400 in FIG. 4. The pre-computation procedure is completed and the run-time steps are performed. In step 210, the renderer 200 obtains data of the source position and the listener position. In step 212, a direction path occlusion test between the source and the listener is performed. The procedure continues only if the result of the test in block 212 is that the direction path is occluded. If the direct path is not occluded, direct propagation occurs and no diffraction is a problem for this path.

[0040] In step 214, a visible edge list from one source and a visible list from the other listener are determined. This procedure corresponds to steps 102, 104 and 106 in FIG. 6. In step 216, it is verified that the path starts from an input edge of the edge list and ends at an output edge of the listener's edge list. This corresponds to the procedure performed in block 108 in FIG. 10. In step 218, a filter representation is determined so that the source location can be updated by rotation with respect to the relevant edge, e.g. the diffraction filter from the UTD model database can be updated. In general, however, the invention is not limited to UTD model database applications, and the invention can be applied using any particular calculation and application of filter information from the diffraction path. In step 220, the audio output signal of the audio scene is calculated by binaural rendering using the associated delay line module therein, for example to render the distance effect if the distance effect is not included in the corresponding binaural rendering directional filter, such as a particular HRTF filter.

[0041] FIG. 12 shows a rotation made to the pre-rotated source location to enhance the audio quality of the rendered audio scene. This rotation is preferably applied in step 218 of FIG. 5 or step 218 of FIG. 6. Rotation of the source location for rendering or spatialization purposes is useful to improve the spatial perception with respect to the original source location. Thus, with reference to FIG. 12, the sound source is rendered to a new location 142 obtained by rotating from the original sound source location 143 around edge 9 through an angle DA_9 to the intermediate location 141. This angle is determined by the line connecting edge 13 and edge 9 so that a straight line is obtained. The intermediate location 141 is then rotated around edge 13 through an angle DA_13 in order to have a straight line from the listener to the final rotated source location 142. Thus, not only are the frequency-dependent equalization or attenuation values ​​spatialized, but also the perceived direction of the original source at the rotated source location 142 is spatialized. This final rotated source location is the perceived source location due to sound wave diffraction effects that change the angle of sound propagation in each diffraction process.

[0042] See one example diffraction path from source to listener: "Source-(9)-(13)-Listener." Additional phi, theta information for spatial sound reproduction is generated using the position 142 of the source after rotation.

[0043] The complete relevant filter information, taking into account the exact source / listener location, already provides the exact EQ information per frequency, i.e. the attenuation due to diffraction effects. Using the original source position and the distance to the original source already constitutes the low-level implementation. This low-level implementation is enhanced by further creating the information required to select the appropriate HRTF filter. For this purpose, the original sound source is rotated by the amount of the diffraction angle relative to the relevant edge to generate the position of the diffracted source. The azimuth and elevation angles can then be derived from this position relative to the listener, giving the total propagation distance along the path.

[0044] 12 also shows the calculation of the distance between the final rendered sound source position 142 and the listener position obtained by the rotation process. This distance is preferably further used to determine the delay of both distance-dependent attenuations for rendering the source.

[0045] Subsequently, the use and determination of the position 143 after rotation of the original source position 143 is further described. Each step of the calculation to obtain the effective path deals with the original source position 143. However, in order to realize binaural rendering for a user equipped with headphones to feel a better immersive sound in the VR space, the location of the sound source is preferably given to the binauralizer so that the binauralizer can apply appropriate spatial filtering (H_L and H_R) to the original audio signal, where H_L / H_R is called the Head Related Transfer Function (HRTF), as described for example in https: / / www.ece.ucdavis.edu / cipic / spatial-sound / tutorial / hrtf / . Filtered_S_L = H_L(phi, theta, w) * S(w) Filtered_S_R = H_R(phi, theta, w) * S(w)

[0046] The mono signal S(w) does not have any localization cues that are used to generate spatial sound. However, the filtered sound by HRTF can recreate the spatial impression. To do this, phi and theta (i.e., the relative azimuth and elevation angles of the diffracted source) should be given throughout the process. This is why the original sound source is rotated. Therefore, the renderer receives the information of the final source position 142 in FIG. 12 in addition to the filter information. In low-level implementations, it is generally possible to use the direction of the original sound source 143 so that the complexity of source rotation can be avoided, but this procedure results in a direction error seen in FIG. 12. However, in low-level implementations, this error is acceptable. The same is true for the sense of distance. As shown in FIG. 12, the distance to the listener of the source 142 after rotation is somewhat longer than the distance to the listener of the original source 143. This distance error can be accepted in low-level injection to reduce the complexity. However, in high-level applications, this error can be avoided.

[0047] Thus, generating the location of a diffracted sound source by rotating the original source about the relevant edge can also provide the propagation distance from the source and listener, which is used for attenuation with distance.

[0048] This process of generating additional information for phi, theta, and distance is also useful for multi-channel playback systems, the only difference being that in that case a different set of spatial filters is applied to S(w) to provide Filtered_S_i to the i-th loudspeaker as Filtered_S_i=H_i(phi, theta, w, other parameters)*S(w).

[0049] A preferred embodiment relates to the operation of a renderer configured to calculate that the audio source position after rotation, depending on the effective intermediate diffraction path or depending on the full diffraction path, differs from the audio source position due to the diffraction effects experienced by the effective intermediate diffraction path or depending on the full diffraction path, and to use the rotated position of the audio source in calculating (220) an audio output signal of the audio scene, or to calculate the audio output signal of the audio scene using, in addition to the filter representation, an edge sequence associated with the full diffraction path and a diffraction angle sequence associated with the full diffraction path.

[0050] In another embodiment, the renderer is configured to determine the distance from the listener position to the rotated source position and use this distance when calculating the audio output signal for the audio scene.

[0051] In another embodiment, the renderer is configured to select one or more directional filters depending on the rotated source position and a predetermined output format of the audio output signal, and to apply the one or more directional filters and filter expressions to the audio signal when calculating the audio output signal.

[0052] In another embodiment, the renderer is configured to determine an attenuation value depending on the distance between the rotated source position and the listener position and apply it to the audio signal in addition to the filter representation or one or more directional filters depending on the audio source position or the rotated audio source position. In another embodiment, the renderer is configured to determine the rotated source position in a series of rotation operations including at least one rotation operation.

[0053] In the first step of the sequence, starting from the first diffraction edge of the full diffraction path, the path portion from the first diffraction edge to the source location is rotated in a first rotational motion to obtain a straight line from the listener location to the second diffraction edge, or to the first intermediate post-rotation source location if the full diffraction path has only the first diffraction edge, the first intermediate post-rotation source location being the post-rotation source location when the full diffraction path has only the first diffraction edge. The sequence ends for a single diffraction edge. If there are two diffraction edges, the first edge would be edge 9 in FIG. 12 and the first intermediate location would be item 141.

[0054] In the case of two or more diffraction edges, the result of the first rotation operation is rotated around the second diffraction edge in a second rotation operation to obtain a third diffraction edge, or a straight line from the listener position to a second intermediate post-rotation source position when the full diffraction path has only the first and second diffraction edges, which is the rotated source position when the full diffraction path has only the first and second diffraction edges. The sequence ends for two diffraction edges. In the case of two diffraction edges, the first edge is edge 9 in FIG. 12 and the second edge is edge 13.

[0055] For a path having three or more diffraction edges, such as path 300 in FIG. 3, the procedure continues and one or more rotation operations are further performed using the third diffraction edge 11 in FIG. 3, then edge 12 in FIG. 3, then edge 13 in FIG. 3, generally until the full diffraction path has been processed and a straight line is obtained from the listener position to the rotated source position obtained at that time.

[0056] Subsequently, a preferred embodiment for processing a dynamic object (DO) is shown. For this purpose, reference is made to FIG. 2b, which shows a dynamic object DO located in a central position of the audio scene from one moment to another for the situation of FIG. 1. This means that the intermediate diffraction path from edge 1 to edge 5 shown in the top row of FIG. 2d is interrupted by the diffractive object DO, and two new diffraction paths are generated, namely one from edge 1 to edge 7 and then to edge 5, and the other from edge 1 to edge 3 and then to edge 5. These diffraction paths are appropriate when the listener is located on the left side of FIG. 2a. Due to the dynamic object being located in the sound scene, the MAAS and MAAL conditions are also changed with respect to the situation without a dynamic object. The intermediate diffraction path list of FIG. 1 is directly augmented by two additional intermediate diffraction paths shown in the bottom row of FIG. 2b, as shown for example in item 226 of FIG. 6. In particular, if we assume that the original path from edge 1 to edge 5 in Figure 5 is only part of a larger reflection situation in which the source is not located near edge 1, but there are, for example, one or more other reflection paths between the objects and a similar situation is with respect to the listener, the pre-computed diffraction path present in Figure 1 without the dynamic objects can be easily updated at run-time by simply replacing the diffraction paths 1-5 in Figure 1 with two additional paths, nevertheless by leaving intact the previous portion from edge 1 to any starting edge of the diffraction path, and also leaving intact the path portion from edge 5 to any output or final edge.

[0057] FIG. 6 shows the status of the procedure carried out with a dynamic object. In step 222, it is determined whether the dynamic object DO is changing its location, for example by translation and rotation. The edges attached to the dynamic object are updated. In the example of FIG. 2a, step 222 determines that there is a dynamic object with certain edges 70, 60, 20, 30, in contrast to the previous time shown in FIG. 1. In step 214, an intermediate diffraction path is found that includes edges 1 and 5. In step 224, it is determined that there is an interruption due to a dynamic object located in the path between edges 1 and 5. In step 224, an additional path is found, on the one hand from edge 1 to edge 5 above the dynamic object edge 30, and on the other hand from edge 1 to edge 5 above the dynamic object edge 60, as shown in the lower part of FIG. 2b. In step 226, the uninterrupted path is extended by two additional paths. This means that a path (not shown in the figure) going from any (not shown) input edge to edge 1 and going from edge 5 to any (not shown) output edge is modified by replacing the path portion between edge 1 and edge 5 with two other path portions shown in FIG. 2b. The first path portion from the input edge to edge 1 is stitched together with these two path portions to obtain two additional intermediate diffraction paths, and the output portion of the original path extending from edge 5 to the output edge is also stitched into two corresponding extended intermediate diffraction paths, thus two new extended previous diffraction paths have been generated from one previous intermediate diffraction path (without dynamic objects) due to the dynamic object. The other steps shown in FIG. 6 are performed similarly to those shown in FIG. 5.

[0058] Rendering (at run-time) diffraction effects with dynamic objects is one of the best ways to present an interactive impression for immersive media entertainment. The preferred strategy for considering diffraction with dynamic objects is as follows:

[0059] 1) In the precomputation step, A. Given dynamic objects / geometry, pre-compute possible (intermediate) diffraction paths around the given dynamic object. B. In the case of multiple dynamic objects / geometry, pre-compute possible (intermediate) diffraction paths around a single object based on the assumption that diffraction is not allowed between different dynamic objects. C. If there are dynamic / geometry and static objects / geometry, pre-compute possible paths around the dynamic or static objects based on the assumption that diffraction is not allowed between the static and dynamic objects.

[0060] 2) In the run-time step, A. Update the latent edges belonging to a repositioned dynamic mesh only if the dynamic mesh is repositioned (in translation and rotation). B. Find the visible edge list from the source and listener. C. Examine the path that starts at the source's edge list and ends at the listener's edge list. D. Check the visibility of intermediate edges pairwise to each other, and extend the path in the verified path by edges, triangles, and angles if there is an intrusion by an occluding object, which can be a dynamic or static object.

[0061] The extended algorithm for processing dynamic objects / geometry is shown in Figure 6, which has an additional step compared to the one for static scenes.

[0062] Considering that the preferred method pre-calculates (intermediate) diffraction path information that does not need to be reconsidered except in special cases, several practical advantages arise compared to the state of the art, which does not allow updating pre-calculated data. Furthermore, the flexible feature of combining multiple diffraction paths to generate an extended one allows both static and dynamic objects to be considered.

[0063] (1) Low computational complexity: The preferred method does not need to construct a full path from a given source location to a listener location at run-time. Instead, it only needs to find valid intermediate paths between the two points.

[0064] (2) Ability to render diffraction effects of a combination of static and dynamic objects or a combination of multiple dynamic objects: State-of-the-art techniques require updating the entire visibility graph between edges (static or dynamic) at runtime to simultaneously consider diffraction effects due to static and dynamic objects. A preferred method requires an efficient stitching process of two valid paths / path parts.

[0065] On the other hand, pre-computing the (intermediate) diffraction paths requires more time compared to state-of-the-art techniques. However, it is possible to control the size of the pre-computed path data by applying reasonable constraints such as the maximum allowed attenuation level in one full path, the maximum propagation distance, the maximum number of diffraction orders, etc.

[0066] 1) [Geometric-Acoustic-Based Approach] A preferred method has been invented to apply the UTD model to multiple visible / well-oriented edges based on pre-computed (intermediate) path information. This pre-computed data does not need to be monitored in real-time (most of the time) except in very rare cases such as interruptions by dynamic objects. Thus, the invention minimizes real-time computations.

[0067] 2) [Modularity] Every precomputed path functions as a module. A. In a static scene, in a real-time step, we only need to find valid modules between two spatial points. B. In dynamic scenes, the path via object (B) needs to be extended with the effective path via A, even if there is an interruption by a different object (A) in the effective path via B (imagine stitching two different images).

[0068] 3) [Supports fully dynamic interactions] Real-time rendering diffraction effects involving a combination of static and dynamic objects or a combination of multiple dynamic objects can be achieved.

[0069] It is noted herein that all alternatives or aspects described above, and all aspects defined by the independent claims in the following claims, may be used individually, i.e., without the alternatives or purposes other than those contemplated in the alternatives, purposes or independent claims. However, in other embodiments, two or more of the alternatives or aspects or independent claims may be combined with each other, and in other embodiments, all aspects or alternatives and all independent claims may be combined with each other.

[0070] The inventive coded signal may be stored on a digital or non-transitory storage medium or may be transmitted over a transmission medium, such as a wireless or wired transmission medium, such as the Internet.

[0071] Although some aspects are described in the context of an apparatus, it is understood that these aspects also represent a description of a corresponding method, with the blocks or devices corresponding to method steps or features of a method step. Similarly, aspects described in the context of a method step also represent a corresponding block or item or feature of a corresponding apparatus.

[0072] Depending on certain implementation requirements, embodiments of the invention can be implemented in hardware or in software. This can be done using a digital storage medium, such as a floppy disk, DVD, CD, ROM, PROM, EPROM, EEPROM or flash memory, on which electronically readable control signals are stored, which cooperates (or can cooperate) with a programmable computer system such that the method is performed.

[0073] Some embodiments according to the invention comprise a data carrier having electronically readable control signals, which can cooperate with a programmable computer system to cause one of the methods described herein to be performed.

[0074] In general, embodiments of the present invention can be implemented as a computer program product having program code which operates to perform one of the methods when the computer program product runs on a computer. The program code may for example be stored on a machine readable carrier.

[0075] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier or a non-transitory storage medium.

[0076] In other words, an embodiment of the inventive method is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0077] A further embodiment of the inventive method is therefore a data carrier (or a digital storage medium or a computer readable medium) comprising, recorded on the data carrier, the computer program for performing one of the methods described herein.

[0078] A further embodiment of the inventive method is therefore a data stream or a sequence of signals representing the computer program for performing one of the methods described herein, the data stream or sequence of signals being for example adapted to be transmitted over a data communications connection, for example via the Internet.

[0079] Another embodiment comprises a processing means, for example a computer, or a programmable logic device, configured to or adapted to perform one of the methods described herein. Another embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0080] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. In general, the methods are preferably performed by any hardware apparatus.

[0081] The above-described embodiments are merely illustrative of the principles of the present invention. It will be understood that modifications and variations of the arrangements and details described herein will be apparent to those skilled in the art. It is therefore intended to be limited only by the scope of the appended claims, and not by the specific details presented in the description and illustration of the embodiments herein.

[0082] References [1] L. Savioja and V. Vaelimaeki. Interpolated rectangular 3-D digital waveguide mesh algorithms with frequency warping. IEEE Trans. Speech Audio Process., 11(6) 783-790, 2003

[0083] [2] Mehra, R., Raghuvanshi, N., Antani, L., Chandak, A., Curtis, S., And Manocha, D. Wave-based sound propagation in large open scenes using an equivalent source formulation, ACM Trans. on Graphics 32(2) 19:1-19:13, 2013

[0084] [3] Mehra, R., Antani, L., Kim, S., and Manocha, D. Source and listener directivity for interactive wave-based sound propagation, IEEE Transactions on Visualization and Computer Graphics, 20(4) 495-503, 2014

[0085] [4] Nikunj Raghuvanshi and John M. Snyder, Parametric directional coding for precomputed sound propagation, ACM Trans. on Graphics 37(4) 108:1-108:14, 2018

[0086] [5] J. B. Allen, and D. A. Berkley, Image method for efficiently simulating small-room acoustics. The Journal of the Acoustical Society of America 65(4) 943-950, 1979

[0087] [6] M. Vorlaender, Simulation of the transient and steady-state sound propagation in rooms using a new combined raytracing / image-source algorithm, The Journal of the Acoustical Society of America 86(1) 172-178, 1989

[0088] [7] T. Funkhouser, I. Carlbom, G. Elko, G. Pingali, M. Sondhi, and J. West, A beam tracing approach to acoustic modeling for interactive virtual environments, In Proc. of ACM SIGGRAPH, 21-32, 1998

[0089] [8] M. Taylor, A. Chandak, L. Antani, and D. Manocha, Resound: interactive sound rendering for dynamic virtual environments, In Proc. of the seventeen ACM international conference on Multimedia, 271-280, 2009

[0090] [9] R. G. Kouyoumjian and P. H. Pathak, A uniform geometrical theory of diffraction for an edge in a perfectly conducting surface, In Proc. of the IEEE 62, 11, 1448-1461, 1974

[0091]

[10] U. P. Svensson, R. I. Fred, and J. Vanderkooy, An analytic secondary source model of edge diffraction impulse responses, Acoustical Society of America Journal 106 2331-2344, 1999

[0092]

[11] N. Tsingos, T. Funkhouser, A. Ngan, and I. Carlbom, Modeling acoustics in virtual environments using the uniform theory of diffraction, In Proc. of the SIGGRAPH, 545-552, 2001

[0093]

[12] M. Taylor, A. Chandak, Q. Mo, C. Lauterbach, C. Schissler, and D. Manocha, Guided multiview ray tracing for fast auralization. IEEE Transactions on Visualization and Computer Graphics 18, 1797-1810, 2012

[0094]

[13] H. Yeh, R. Mehra, Z. Ren, L. Antani, D. Manocha, and M. Lin, Wave-ray coupling for interactive sound propagation in large complex scenes, ACM Trans. Graph. 32, 6, 165:1-165:11, 2013

Claims

1. 1. An apparatus for rendering an audio scene (50) including an audio source and a plurality of diffractive objects at an audio source location, comprising: a diffraction path provider (100) for providing a plurality of intermediate diffraction paths (300, 400) through the plurality of diffractive objects, an intermediate diffraction path among the plurality of intermediate diffraction paths (300, 400) having a start point and an output edge of the plurality of diffractive objects, and associated filter information for the intermediate diffraction path; a renderer (200) for rendering said audio source to a listener location; The renderer (200) comprises: determining (216) one or more effective intermediate diffraction paths from the audio source position to the listener position based on the output edges of the intermediate diffraction paths and the listener position through the plurality of diffractive objects provided by the diffraction path provider (100) from the plurality of intermediate diffraction paths (300, 400); for each effective intermediate diffraction path of the one or more effective intermediate diffraction paths, determine (218) a filter representation of a full diffraction path from the audio source position to the listener position corresponding to an effective intermediate diffraction path of the one or more effective intermediate diffraction paths using a combination of the associated filter information of the effective intermediate diffraction path and filter information describing audio signal propagation from the output edge of the effective intermediate diffraction path to the listener position; and Calculating (220) an audio output signal for the audio scene (50) using the audio signals associated with the audio sources and the filtered representation of the full diffraction paths. The apparatus is configured so as to

2. the audio source position is fixed, and the diffraction path provider (100) is implemented as a pre-processor and configured to determine each effective intermediate diffraction path such that the starting point of each effective intermediate diffraction path corresponds to the audio source position; or the audio source position is variable, and the diffraction path provider (100) is implemented as a pre-processor and configured to determine an input edge of the plurality of diffractive objects as the starting point of an intermediate diffraction path; 2. The apparatus of claim 1, wherein the renderer (200) is configured to determine the one or more effective intermediate diffraction paths further based on the input edge(s) of the one or more intermediate diffraction paths and the audio source position, and to determine the filtered representation of the full diffraction path further based on another filter information describing an audio signal propagation from the audio source position to the input edge of the effective intermediate diffraction path associated with the full diffraction path.

3. 3. The apparatus of claim 1, wherein the renderer is configured to perform an occlusion test on a direct path from the audio source position to the listener position and to determine only the one or more effective intermediate diffraction paths when the occlusion test indicates that the direct path is obstructed.

4. 3. The apparatus of claim 2, The apparatus, wherein the renderer (200) is configured to determine (132) the filter representation of the full diffraction path by multiplying a frequency domain representation of the relevant filter information with a frequency domain representation of the filter information of the audio signal propagation from the output edge of the effective intermediate diffraction path to the listener position or a frequency domain representation of another filter information describing the audio signal propagation from the audio source position to the input edge of the effective intermediate diffraction path.

5. The renderer (200) determining (102) a starting group of potential input edges in response to said audio source position or determining (104) a final group of potential output edges in response to said listener position; searching (106) for one or more potential useful intermediate diffraction paths from a pre-stored list of intermediate diffraction paths using the starting group or the final group; Validating (108) the one or more potential effective intermediate diffraction paths using a source angle criterion (116) and a source angle between the audio source position and the corresponding input edge, or using a final angle criterion (118) and a listener angle between the listener position and the corresponding output edge.

5. An apparatus as claimed in claim 1 , configured to:

6. the renderer (200) is configured to calculate (112) the source angle, compare (116) the source angle with a maximum allowable angle (MAAS) of the audio source as the source angle criterion, and validate (122) a potential intermediate diffraction path to be the valid intermediate diffraction path when the source angle is smaller than the maximum allowable angle of the audio source; or 6. The apparatus of claim 5, wherein the renderer is configured to calculate the listener angle, compare the listener angle to a minimum acceptable angle for a listener (MAAL) as a listener angle criterion, and validate a potential intermediate diffraction path to become the valid intermediate diffraction path when the listener angle is greater than the minimum acceptable angle for the listener.

7. 7. An apparatus according to any one of claims 1 to 6, comprising: The diffraction path provider (100) is configured to access a memory having a pre-stored list containing entries for the plurality of intermediate diffraction paths (300, 400), each intermediate diffraction path entry including a series of edges, the series of edges extending from an input edge to an output edge, or each intermediate diffraction path entry including a series of triangles, the series of triangles extending from an input triangle to an output triangle, or each intermediate diffraction path entry including a series of items, the series of items starting from a source angle reference and further including one or more intermediate angles, and further including a listener angle reference.

8. 8. The apparatus of claim 7, the intermediate diffraction path entry of the pre-stored list includes the associated filter information or a reference to the associated filter information; or The apparatus, wherein the renderer (200) is configured to derive the associated filter information from data in the intermediate diffraction path entries of the pre-stored list.

9. 9. An apparatus according to any one of claims 1 to 8, comprising: An apparatus, wherein the plurality of diffractive objects of the audio scene (50) includes a dynamic object, and the diffraction path provider (100) is configured to provide at least one intermediate diffraction path around the dynamic object.

10. 9. An apparatus according to any one of claims 1 to 8, comprising:

4. An apparatus according to claim 1, wherein the plurality of diffractive objects of the audio scene (50) includes two or more dynamic diffractive objects, and the diffraction path provider (100) is configured to provide an intermediate diffraction path around a single dynamic diffractive object based on the assumption that diffraction is not allowed between two different dynamic objects.

11. 9. An apparatus according to any one of claims 1 to 8, comprising:

11. The apparatus of claim 10, wherein the plurality of diffractive objects of the audio scene (50) includes one or more dynamic objects and one or more static objects, and the diffraction path provider (100) is configured to provide intermediate diffraction paths around dynamic or static objects based on an assumption that diffraction is not allowed between static and dynamic objects.

12. 12. An apparatus according to any one of claims 1 to 11, comprising: the plurality of diffractive bodies includes at least one dynamic diffractive body; The renderer (200) determining (222) whether the at least one dynamic diffractive object has been repositioned by a repositioning at least with respect to translation and rotation, the at least one dynamic diffractive object representing a repositioned dynamic object; updating (222) edges attached to the repositioned dynamic object; In the determination of the one or more effective intermediate diffraction paths (216), potential effective intermediate diffraction paths are examined for visibility between internal edge pairs, and if the visibility between the internal edge pairs is interrupted due to the repositioning of the repositioned dynamic object, the potential effective intermediate diffraction paths are extended (226) with additional paths taken due to the repositioned diffracting object to obtain the effective intermediate diffraction paths. The apparatus is configured so as to

13. The apparatus of claim 1 , wherein the renderer (200) is configured to determine the relevant filter information by applying a Uniform Theory of Diffraction (UTD) or the renderer (200) is configured to determine the relevant filter information in a frequency-dependent manner.

14. the renderer (200) is configured to calculate, in response to the effective intermediate diffraction paths, that a rotated audio source position differs from the audio source position due to diffraction effects experienced by the effective intermediate diffraction paths, and to use the rotated position of the audio source in calculating (220) the audio output signal of the audio scene (50); or the renderer (200) is configured to calculate, in response to the full diffraction path, that a rotated audio source position differs from the audio source position due to diffraction effects experienced by the full diffraction path, and to use the rotated position of the audio source in calculating (220) the audio output signal of the audio scene (50); or 14. The apparatus of claim 1, wherein the renderer (200) is configured to calculate (220) the audio output signal of the audio scene (50) using the filter representation as well as an edge sequence associated with the full diffraction path and a diffraction angle sequence associated with the full diffraction path.

15. 15. The apparatus of claim 14, wherein the renderer (200) is configured to determine a distance from the listener position to a rotated audio source position and to use the distance when calculating (220) the audio output signal for the audio scene (50).

16. The apparatus of claim 14 or 15, wherein the renderer (200) is configured to select one or more directional filters depending on the rotated audio source position and a predetermined output format of the audio output signal, and to apply the one or more directional filters and the filter representations to the audio signal when calculating the audio output signal.

17. The apparatus of claim 14, 15 or 16, wherein the renderer (200) is configured to determine an attenuation value depending on a distance between the rotated audio source position and the listener position and apply it to the audio signal in addition to the filter representation or one or more directional filters depending on the audio source position or the rotated audio source position.

18. The renderer (200) is configured to determine the rotated audio source position in a series of rotation operations including at least one rotation operation; starting from a first diffraction edge of the full diffraction path, a path portion from the first diffraction edge to an audio source position is rotated in a first rotational motion to obtain a straight line from a second diffraction edge or from the listener position if the full diffraction path only has the first diffraction edge to a first intermediate rotated audio source position, the first intermediate rotated audio source position being the rotated audio source position when the full diffraction path only has the first diffraction edge; or the result of the first rotation operation is rotated around the second diffraction edge in a second rotation operation to obtain a straight line from a third diffraction edge or from the listener position to a second intermediate rotated audio source position when the full diffraction path has only the first diffraction edge and the second diffraction edge, the second intermediate rotated audio source position being the rotated audio source position when the full diffraction path has only the first diffraction edge and the second diffraction edge; 18. The apparatus of claim 14, wherein the full diffraction path is processed and one or more further rotation operations are performed until a straight line from the listener position to a final rotated audio source position is obtained, and the final rotated audio source position is achieved after the one or more rotation operations.

19. 1. A method for rendering an audio scene (50) including an audio source and a number of diffractive objects at an audio source location, comprising: providing a plurality of intermediate diffraction paths (300, 400) through the plurality of diffractive objects, an intermediate diffraction path among the plurality of intermediate diffraction paths (300, 400) having a start point and an output edge of the plurality of diffractive objects, and associated filter information for the intermediate diffraction path; rendering the audio source to a listener position, the rendering step including determining (216) one or more effective intermediate diffraction paths from the audio source position to the listener position based on the output edges of the intermediate diffraction paths and the listener position through the plurality of diffractive objects provided by the provision of the plurality of intermediate diffraction paths (300, 400); for each effective intermediate diffraction path of the one or more effective intermediate diffraction paths, determining (218) a filter representation of a full diffraction path from the audio source position to the listener position corresponding to an effective intermediate diffraction path of the one or more effective intermediate diffraction paths using a combination of the associated filter information of the effective intermediate diffraction path and filter information describing an audio signal propagation from the output edge of the effective intermediate diffraction path to the listener position; calculating (220) an audio output signal for the audio scene (50) using the audio signals associated with the audio sources and the filtered representation of the full diffraction path; A method comprising:

20. 20. A computer program for carrying out the method according to claim 19 when running on a computer or processor.

Citation Information

Patent Citations

  • Video game processing device and video game processing program

    JP2012054698A

  • Methods, systems, and computer readable media for fast geometric sound propagation using visibility computations

    US20120249556A1