Method, apparatus, and system for early reflectance estimation based on geometric representations of voxels

Abstract

Methods, apparatus, programs, and storage media for improving the estimation of early reflection trajectories of audio sources in a three-dimensional audio scene are described. The method includes obtaining a voxel-based representation of the audio scene, information about a listener location in the audio scene, and information about an audio source location in the audio scene. A ray pattern is applied to one or more points on a connection line between the audio source position and the listener position to obtain a plurality of rays starting at the respective point for each of the points. A set of collision voxels is determined based on the ray and a voxel-based representation of the audio scene. An early reflection trajectory is determined based on the set of collision voxels, the listener location, the audio source location, and a geometric effectiveness test.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 344,895 filed on May 23, 2022 and U.S. Provisional Patent Application No. 63 / 387,339 filed on December 14, 2022, both of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to modeling of audio sources, and more particularly to voxel-based early sound source reflection estimation methods and apparatus. Background Art

[0004] Sound reflections from acoustically reflective surfaces can affect the perceived sound of an audio source. Sounds that are reflected and received shortly after the direct sound at a target location (e.g., the listener location), which will be referred to herein as early reflections (ER), are of particular interest when modeling a sound source because the perceived sound of an audio source can be accurately modeled by considering only the direct sound and the ER. On the other hand, higher-order acoustic reflections are often less important because they are lower in energy and are temporally / spatially psychoacoustically masked by the ER and other components.

[0005] ERs cause several perceptual effects, such as apparent source width, perceived distance, timbre, and spaciousness. ERs are relatively sparse in time and span a relatively short period of time, typically contained within the first ~80 ms of the room impulse response (see Figure 1 ). Figure 1 The diagram shows the echogram of a room, including the echograms of the direct sound source, early reflections, and late reflections. Figure 1 It also allows visualization of the differences between direct sound, early reflections and late reflections.

[0006] The psychoacoustic relevance of ER depends strongly on several factors, such as direction, level, time delay and spectral content of the audio signal.

[0007] The direction of the ER specifically affects the time delay and frequency response at the listener's ear. Therefore, the direction of the ER plays an important role in the perceived reflected sound. When the direction of arrival changes, it means that the path from the source to the listener's ear has changed due to movement, obstacles, etc. The change in path length affects the time delay and, due to the shape of the pinna, will produce a different frequency response depending on the direction of arrival at the ear.

[0008] To estimate the trajectory of the ER, the Image-Source (IS) method aims to find a purely specular path between the audio source and the receiver (i.e., listener). This process is simplified by assuming that sound propagates only along straight lines (i.e., rays). The audio image source is propagated on a line perpendicular to the boundary and at the same distance from it as the original source 101 (see Figure 2 ). Figure 2 A sound source 101, a listener 102, boundaries and an image source are illustrated.

[0009] Since the sound reflects from the boundary surface at the same angle as the angle of incidence, the impression is created that the original source 101 is mirrored at the boundary surface. Reflection from a single boundary then represents a (first order) ER.

[0010] However, sometimes the boundaries are unknown or poorly defined. An example is a voxel-based representation of a 3D environment for sound rendering in VR applications. A voxel is a volume of space with certain acoustic properties (e.g., reflectivity). In order to find the boundaries for IS methods, multiple groups of voxels should be considered, because a single voxel has no orientation information if the reflective surface orientation is not explicitly assigned to its properties. Therefore, complex trigonometric considerations are needed to estimate the boundaries. Figure 3 An exemplary scene is depicted in FIG. In this figure, gray voxels represent reflecting objects, and gray voxels next to white voxels represent reflecting boundaries of the surface of the object. Without reflection direction information, a single voxel is not sufficient to determine the reflection trajectory of the sound emitted by source 101.

[0011] Therefore, there is a need for improved, efficient methods for ER estimation in voxel-based environments, especially when audio reflection boundary orientation information is not available in advance. Summary of the invention

[0012] In view of the above, the present disclosure provides a method, an apparatus and a program and a computer-readable storage medium for early sound source reflection estimation in a voxel-based 3D environment (3D voxel grid), which have the features of the corresponding independent claims.

[0013] According to one aspect of the present disclosure, a method for estimating early reflections is provided. A voxel-based representation of a three-dimensional audio scene, information about a listener position of a listener in the three-dimensional audio scene, and information about an audio source position of an audio source in the three-dimensional audio scene may be obtained (e.g., received or determined). A ray pattern may be applied to one or more points on a connecting line between an audio source position and a listener position to obtain multiple rays starting from the corresponding point for each of the one or more points. A set of collision voxels may be determined based on the multiple rays and the voxel-based representation of the three-dimensional audio scene. An early reflection trajectory may be determined based on a set of collision voxels, a listener position, an audio source position, and a geometric validity test. For example, for each collision voxel in a set of collision voxels, a path connecting the listener position and the audio source position via the corresponding collision voxel may be determined. Then, for each path, if the path is geometrically valid, the path may be determined as an early reflection trajectory.

[0014] By adopting the heuristic method specified above, early reflections can be efficiently estimated in a voxel-based environment without requiring any reflecting surface orientation information of the voxels. As a result, sound sources can be modeled with high accuracy and low computational complexity, enabling accurate and efficient sound representation in real-time applications (e.g., VR games).

[0015] In some embodiments, the method may further include determining a ray pattern. Determining the ray pattern may include selecting a ray pattern from a plurality (a set) of predefined ray patterns or calculating a ray pattern. Alternatively, the ray pattern may be fixed. Also alternatively, an indication of a ray pattern to be used may be received using a bitstream.

[0016] In some embodiments, the method may further include determining one or more points based on a number (e.g., count, cardinality) of the one or more points. That is, a number of one or more points may be obtained or determined (e.g., set to N points), and the resulting (e.g., N) number (count or cardinality) of one or more points may correspond to the coordinates of the one or more points (e.g., in the sense that there is a corresponding coordinate for each of the one or more points).

[0017] In some embodiments, the ray direction map may be defined as (e.g., may include) a predefined number of rays and a predefined direction of the rays from the origin. The predefined number of rays may be, for example, 6, 8, or 12. The direction of the ray may be defined by a grid index of the voxel grid.

[0018] In some embodiments, the predefined direction of the ray may include one or more of the following: horizontal and vertical directions to adjacent grid indices; and diagonal directions to adjacent grid indices. Thus, the predefined direction may define a relative direction from the origin of the ray (i.e., the grid index (l, m, i) in the voxel grid). The relative direction may be expressed as:

[0019] (+1,0,0), (-1,0,0), (0,+1,0), (0,-1,0), (0,0,+1), (-0,0,-1);

[0020] (+1,+1,0), (+1,-1,0), (-1,+1,0), (-1,-1,0), (+1,0,+1), (+1,0,-1), (-1,0,+1), (-1,0,-1), (0,+1,+1), (0,+1,-1), (0,-1,+1), (0,-1,-1); and

[0021] (+1,+1,+1), (+1,+1,-1), (+1,-1,+1), (+1,-1,-1), (-1,+1,+1), (-1,+1,-1), (-1,-1,+1), (-1,-1,-1).

[0022] In some embodiments, determining the ray pattern may be based on a scene type of the three-dimensional audio scene, available computing resources, encoder presets, or a combination thereof.

[0023] In some embodiments, coordinates of one or more points on a line connecting the audio source location and the listener location may be determined based on a quantity (eg, count, cardinality) of the one or more points.

[0024] In some embodiments, one or more points may be determined to divide the line connecting the audio source position and the listener position into N-1 equal line segments, where N is the number (e.g., count, cardinality) of the one or more points. N may be greater than or equal to 2, for example.

[0025] In some embodiments, the number of the one or more points may depend on the scene type of the three-dimensional audio scene, available computing resources, encoder presets, or a combination thereof.

[0026] In some embodiments, scene types may include indoor scenes and outdoor scenes.

[0027] In some embodiments, each collision voxel may be an occluding voxel in the voxel-based representation of the three-dimensional audio scene.

[0028] In some embodiments, occluding voxels may represent acoustically reflective surfaces.

[0029] In some embodiments, occluding voxels may represent any material other than air in the voxel-based representation of the three-dimensional audio scene. That is, occluding voxels may represent reflective surfaces and non-occluding voxels may represent non-reflective surfaces (or no surface defined at all).

[0030] In some embodiments, determining a set of collision voxels based on a plurality of rays and a voxel-based representation of a three-dimensional audio scene may include determining one or more intersections (e.g., intersection points) between each ray in the plurality of rays and an occluding object voxel. The method may also include, for each ray, determining an occluding object voxel containing an intersection closest to the origin of the corresponding ray as a collision voxel in the set of collision voxels. That is, the collision voxel may be an occluding object voxel that the corresponding ray hits first.

[0031] In some embodiments, determining the early reflection trajectory based on a set of collision voxels, a listener position, an audio source position, and a geometric validity test may include determining, for each collision voxel in the set of collision voxels, whether the collision voxel can produce a geometrically valid representation of a first-order reflection. If it is determined that the collision voxel can produce a geometrically valid representation of a first-order reflection, then a path connecting the listener position and the audio source position via the corresponding collision voxel may be determined as the early reflection trajectory.

[0032] In some embodiments, determining whether a collision voxel can produce a geometrically valid representation of a first-order reflection may include determining a preceding voxel of the collision voxel. The preceding voxel may be a voxel that contains an intersection with a corresponding ray and is before the collision voxel in the direction of the corresponding ray. A second path connecting the listener position and the audio source position via the corresponding preceding voxel may be determined. If the second path does not contain an intersection with an occluding object voxel, then the collision voxel can produce a geometrically valid representation of a first-order reflection. Generally, if the path connecting the listener position and the preceding voxel and the path connecting the audio source position and the preceding voxel do not contain an intersection with an occluding object voxel, then the collision voxel can produce a geometrically valid representation of a first-order reflection. In other words, if the path connecting the listener position and the preceding voxel and the path connecting the audio source position and the preceding voxel both pass a line of sight check ("visibility check"), then the collision voxel can produce a geometrically valid representation of a first-order reflection.

[0033] Thereby, collision voxels that cannot lead to a geometrically valid path from the audio source position to the listener position can be efficiently sorted out.

[0034] Alternatively or additionally, determining the early reflection trajectory based on a set of collision voxels, the listener position, the audio source position, and a geometric validity test can include determining, for each collision voxel in the set of collision voxels, a path connecting the listener position and the audio source position via the corresponding collision voxel. For each path, if the path is geometrically valid, then the path can be determined as an early reflection trajectory. If the path passes a line of sight check ("visibility check"), that is, if both the path connecting the listener position and the collision voxel and the path connecting the collision voxel and the audio source position pass the line of sight check, then the path is said to be geometrically valid.

[0035] In some embodiments, the path may include a straight line connecting the audio source position to a collision voxel in the set of collision voxels and a straight line connecting the same collision voxel in the set of collision voxels to the listener position.

[0036] In some embodiments, a path may be determined to be geometrically valid if it contains no intersections with occluder voxels other than the corresponding path's collision voxel. That is, paths with intersections with more than one occluder voxel may be discarded. In other words, a path may be determined to be geometrically valid if it is not blocked by any occluder voxels other than the collision voxel.

[0037] In the case of performing a test on collision voxels that can produce a geometrically valid representation of a first-order reflection and a test on a geometrically valid path, the collision voxels that cannot produce a geometrically valid representation of a first-order reflection can be sorted out by first determining whether there is an intersection between the occluding voxel and the path connecting the audio source position, the preceding voxel, and the listener position. For the remaining collision voxels, the path connecting the audio source position, the collision voxel, and the listener position can be determined. Finally, it can be determined whether there is an intersection between these paths and the occluding voxels other than the collision voxels.

[0038] By combining the two geometric validity tests, only geometrically valid early reflection trajectories can be determined, regardless of the geometry of the three-dimensional audio scene.

[0039] In some embodiments, the method may further comprise selecting a set of acoustically most relevant early reflection trajectories from the early reflection trajectories.

[0040] In some embodiments, selecting a set of acoustically most relevant early reflection trajectories may be based on the lengths of the early reflection trajectories and / or the reflection coefficients of the collision voxels of the corresponding early reflection trajectories. In particular, for example, the acoustically relevant early reflection trajectories may have short lengths and / or large reflection coefficients compared to non-acoustically relevant early reflection trajectories.

[0041] In some embodiments, the reflection coefficient may depend on the material modeled (or otherwise indicated) by the collision voxel.

[0042] In some embodiments, selecting a set of acoustically most relevant early reflection trajectories may include discarding early reflection trajectories having values ​​indicating an internal angle at the collision voxel close to 180°. Here, close to 180° may mean 180°-ε, where ε is a small angle. In some implementations, for example, early reflection trajectories having values ​​indicating an internal angle greater than 160° may be discarded.

[0043] In some embodiments, the value indicating an interior angle close to 180° may be the interior angle or the length of the early reflection trace.

[0044] In some embodiments, the method may further include outputting the early reflection trace. That is, for example, the early reflection trace or the acoustically most relevant early reflection trace may be output for rendering or further processing, such as occlusion, diffraction, 3D range or reverberation processing prior to rendering.

[0045] In some embodiments, the method may also include rendering a three-dimensional audio scene, for example, via a virtual reality VR, augmented reality AR, mixed reality MR, and / or extended reality XR device.

[0046] In some embodiments, the early reflection trajectory may represent a first-order trajectory. In some embodiments, the first-order trajectory may be a reflection trajectory having a single reflection between the audio source position and the listener position.

[0047] According to another aspect of the present disclosure, a method for processing a frame (e.g., a time frame) of a three-dimensional audio scene is provided. A reflection trajectory of the frame can be estimated based on the method according to the previous aspect. The estimated early reflection trajectory can be stored (e.g., stored locally or submitted to a shared storage device or a cloud storage device). Alternatively, the estimated early reflection trajectory of the previous frame can be accessed (e.g., from a local storage device, a shared storage device, or a cloud storage device). The estimated early reflection trajectory of the previous frame can be calculated based on the method according to the previous aspect. The estimated early reflection trajectory of the previous frame can be accessed only when the voxels containing the listener position, the voxels containing the audio source position, and the geometry of the voxel-based representation of the three-dimensional audio scene have not changed between the frame and the previous frame.

[0048] By using a prior estimate of the early reflection trajectory when the three-dimensional audio scene is static, the complexity of processing the audio data of the three-dimensional audio scene can be reduced without any impact on the accuracy of the output.

[0049] According to another aspect of the present disclosure, a method for creating trajectories for geometrically connected audio sources for audio processing that is efficiently implemented on a voxel 3D grid is provided. Information related to a ray map 'R' can be received. A first set of points 'P' to which ray casting is to be applied based on the ray map 'R' can be determined. A second set of ray-voxel 'collision' voxels 'C' based on the first set of points and reflecting voxels 'VOX' can be determined. A third set of valid reflection trajectories 'SC-L' based on the second set of ray-voxel 'collision' voxels 'C' can be determined. From the third set of valid reflection trajectories, a subset of the most acoustically relevant valid reflection trajectories can be selected and output.

[0050] Aspects of the present disclosure may be implemented via an apparatus. The apparatus may include a processor and a memory coupled to the processor. The processor may be adapted to execute methods according to aspects and embodiments of the present disclosure.

[0051] Aspects of the present disclosure may be implemented via a program. When the instructions of the program are executed by a processor, the processor may execute aspects and embodiments of the present disclosure. A computer-readable storage medium may store a program. Such a computer-readable storage medium may include a memory device, such as those described herein, including but not limited to a random access memory (RAM) device, a read-only memory (ROM) device, etc. Thus, some innovative aspects of the subject matter described in the present disclosure may be implemented via one or more computer-readable storage media having software stored thereon.

[0052] It will be appreciated that device features and method steps can be interchanged in many ways. In particular, as will be appreciated by those skilled in the art, details of the disclosed method can be implemented by a corresponding device (or system), and vice versa. Moreover, any statement made above about a method is understood to apply equally to a corresponding device (or system), and vice versa. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Exemplary embodiments of the present disclosure are explained below with reference to the accompanying drawings, in which

[0054] Figure 1 is a diagram showing an example of an echogram of a room,

[0055] Figure 2 An example of determining ER using the IS method is schematically illustrated.

[0056] Figure 3 Schematic illustration of a voxel grid, audio sources, and associated reflection boundaries,

[0057] Figure 4 The voxel grid, the audio source, the listener and the single collision voxel considered for reflection trajectory estimation are schematically illustrated.

[0058] FIG. 5A to FIG. 5D schematically illustrates an example of a ray pattern according to an embodiment of the present disclosure,

[0059] FIG. 6A to FIG. 6B Schematically illustrates an example of determining whether a collision voxel can generate a geometrically valid representation of a first-order reflection according to an embodiment of the present disclosure,

[0060] Figure 7 Schematically illustrates an example 2D audio scene with occluding voxels (points), non-occluding voxels (plain colors), audio sources, and listeners according to an embodiment of the present disclosure,

[0061] Figure 8 Schematically illustrates an embodiment of the present disclosure Figure 7 An example 2D audio scene with a ray map applied to the audio source position and the collision voxels hit by the ray.

[0062] Fig. 9 Schematically illustrates an embodiment of the present disclosure Figure 8 An example 2D audio scene with lines connecting the audio source to the listener via collision voxels.

[0063] Fig.10 Schematically illustrates an embodiment of the present disclosure Fig. 9 An example 2D audio scene and a geometrically valid ER track,

[0064] Fig.11 Schematically illustrates an embodiment of the present disclosure Figure 7 An example 2D audio scene with all geometrically valid ER tracks,

[0065] FIG. 1 schematically illustrates a configuration of a wireless receiver with different listener positions according to an embodiment of the present disclosure. Figure 7 An example 2D audio scene and all associated geometrically valid ER tracks,

[0066] Fig.13 schematically illustrates an enlarged view of an example 2D audio scene with an inner angle close to 180° having a geometrically valid ER trajectory according to an embodiment of the present disclosure,

[0067] Fig.14 is a flowchart illustrating an example of a method of estimating ER trajectories in a voxel-based audio scene representation according to an embodiment of the present disclosure,

[0068] Fig.15 is a flowchart illustrating an example of a method of determining a set of collision voxels based on a plurality of rays and a voxel-based representation of an audio scene according to an embodiment of the present disclosure,

[0069] Fig.16is a flowchart illustrating an example of a method for determining early reflection trajectories based on a set of collision voxels, a listener position, an audio source position, and a geometric validity test according to an embodiment of the present disclosure,

[0070] Fig.17 is a flowchart illustrating another example of a method for determining early reflection trajectories based on a set of collision voxels, a listener position, an audio source position, and a geometric validity test according to an embodiment of the present disclosure, and

[0071] Fig.18 An example of an apparatus for ER trajectory estimation according to an embodiment of the present disclosure is schematically illustrated. DETAILED DESCRIPTION

[0072] The drawings and the following description relate to the preferred embodiments by way of illustration only. It should be noted that from the following discussion, alternative embodiments of the structures and methods disclosed herein will be readily identified as viable alternatives that can be employed without departing from the principles of the claimed content.

[0073] Reference will now be made in detail to several embodiments, examples of which are illustrated in the accompanying drawings. It is noted that, where feasible, similar or identical reference numerals may be used in the accompanying drawings and may indicate similar or identical functions. The accompanying drawings depict embodiments of the disclosed systems (or methods) for illustrative purposes only. Those skilled in the art will readily appreciate from the following description that alternative embodiments of the structures and methods described herein may be employed without departing from the principles described herein.

[0074] The Moving Picture Experts Group (MPEG) is a consortium of working groups jointly established by the International Organization for Standardization (ISO) and the International Electrotechnical Commission (IEC) that sets standards for media coding, including audio coding. MPEG is arranged under ISO / IECSC 29, and the audio group is currently identified as Working Group (WG) 6. WG6 is currently working on the MPEG-I Audio standard.

[0075] The new MPEG-I standard enables acoustic experiences from different viewpoints and / or perspectives or listening positions by supporting scenes and various movements around such scenes, such as movement using various degrees of freedom in virtual reality (VR), augmented reality (AR), mixed reality (MR) and / or extended reality (XR) applications.

[0076] For audio rendering in VR, AR, MR, and XR applications, object-based approaches have been widely adopted by representing complex auditory scenes as multiple individual audio objects, each of which is associated with parameters or metadata that define the location / position and trajectory of the object in the scene. Alternatively, audio rendering in such environments can also use high-order ambisonics (HOA).

[0077] Voxels for audio rendering are relevant for media environments implemented in both hardware and software, such as video games and / or VR, AR, MR, and XR environments. The following describes and defines some concepts related to voxels for audio rendering:

[0078] What are voxels used for audio rendering?

[0079] A voxel is a volume of space that has acoustic properties or audio rendering instructions assigned to it.

[0080] What is the voxel size for audio rendering?

[0081] The voxel size is an encoder configuration parameter and it can be chosen (manually or automatically) according to the scene geometry level of detail (eg, in the range of 10cm-1m).

[0082] How can voxels be obtained?

[0083] Voxels used for audio rendering can be obtained by:

[0084] voxelization (or conversion) of a mesh-based scene representation, and / or

[0085] From a scene representation used for scene generation (or even video rendering) (e.g., by downsampling to smaller sized voxels).

[0086] How to represent a voxel-based audio scene?

[0087] Any voxel-based representation of an audio scene may contain an indication of voxels that are not transmissive voxels (e.g., these voxels are occluding voxels), i.e., voxels in which sound cannot propagate or cannot propagate freely - a representation of the occluding geometry. This indication may be associated with an indication of the coordinates of the corresponding voxels (e.g., center coordinates, corner coordinates). The coordinates of these voxels may be represented, for example, by a grid index. In addition, the voxel-based representation may include an indication of material properties (such as absorption coefficients, reflection coefficients, etc.) of voxels that are not transmissive voxels. In addition to occluding voxels, the voxel-based representation may also indicate transmissive voxels (e.g., air voxels), i.e., voxels in which sound can propagate, a representation of the sound propagation medium. Thus, some implementations of a voxel-based representation of an audio scene may include, for each voxel in a predefined segment of space (e.g., within a boundary surrounding the audio scene), an indication of the corresponding material property.

[0088] However, existing traditional approaches using voxels to provide realistic sound for user experiences in VR, AR, MR, and XR environments (including those involving movement) are challenging and computationally complex. Estimating ER trajectories is of particular interest for these use cases. For VR use cases, the psychoacoustic relevance of ER directions is higher than the ER audio signal level because VR users can see them better than estimating the reflective properties of reflective surfaces (because, for example, the reflective properties should give an estimate of the reflected energy or audio signal level).

[0089] In conventional methods for estimating ER trajectories (e.g., IS methods), the boundaries for determining reflections must be known (and clearly defined). Since a single voxel cannot provide sufficient information about the orientation of the boundaries of the reflecting surface, the method of the present invention relies on heuristic methods to estimate the ER trajectory in order to generate a perceived first-order ER sound effect with sufficient accuracy or fidelity.

[0090] The heuristic method according to the present disclosure is based on finding geometrically valid reflection trajectories between audio sources and listeners that are sufficient to generate a perceived first-order ER sound effect by performing several low-complexity steps based on their locations and a voxel-based geometric representation.

[0091] Figure 4 The general concept of the heuristic method is depicted. The position of the audio source 101 and the position of the listener 102 are known. Then, the effective reflection trajectory from the audio source 101 to the listener 102 through the collision voxel 104 should be estimated without considering the information of the reflection surface based on multiple voxels, that is, the heuristic method works voxel by voxel and acts on the grid index representing the voxel position.

[0092] Now it will Figures 7 to 11The heuristic method is explained in detail with reference to a specific audio scene example depicted in . However, the present disclosure should not be construed as being limited to this specific example. Moreover, although the examples relate to 2D situations or only show 2D projections, it is understood that the method according to the present disclosure is generally applicable to 3D audio scenes. Figure 7 An example 2D audio scene is depicted with occluded voxels (105 dots), non-occluded voxels (plain colors), an audio source 101, and a listener 102. The positions of the audio source 101 and the listener 102 are marked with S and L, respectively. This example is in 2D and is only for illustration purposes. It is straightforward to extend the algorithm to a 3D environment.

[0093] The voxel-based representation of the audio scene, the information about the listener position, and the audio source 101 position are inputs to the ER trajectory estimation method. In other words, the voxel-based representation of the audio scene, the information about the listener position, and the audio source position may be received. Alternatively, the voxel-based representation of the audio scene, the information about the listener position, and the audio source position may be determined in the ER trajectory estimation method.

[0094] To find the reflection trajectory between the audio source 101 and the listener 102, the ray pattern is applied to the point 103 in the audio scene. Figure 7 In the example of , five equidistant points on the line connecting the audio source 101 and the listener 102 are depicted. However, the depicted example should not be interpreted as limiting the location and number of points. For the method, a different number of points and different positions of these points can be used. The ray pattern can be determined in advance. In addition, the ray pattern can define a predefined number of rays and predefined (corresponding) directions of the rays from the origin. The example ray pattern is in FIG. 5A to FIG. 5D Thus, the predefined number of rays may be, for example, 6 ( Figure 5A ))、8( Figure 5C )) or 12( Figure 5B )) or a combination thereof ( Figure 5D)), and the predefined direction of the ray may include a direction from a ray origin (l,m,i) in a voxel grid. The ray origin in the audio scene may be point 103. The direction may be any combination of the following directions relative to the ray origin: horizontal and vertical directions to adjacent grid indices; and diagonal directions to adjacent grid indices. Relative directions may be expressed as: (+1,0,0), (-1,0,0), (0,+1,0), (0,-1,0), (0,0,+1), (-0,0,-1); (+1,+1,0), (+1,-1,0), (-1,+1,0), (-1,-1,0), (+1,0,+1), (+1,0,-1), (-1,0,+1), (- and (+1,+1,+1), (+1,+1,-1), (+1,-1,+1), (+1,-1,-1), (-1,+1,+1), (-1,+1,-1), (-1,-1,+1), (-1,-1,-1,+1).

[0095] Determining the ray pattern may be understood as selecting one of the predefined ray patterns. Determining the ray pattern may be based on the scene type of the audio scene, available computing resources, encoder presets, or a combination thereof. The scene type may include, for example, indoor scenes and outdoor scenes.

[0096] In the next step, the coordinates of the points need to be defined (e.g., determined or calculated) to apply the ray pattern to the corresponding points 103. The number of points (e.g., count, cardinality) can be determined. The number of one or more points can depend on the scene type of the audio scene, the available computing resources, the encoder preset, or a combination thereof. The scene type can include, for example, indoor scenes and outdoor scenes. Alternatively, the number of points can be fixed. In some implementations, the number of points can alternatively or additionally depend on the selected ray pattern.

[0097] In addition, it has been found that positioning the point on the line between the audio source 101 and the listener 102 improves the quality and efficiency of the ER trajectory estimation. To this end, the point 103 (the position of the point 103) can be determined based on the number of one or more points. In addition, the position of the point can be determined based on the line connecting the audio source position and the listener position (e.g., arranged on the line). More specifically, the one or more points 103 can be determined so that the line connecting the audio source position and the listener position is divided into, for example, N-1 equal line segments. Here, N is the number of one or more points, and in this case can be greater than or equal to 2.

[0098] It is worth noting that for the number of points to be selected as 1, a single point may correspond to the audio source position. For the number of points to be selected as 2, two points may correspond to the audio source position and the listener position, respectively.

[0099] Figure 8 Depicted is an example of a ray pattern having 8 rays being applied to a point 103 located at the position of an audio source 101. The rays are depicted as dashed lines.

[0100] In the next step, a set of collision voxels is determined based on the multiple rays and the voxel-based representation of the audio scene. In particular, a set of collision voxels can be determined by searching for intersections between the rays and any occluding voxels 105 (point-like) in the audio scene. The occluding voxels 105 can represent acoustically reflective surfaces. In other words, the occluding voxels 105 can represent any material in the voxel-based representation of the audio scene except air or other representations of the sound propagation medium. Then, for each ray, the occluding voxel 105 containing the intersection closest to the origin of the corresponding ray can be determined as the collision voxel 104. In other words, the collision voxel 104 can be defined as the first occluding voxel hit by the corresponding ray. This step ensures that only occluding voxels that can represent reflective surfaces are selected. Figure 8 In FIG. 1 , all collision voxels 104 of the ray starting at the audio source position are marked with dots at the end of the ray. Note that in this example, the lower right ray does not intersect any occluding voxels, so Figure 8 It is not depicted and is not considered further in the algorithm.

[0101] In the next step, it may be determined whether the collision voxel 104 can produce a geometrically valid representation of the first-order reflection. Fig. 6A , a scene is depicted in which a collision voxel can produce a geometrically valid representation of a first-order reflection. For a collision voxel 104, a preceding voxel 107 can be determined. The preceding voxel 107 can be a voxel that is before the collision voxel 104 in the direction of the corresponding ray. For the preceding voxel 107, a path connecting the audio source position and the listener position via the preceding voxel 107 can be determined. If the path does not contain an intersection with an occluding voxel (i.e., by a line of sight or visibility check), then the collision voxel 104 is determined to be a collision voxel that can produce a geometrically valid representation of a first-order reflection. In Fig. 6A In the case of , the path contains no intersections with occluding voxels. Figure 6B A scene of a collision voxel 104 that does not produce a geometrically valid representation of a first order reflection is depicted in FIG. In this example, a line connecting the front voxel and the listener position intersects an occluding voxel to the right of the collision voxel 104. Collision voxels 104 that do not produce a geometrically valid representation of a first order reflection can be discarded.

[0102] In the next step, for each collision voxel 104 determined for the ray and the starting point 103, a path may be determined to connect the audio source 101 and the listener 102 via the corresponding collision voxel 104. The path may include a straight line connecting the audio source position to the collision voxel 104 and a straight line connecting the same collision voxel 104 to the listener position. Alternatively, the path may include a straight line connecting the audio source position to the preceding voxel 107 and a straight line connecting the preceding voxel 107 to the listener position, or a path derived from the two possible paths mentioned above.

[0103] Fig. 9 Shows Figure 8 , where there is a determined path between the audio source 101 and the listener 102.

[0104] In a final step (which may be optional), it is determined whether the path from the audio source 101 to the listener 102 is geometrically valid. This step may be combined with the previous selection of the collision voxels 104 that can produce a geometrically valid representation of a first-order reflection. Then, in the following geometric validity test, only the paths associated with the collision voxels that can produce a geometrically valid representation of a first-order reflection may be considered. Alternatively, the following geometric validity test may consider the paths associated with all collision voxels 104. Since the path determined in the previous step may be defined only by straight lines between the audio source 101, the collision voxels 104, and the listener 102, the line may pass through (e.g., intersect or rub) occlusion voxels other than the collision voxel 104. In reality, such a reflection trajectory would not be possible in the sense that it would not allow the propagation of sound. Therefore, a path including a line passing through an occlusion voxel other than the corresponding collision voxel 104 may be determined to be geometrically invalid. To find intersections with occluding voxels other than the corresponding collision voxels 104, a line-mesh intersection algorithm may be applied to the line connecting the audio source 101, the collision voxel 104, and the listener 102. As an example, a fast traversal algorithm for ray tracing may be used (see Amanatides, J. and A. Woo, A Fast Voxel Traversal Algorithm for Ray Tracing. Proceedings of Euro Graphics, 1987. 87.).

[0105] Fig.10 The determined geometrically valid paths are depicted as solid lines. It is noteworthy that in this example, only one of the 7 paths previously found was determined to be geometrically valid.

[0106] As stated previously, this process is repeated for each point 103 on the line between the audio source 101 and the listener 102 .

[0107] Fig.11 The final result of the algorithm for the example of N=7 points 103 and 8 rays per point 103 is depicted. From 7*8=56 possible paths, only 11 are determined to be geometrically valid. These paths can then be considered as ER trajectories 106.

[0108] Fig.12 a) to d) show the results of the algorithm for the previous audio scene with different listener positions and N=7.

[0109] Optionally, the resulting ER trajectory 106 may be output for further processing, such as rendering an audio scene, for example. Alternatively, the determined ER trajectory 106 may be further analyzed for improved audio scene rendering.

[0110] For example, a set (one or more) of the most acoustically relevant ER trajectories from the ER trajectories 106 may be selected. The selection may be based on the length of the ER trajectories 106 and / or the reflection coefficients of the collision voxels 104 of the ER trajectories 106. The reflection coefficients may depend on the material modeled by the corresponding collision voxels 104. For example, ER trajectories with very large path lengths (e.g., greater than a certain threshold or greater than a certain fraction or multiple of the length of the connecting line between the audio source and the listener) may be discarded. Alternatively or additionally, ER trajectories with small reflection coefficients (e.g., less than a certain threshold) may be discarded.

[0111] Alternatively or additionally, ER trajectories 106 having values ​​indicating large internal angles at the collision voxels 104 may be discarded. A large angle may be defined as an internal angle close to 180°, such as 180°-ε, where ε is a small angle. Alternatively, a large angle in this context may be an internal angle greater than 160°. The value indicating the internal angle may be the internal angle itself or the length of the ER trajectory 106 (note that a large internal angle means a relatively short path length, while a small internal angle means a relatively long path length). Fig.13 An example of a geometrically valid ER trajectory 106 with an interior angle close to 180° is depicted. As a result, the length of the path is very close to the direct path between the audio source 101 and the listener 102. Therefore, the ER will be masked by the direct sound (i.e., sound without reflections) received from the audio source 101. The ER can therefore be determined to be psychoacoustically invalid / irrelevant. Therefore, such an ER can be discarded.

[0112] In another example, two or more determined ER trajectories may be averaged (e.g., spatially averaged). To this end, corresponding image sources may be determined for the two or more ER trajectories, and the determined image sources may be spatially averaged to obtain an averaged image source. Additionally, associated gains of the image sources may be determined (e.g., based on reflection coefficients and / or path lengths), and the gain of the averaged image source may be obtained by averaging the individual gains.

[0113] Through the methods disclosed above, ER directions may be obtained in an efficient manner while still enabling an accurate (eg, faithful or at least realistic) representation of ER effects during sound rendering.

[0114] The method disclosed above can be used for audio processing of a three-dimensional audio scene frame by frame. Alternatively, subdivisions of time (time units) other than frames can be used here. The proposed method is independent of the type of time unit considered.

[0115] The reflection trajectory of a given frame can be estimated based on the method according to the method disclosed above. The estimated early reflection trajectory and the coordinates of the listener position and the audio source position can be stored. The estimated early reflection trajectory can also include the corresponding gain of the audio image source. Generally, the estimated early reflection trajectory can be related to the indication of the image source position and / or the indication of the image source gain. Alternatively, the estimated early reflection trajectory of the previous frame can be accessed. In particular, the coordinates and gain of the audio image source of the stored previous frame can be accessed. The estimated early reflection trajectory of the previous frame can also be estimated based on the method disclosed above. The estimated early reflection trajectory of the previous frame can only be accessed when the voxel containing the listener position, the voxel containing the audio source position, and the geometry of the voxel-based representation of the three-dimensional audio scene have not changed between the frame and the previous frame. The voxel containing the listener position can be represented by the listener head position voxel index. The voxel containing the audio source position can be represented by the audio point source position voxel index. The geometry of a voxel-based representation of a three-dimensional audio scene may be represented by a 3D voxel matrix (eg, associated with reflection coefficients). Fig.14 As depicted in the flowchart of , a method 200 is provided for estimating ER trajectories. The method can be implemented in a decoder or a renderer or in both a decoder and a renderer in an AR / VR / MR / XR environment. The decoder and / or the renderer can be implemented in a network / cloud or a processing device (such as a mobile device and an AR / VR / MR / XR google / lens), or distributed in both a network / cloud and a processing device. In addition to the following method steps, the method 200 can also optionally include the above method steps for combining Figures 7 to 13 All the described variants of the previously mentioned ER estimation algorithms are discussed.

[0116] exist Step S201 In the embodiment of the present invention, a voxel-based representation of a three-dimensional audio scene, information about a listener position of a listener 102 in the three-dimensional audio scene, and information about an audio source position of an audio source 101 in the three-dimensional audio scene are obtained. The voxel-based representation, the information about the listener position, and the information about the audio source position may each be received and / or predetermined (i.e., previously calculated, stored, and then read from a memory).

[0117] exist Step S202 In the method, a ray pattern is applied to one or more points 103 on a connecting line between an audio source position and a listener position to obtain a plurality of rays starting from the corresponding point 103 for each of the one or more points 103. The ray pattern can be received and / or predetermined. Alternatively, the ray pattern can be determined in the context of the proposed method. For example, determining the ray pattern can be understood as selecting one of a set of predefined ray patterns. Determining the ray pattern can be based on the scene type of the audio scene, available computing resources, encoder presets, or a combination thereof. The scene type can include, for example, an indoor scene and / or an outdoor scene. In some embodiments, the scene type can be completely indoor, completely outdoor, or a combination of indoor and outdoor.

[0118] exist Step S203 In step S202, a set of collision voxels is determined based on the plurality of rays determined at step S202 and the voxel-based representation of the three-dimensional audio scene. Fig.15 The method described determines a set of colliding voxels.

[0119] exist Step S204 In , the early reflection trajectory is determined based on a set of collision voxels, the listener position, the audio source position, and a geometric validity test. Fig.16 and / or Fig.17 A method is described for determining early reflection trajectories.

[0120] In the optional Step S205 In the embodiment of the present invention, a set of acoustically most relevant early reflection traces are selected from the ER traces 106. This selection may mean discarding at least one of the ER traces 106.

[0121] In the optional Step S206 In the step S205, an ER track 106 is output for rendering of a three-dimensional audio scene. The ER track 106 may be the ER track of step S205 or step S206.

[0122] Fig.15 A method 300 for determining a set of collision voxels is shown. The method 300 may implement, for example, step S203.

[0123] exist Step 301 In the embodiment of the present invention, one or more intersections between each ray of the plurality of rays and the occluding object pixel 105 are determined.

[0124] exist Step 302 In the example of FIG. 1 , for each ray, an occluder voxel 105 containing an intersection closest to the origin of the corresponding ray is determined as a collision voxel 104. The collision voxels 104 determined in this way form a set of collision voxels.

[0125] Fig.16 A method 400 for determining early reflection trajectories is shown. The method 400 may implement, for example, step S204.

[0126] exist Step 401 In, for each collision voxel in a set of collision voxels, it is determined whether the collision voxel can produce a geometrically valid representation of the first-order reflection.

[0127] exist Step 402 In the method, if the collision voxel can produce a geometrically valid representation of the first-order reflection, a path connecting the listener position and the audio source position via the corresponding collision voxel is determined as an early reflection trajectory.

[0128] Fig.17 A method 500 for determining early reflection trajectories is shown. The method 500 may implement, for example, step S204.

[0129] exist Step 501 In the embodiment of the present invention, for each collision voxel in a set of collision voxels, a path connecting the listener position and the audio source position via the corresponding collision voxel 104 is determined. The path may include two straight line segments, as described above. In other words, the audio source 101 and the listener 102 may be connected via the collision voxel 104 via a straight line.

[0130] exist Step 502 In step S501, for each path determined at step S501, if the path is geometrically valid, the corresponding path is determined as the ER trajectory 106. If the path is not blocked by occluding voxels other than the corresponding collision voxel, it can be judged to be geometrically valid.

[0131] Although the method for estimating ER trajectory has been described above, the present disclosure also relates to corresponding devices and the like. Fig.18 Embodiments providing such an apparatus are described.

[0132] like Fig.18As shown in , the device 400 includes a processor 401 and a memory 402. The memory 402 is configured to store program code. The processor 401 is configured to run the instructions in the program code so that the device 400 performs the ER trajectory estimation method in any of the above embodiments and implementations. Among other things, the processor 401 can also receive suitable input data (e.g., voxel grid, voxel data, audio source and listener position, etc.), depending on the use case and / or implementation. The processor 401 can be suitable for executing the methods / techniques described throughout the present disclosure (e.g., as described above with reference to each other). Figures 14 to 17 The apparatus may include methods 200, 300, 400, and 500 of the present invention and generate corresponding output data (e.g., ER trajectories, etc.), depending on the use case and / or implementation. The apparatus may be part of a virtual reality (VR), augmented reality (AR), mixed reality (MR), and / or extended reality (XR) device. Additionally, the apparatus may relate to a decoder device (decoder-side device) or a rendering device, such as in the context of a VR / AR / MR / XR environment.

[0133] Aspects of the systems described herein can be implemented in a computer-based sound processing network environment suitable for processing digital or digitized audio files. Portions of the adaptive audio system may include one or more networks comprising any desired number of individual machines, including one or more routers (not shown) for buffering and routing data transmitted between computers. Such networks may be constructed on a variety of different network protocols and may be the Internet, a wide area network (WAN), a local area network (LAN), or any combination thereof.

[0134] One or more of the components, blocks, processes, or other functional components may be implemented by a computer program executed by a processor-based computing device that controls the system. It should also be noted that the various functions disclosed herein may be described using any number of combinations of hardware, firmware, and / or as data and / or instructions contained in various machine-readable or computer-readable media in terms of their behavior, register transfers, logic components, and / or other characteristics. Computer-readable media that may contain such formatted data and / or instructions include, but are not limited to, various forms of physical (non-transient), non-volatile storage media, such as optical, magnetic, or semiconductor storage media.

[0135] Although one or more implementations have been described by way of example and with respect to specific embodiments, it is to be understood that one or more implementations are not limited to the disclosed embodiments. On the contrary, it is intended to cover various modifications and similar arrangements that will be clear to those skilled in the art. Therefore, the scope of the appended claims should be given the broadest interpretation so as to cover all such modifications and similar arrangements.

[0136] explain

[0137] A computing device implementing the techniques described above may have the following example architecture. Other architectures are possible, including architectures with more or fewer components. In some implementations, the example architecture includes one or more processors (e.g., dual-core The components may include a processor, one or more output devices (e.g., LCD), one or more network interfaces, one or more input devices (e.g., mouse, keyboard, touch-sensitive display), and one or more computer-readable media (e.g., RAM, ROM, SDRAM, hard disk, optical disk, flash memory, etc.). These components may exchange messages and data through one or more communication channels (e.g., buses), which may utilize various hardware and software to facilitate the transmission of data and control signals between components.

[0138] The term "computer-readable medium" refers to media that participate in providing instructions to a processor for execution, including but not limited to non-volatile media (e.g., optical or magnetic disks), volatile media (e.g., memory), and transmission media. Transmission media include but are not limited to coaxial cables, copper wire, and fiber optics.

[0139] The computer-readable medium may also include an operating system (e.g., The operating system may include an operating system, an audio interface manager, an audio processing manager, and a live content distributor. The operating system may be multi-user, multi-processing, multi-tasking, multi-threaded, real-time, and the like. The operating system performs basic tasks including, but not limited to: identifying input from and providing output to a network interface and / or device; keeping track of and managing files and directories on a computer-readable medium (e.g., a memory or storage device); controlling peripheral devices; and managing traffic on one or more communication channels. The network communication module includes various components for establishing and maintaining network connections (e.g., software for implementing communication protocols such as TCP / IP, HTTP, etc.).

[0140] The architecture may be implemented in a parallel processing or peer-to-peer infrastructure or on a single device having one or more processors.The software may include multiple software components or may be a single body of code.

[0141] The features described may advantageously be implemented in one or more computer programs executable on a programmable system including at least one programmable processor coupled to receive data and instructions from and transmit data and instructions to a data storage system, at least one input device, and at least one output device. A computer program is a set of instructions that can be used directly or indirectly in a computer to perform an activity or cause a result. A computer program may be written in any form of programming language (e.g., Objective-C, Java), including compiled or interpreted languages, and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, browser-based web application, or other unit suitable for use in a computing environment.

[0142] Processors suitable for the execution of a program of instructions include, for example, both general-purpose and special-purpose microprocessors of any kind of computer, and a single processor or one of multiple processors or cores. Generally, the processor will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for executing instructions and one or more memories for storing instructions and data. Generally, a computer will also include one or more mass storage devices for storing data files, or be operatively coupled to communicate with one or more mass storage devices for storing data files; such devices include disks, such as internal hard disks and removable disks; magneto-optical disks; and optical disks. Storage devices suitable for tangibly containing computer program instructions and data include all forms of non-volatile memory, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and memory can be supplemented by or incorporated into an ASIC (application-specific integrated circuit).

[0143] To provide for interaction with a user, features may be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor or retina display device. The computer may have a touch surface input device (e.g., a touch screen) or a keyboard and pointing device (such as a mouse or trackball) through which a user can provide input to the computer. The computer may have a voice input device for receiving voice commands from the user.

[0144] Features may be implemented in a computer system that includes a back-end component, such as a data server, or includes a middleware component, such as an application server or an Internet server, or includes a front-end component, such as a client computer with a graphical user interface or an Internet browser, or any combination thereof. The components of the system may be connected by any form or medium of digital data communication, such as a communication network. Examples of communication networks include, for example, a LAN, a WAN, and the computers and networks that form the Internet.

[0145] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by means of computer programs that run on respective computers and have a client-server relationship with each other. In some embodiments, the server transmits data (e.g., an HTML page) to a client device (e.g., for the purpose of displaying data to a user interacting with the client device and receiving user input from a user interacting with the client device). Data generated at the client device (e.g., the result of the user interaction) can be received from the client device at the server.

[0146] A system of one or more computers may be configured to perform specific actions by installing software, firmware, hardware, or a combination thereof on the system, which, when operated, causes the system to perform the actions. One or more computer programs may be configured to perform specific actions by including instructions that, when executed by a data processing device, cause the device to perform the actions.

[0147] Although this specification contains many specific implementation details, these should not be interpreted as limitations on the scope of any invention or content that can be claimed, but should be interpreted as descriptions of features specific to specific embodiments of specific inventions. Certain features described in the context of separate embodiments in this specification may also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented separately or in any suitable sub-combination in multiple embodiments. Moreover, although features may be described above as working in certain combinations, and even initially claimed as such, one or more features from the claimed combination may be deleted from the combination in some cases, and the claimed combination may be directed to a variant of a sub-combination or a sub-combination.

[0148] Similarly, although operations are described in a particular order in the figures, this should not be understood as requiring such operations to be performed in the particular order shown or in a sequential order, or all illustrated operations to be performed, to achieve the desired result. In some cases, multitasking and parallel processing can be advantageous. Moreover, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described can generally be integrated together in a single software product or packaged into multiple software products.

[0149] Unless otherwise specifically stated, as will be clear from the following discussion, it is recognized that throughout the present invention, discussions utilizing terms such as "processing," "computing," "calculating," "determining," "analyzing," etc., refer to the actions and / or processes of a computer or computing system or similar electronic computing device that manipulates and / or transforms data represented as physical quantities (such as, electronic quantities) into other data similarly represented as physical quantities.

[0150] References throughout the present invention to "an example embodiment," "some example embodiments," or "example embodiments" mean that a particular feature, structure, or characteristic described in connection with the example embodiment is included in at least one example embodiment of the present invention. Thus, various appearances of the phrases "in an example embodiment," "in some example embodiments," or "in an example embodiment" throughout the present invention are not necessarily all referring to the same example embodiment. Furthermore, the particular features, structures, or characteristics may be combined in one or more example embodiments in any suitable manner as will be apparent to one of ordinary skill in the art from this disclosure.

[0151] As used herein, unless otherwise specified, the use of ordinal adjectives "first," "second," "third," etc. to describe common objects indicates merely that different instances of the same object are referred to, and is not intended to mean that the objects so described must be in a given order, whether temporally, spatially, in ranking, or in any other manner.

[0152] Furthermore, it is to be understood that the phraseology and terminology used herein is for descriptive purposes and should not be construed as limiting. The use of "including," "comprising," or "having" and variations thereof are meant to encompass the items listed thereafter and their equivalents as well as additional items. Unless otherwise specified or limited, the terms "mounted," "connected," "supported," and "coupled," and variations thereof, are used broadly and encompass both direct and indirect mountings, connections, supports, and couplings.

[0153] In the following claims and the description herein, any of the terms "comprise", "comprising", or "it includes" is an open term, which means at least including the following elements / features, but not excluding others. Therefore, when used in the claims, the term "comprising" should not be interpreted as being limited to the parts or elements or steps listed later. For example, the scope of the expression "a device comprising A and B" should not be limited to a device consisting of only elements A and B. Any of the terms "comprising" or "it includes" or "comprising" used herein is also an open term, which also means at least including the elements / features following the term, but not excluding others. Therefore, inclusion is synonymous with inclusion and means inclusion.

[0154] It should be appreciated that in the above description of example embodiments of the invention, various features of the invention are sometimes grouped together in a single example embodiment, figure, or description thereof for the purpose of streamlining the invention and aiding in the understanding of one or more of the various inventive aspects. However, this method of invention should not be interpreted as reflecting an intention that the claims require more features than those expressly recited in each claim. Rather, as the following claims reflect, inventive aspects reside in less than all the features of a single, previously disclosed example embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate example embodiment of the invention.

[0155] In addition, although some of the example embodiments described herein include some features included in other example embodiments but do not include other features, as will be understood by those skilled in the art, the combination of features of different example embodiments is meant to be within the scope of the present invention and to form different example embodiments. For example, in the following claims, any of the example embodiments claimed for protection may be used in any combination.

[0156] In the description provided herein, many specific details are set forth. However, it is understood that the exemplary embodiments of the present invention can be practiced without these specific details. In other cases, well-known methods, structures and techniques are not shown in detail in order not to obscure the understanding of this description.

[0157] Therefore, although what is considered to be the best mode of the present invention has been described, it will be appreciated by those skilled in the art that other and further modifications may be made thereto without departing from the spirit of the present invention, and it is intended that all such changes and modifications fall within the scope of the present invention. For example, any formula given above merely represents a procedure that may be used. Functions may be added or deleted from the block diagram, and operations may be interchanged between functional blocks. Steps may be added or deleted to the described method within the scope of the present disclosure.

[0158] Various aspects and implementations of the present disclosure may also be realized from the following non-claimed example embodiments (EEE).

[0159] EEE1. A method for estimating early reflection trajectories of an audio source in a three-dimensional audio scene, the method comprising:

[0160] obtaining a voxel-based representation of the three-dimensional audio scene, information about a listener position of a listener in the three-dimensional audio scene, and information about an audio source position of the audio source in the three-dimensional audio scene;

[0161] applying the ray pattern to one or more points on a connecting line between the audio source position and the listener position to obtain, for each of the one or more points, a plurality of rays originating from the corresponding point;

[0162] determining a set of collision voxels based on the plurality of rays and a voxel-based representation of the three-dimensional audio scene;

[0163] An early reflection trajectory is determined based on the set of collision voxels, the listener position, the audio source position, and a geometric validity test.

[0164] EEE1A. A method for estimating early reflection trajectories of audio sources in a three-dimensional audio scene, the method comprising:

[0165] obtaining a voxel-based representation of the three-dimensional audio scene, information about a listener position of a listener in the three-dimensional audio scene, and information about an audio source position of the audio source in the three-dimensional audio scene;

[0166] applying the ray pattern to one or more points on a connecting line between the audio source position and the listener position to obtain, for each of the one or more points, a plurality of rays originating from the corresponding point;

[0167] determining a set of collision voxels based on the plurality of rays and a voxel-based representation of the three-dimensional audio scene;

[0168] For each collision voxel in the set of collision voxels, determining a path connecting the listener position and the audio source position via the corresponding collision voxel; and

[0169] For each path, if the path geometry is valid, then the path is determined to be an early reflection trajectory.

[0170] EEE2.The method of EEE 1 or EEE 1A, further comprising:

[0171] The ray pattern is determined.

[0172] EEE3. The method of EEE 1, 1A or 2, further comprising:

[0173] The one or more points are determined based on the obtained cardinality of the one or more points.

[0174] EEE4. The method of any one of EEE 1 to 3 or EEE 1A, wherein the ray pattern defines a predefined number of rays and predefined directions of the rays from an origin. EEE4.

[0175] EEE5. The method of EEE 4, wherein the predefined number of rays is 6, 8, or 12.

[0176] EEE6. The method of EEE 5, wherein the voxel position in the three-dimensional audio grid is defined by a grid index, and the predefined direction of the ray includes one or more of the following:

[0177] The horizontal and vertical directions of the grid index to the adjacent grid index; and the diagonal direction of the grid index to the adjacent grid index.

[0178] EEE7.EEE 2's method, wherein determining the ray pattern is based on a scene type of the three-dimensional audio scene, available computing resources, encoder presets, or a combination thereof.

[0179] EEE8.EEE 3. The method of claim 3, wherein the coordinates of the one or more points on the line connecting the audio source position and the listener position are determined based on a cardinality of the one or more points.

[0180] EEE9.EEE 8's method, wherein the one or more points are determined to divide a line connecting the audio source position and the listener position into N-1 equal line segments, where N is the cardinality of the one or more points and is greater than or equal to 2.

[0181] EEE10.EEE 3. The method of claim 3, wherein the cardinality of the one or more points depends on a scene type of the three-dimensional audio scene, available computing resources, encoder presets, or a combination thereof.

[0182] EEE11. The method of EEE 7 or 10, wherein the scene types include indoor scenes and outdoor scenes.

[0183] EEE12. The method of any of EEE 1 to 11 or EEE 1A, wherein each collision voxel in the set of collision voxels is an occluder voxel in the voxel-based representation of the three-dimensional audio scene. EEE13. The method of any of EEE 1 to 11 or EEE 1A, wherein each collision voxel in the set of collision voxels is an occluder voxel in the voxel-based representation of the three-dimensional audio scene.

[0184] EEE13. The method of EEE 12, wherein the occluding object pixels represent acoustically reflecting surfaces.

[0185] EEE14. The method of EEE 12, wherein the occluder voxels represent any material other than air in the voxel-based representation of the three-dimensional audio scene.

[0186] EEE15. The method of any one of EEEs 12 to 14, wherein determining the set of collision voxels based on the plurality of rays and the voxel-based representation of the three-dimensional audio scene comprises:

[0187] determining one or more intersections between each ray of the plurality of rays and the occluder pixel; and

[0188] For each ray, the occluder voxel containing the intersection closest to the origin of the corresponding ray is determined as a collision voxel in the set of collision voxels.

[0189] EEE16. The method of any one of EEE 1 to 15 or EEE 1A, wherein determining an early reflection trajectory based on the set of collision voxels, the listener position, the audio source position, and a geometric validity test comprises:

[0190] For each collision voxel in the set of collision voxels, determining whether the collision voxel can produce a geometrically valid representation of a first-order reflection; and

[0191] If the collision voxel can produce a geometrically valid representation of the first-order reflection, a path connecting the listener position and the audio source position via the corresponding collision voxel is determined as an early reflection trajectory.

[0192] EEE17. The method of EEE 16, wherein determining whether a collision voxel can produce a geometrically valid representation of a first-order reflection comprises:

[0193] determining a preceding voxel of the collision voxel, wherein the preceding voxel is a voxel that contains an intersection with the corresponding ray and is preceding the collision voxel in the direction of the corresponding ray;

[0194] determining a second path connecting the listener position and the audio source position via corresponding preceding voxels; and

[0195] If the second path does not contain an intersection with an occluding voxel, then determining that the collision voxel can produce a geometrically valid representation of the first order reflection.

[0196] EEE18. The method of any one of EEE 1 to 17 or EEE 1A, wherein determining an early reflection trajectory based on the set of collision voxels, the listener position, the audio source position, and a geometric validity test comprises:

[0197] For each collision voxel in the set of collision voxels, determining a path connecting the listener position and the audio source position via the corresponding collision voxel; and

[0198] For each path, if the path geometry is valid, then the path is determined to be an early reflection trajectory.

[0199] EEE19. The method of EEE 16 or EEE 18, wherein the path includes a straight line connecting the audio source position to a collision voxel in the set of collision voxels, and a straight line connecting the same collision voxel in the set of collision voxels to the listener position.

[0200] EEE20. The method of EEE 18, wherein the path is determined to be geometrically valid if the path contains no intersections with occluder voxels other than the corresponding path's collision voxel.

[0201] EEE21. The method of any one of EEE 1 to 20 or EEE 1A, further comprising:

[0202] A set of acoustically most relevant early reflection trajectories is selected from the early reflection trajectories.

[0203] EEE22. The method of EEE 21, wherein selecting the set of acoustically most relevant early reflection trajectories is based on the lengths of the early reflection trajectories and / or the reflection coefficients of the colliding voxels of the early reflection trajectories. EEE23. The method of EEE 23, wherein selecting the set of acoustically most relevant early reflection trajectories is based on the lengths of the early reflection trajectories and / or the reflection coefficients of the coll

[0204] EEE23. The method of EEE 22, wherein the reflection coefficient depends on the material modeled by the collision voxel.

[0205] EEE24. The method of any one of EEEs 21 to 23, wherein selecting the set of acoustically most relevant early reflection trajectories comprises discarding early reflection trajectories having values ​​indicating an internal angle greater than 160° at the collision voxel. EEE25. The method of any one of EEEs 21 to 23, wherein selecting the set of acoustically most relevant early reflection trajectories comprises discarding early reflection trajectories having values ​​indicating an internal angle greater than 160° at the collision voxel.

[0206] EEE25. The method of EEE 24, wherein the value indicating an internal angle greater than 160° is the internal angle or the length of an early reflection trajectory.

[0207] EEE26. The method of any one of EEE 1 to 25 or EEE 1A, further comprising:

[0208] The early reflection trace is output for use in rendering the three-dimensional audio scene.

[0209] EEE27. The method of EEE 26, wherein the rendering is performed by a virtual reality VR, augmented reality AR, mixed reality MR, and / or extended reality XR device.

[0210] EEE28. The method of any of EEE 1 to 27 or EEE 1A, wherein the early reflection trajectory represents a first-order trajectory.

[0211] EEE29. The method of EEE 28, wherein the first-order trajectory is a reflection trajectory having a single reflection between the audio source position and the listener position.

[0212] EEE30. The method of any one of EEE 1 to 29 or EEE 1A, wherein the method is performed by a decoder or a renderer.

[0213] EEE31. A method for processing frames of a three-dimensional audio scene, the method comprising:

[0214] estimating an early reflection trace of the frame and storing the estimated early reflection trace based on the method of any one of claims 1 to 30; or

[0215] if:

[0216] voxels containing listener positions, voxels containing audio source positions, and the geometry of the voxel-based representation of the three-dimensional audio scene have not changed between the frame and the previous frame, then

[0217] An estimated early reflection trajectory of the previous frame estimated based on the method of any one of claims 1 to 30 is accessed.

[0218] EEE32. An apparatus comprising a processor and a memory coupled to the processor, wherein the processor is adapted to perform a method according to any one of EEE 1 to 31 or EEE 1A. EEE32.

[0219] EEE33. A program comprising instructions which, when executed by a processor, cause the processor to perform a method according to any one of EEE 1 to 31 or EEE 1A. EEE33.

[0220] EEE34. A computer-readable storage medium storing a program according to EEE 33.

[0221] EEE35. A method for creating tracks for geometrically connected audio sources for efficient implementation of audio processing on a voxel 3D grid, the method comprising:

[0222] receiving information associated with a ray pattern 'R';

[0223] determining a first set of points 'P' to which ray casting is to be applied based on the ray pattern 'R';

[0224] determining a second set of ray-voxel 'collision' voxels 'C' based on the first set of points and the reflection voxels 'VOX';

[0225] determining a third set of valid reflection trajectories 'SC-L' based on the second set of ray-voxel 'collision' voxels 'C', and

[0226] A subset of the most acoustically relevant valid reflection trajectories is selected and output from the third set of valid reflection trajectories.

[0227] EEE36. The method of EEE 35, wherein the second set of ray-voxel 'collision' voxels 'C' are determined based on the ray direction map 'R' applied to the first set of points 'P' and the reflection voxels 'VOX.

[0228] EEE37. The method of EEE 35, wherein the reflection trajectory 'SC-L' represents a first-order trajectory.

[0229] EEE38. The method of EEE 35 also includes checking whether a first line LC connecting the listener and the collision voxel and a second line 'S-C' connecting the audio source and the collision voxel intersect any blocking / occluding / reflecting voxels, and determining that this is a valid approximation of a first-order reflection based on a determination that no intersection exists.

[0230] EEE39. A non-transitory computer program comprising instructions which, when executed by a processor, cause the processor to perform a method according to any one of EEE 35-38. EEE39.

[0231] EEE40. A device configured to perform the method of EEE 35-38.

Claims

1. A method for estimating early reflection trajectories of audio sources in a three-dimensional audio scene, the method comprising: Obtaining (S201) a voxel-based representation of the three-dimensional audio scene, information about a listener position of a listener in the three-dimensional audio scene, and information about an audio source position of the audio source in the three-dimensional audio scene; Applying (S202) a ray pattern to one or more points on a connecting line between the audio source position and the listener position to obtain, for each of the one or more points, a plurality of rays starting from the corresponding point; determining ( S203 ) a set of collision voxels based on the plurality of rays and the voxel-based representation of the three-dimensional audio scene; An early reflection trajectory is determined ( S204 ) based on the set of collision voxels, the listener position, the audio source position, and a geometric validity test.

2. The method of claim 1, further comprising: The ray pattern is determined.

3. The method according to claim 1 or 2, further comprising: The one or more points are determined based on the obtained cardinality of the one or more points.

4. The method of any one of claims 1 to 3, wherein the ray pattern defines a predefined number of rays and a predefined direction of the rays from an origin. The method of claim 4 , wherein the predefined number of rays is 6, 8, or 12.

6. The method of claim 5, wherein the voxel position in the three-dimensional audio grid is defined by a grid index, and the predefined direction of the ray comprises one or more of the following: The horizontal and vertical directions of the grid index to the adjacent grid index; and the diagonal direction of the grid index to the adjacent grid index.

7. The method of claim 2, wherein determining the ray pattern is based on a scene type of the three-dimensional audio scene, available computing resources, an encoder preset, or a combination thereof.

8. The method of claim 3, wherein the coordinates of the one or more points on the line connecting the audio source location and the listener location are determined based on a cardinality of the one or more points.

9. The method of claim 8, wherein the one or more points are determined to divide a line connecting the audio source position and the listener position into N-1 equal line segments, where N is a cardinality of the one or more points and is greater than or equal to 2.

10. The method of claim 3, wherein the cardinality of the one or more points depends on a scene type of the three-dimensional audio scene, available computing resources, an encoder preset, or a combination thereof.

11. The method according to claim 7 or 10, wherein the scene types include indoor scenes and outdoor scenes.

12. The method of any one of claims 1 to 11, wherein each collision voxel in the set of collision voxels is an occluder voxel in the voxel-based representation of the three-dimensional audio scene. The method of claim 12 , wherein the occluding object voxels represent acoustically reflective surfaces.

14. The method of claim 12, wherein the occluder voxels represent any material other than air in the voxel-based representation of the three-dimensional audio scene.

15. The method of any one of claims 12 to 14, wherein determining the set of collision voxels based on the plurality of rays and the voxel-based representation of the three-dimensional audio scene comprises: determining one or more intersections between each ray of the plurality of rays and the occluder pixel; as well as For each ray, the occluder voxel containing the intersection closest to the origin of the corresponding ray is determined as a collision voxel in the set of collision voxels.

16. The method of any one of claims 1 to 15, wherein determining (S204) an early reflection trajectory based on the set of collision voxels, the listener position, the audio source position, and a geometric validity test comprises: For each collision voxel in the set of collision voxels, determining (S401) whether the collision voxel can produce a geometrically valid representation of a first-order reflection; as well as If the collision voxel can generate a geometrically valid representation of the first-order reflection, then a path connecting the listener position and the audio source position via the corresponding collision voxel is determined (S402) as an early reflection trajectory.

17. The method of claim 16, wherein determining (S401) whether a collision voxel can produce a geometrically valid representation of a first-order reflection comprises: determining a preceding voxel of the collision voxel, wherein the preceding voxel is a voxel that contains an intersection with the corresponding ray and is preceding the collision voxel in the direction of the corresponding ray; determining a second path connecting the listener position and the audio source position via corresponding preceding voxels; as well as If the second path does not contain an intersection with an occluding voxel, then determining that the collision voxel can produce a geometrically valid representation of the first order reflection.

18. The method of any one of claims 1 to 17, wherein determining (S204) an early reflection trajectory based on the set of collision voxels, the listener position, the audio source position, and a geometric validity test comprises: For each collision voxel in the set of collision voxels, determining (S501) a path connecting the listener position and the audio source position via the corresponding collision voxel; as well as For each path, if the path is geometrically valid, then the path is determined (S502) as an early reflection trajectory.

19. The method of claim 16 or claim 18, wherein the path comprises a straight line connecting the audio source position to a collision voxel in the set of collision voxels, and a straight line connecting the same collision voxel in the set of collision voxels to the listener position.

20. The method of claim 18, wherein the path is determined to be geometrically valid if the path contains no intersections with occluder voxels other than the corresponding path's collision voxel.

21. The method of any one of claims 1 to 20, further comprising: A set of acoustically most relevant early reflection trajectories is selected from the early reflection trajectories.

22. The method of claim 21, wherein selecting the set of acoustically most relevant early reflection trajectories is based on the lengths of the early reflection trajectories and / or the reflection coefficients of the colliding voxels of the early reflection trajectories.

23. The method of claim 22, wherein the reflectivity is dependent on the material modeled by the collision voxel.

24. The method of any one of claims 21 to 23, wherein selecting the set of acoustically most relevant early reflection trajectories comprises discarding early reflection trajectories having values ​​indicating an internal angle at the impact voxel close to 180°.

25. The method of claim 24, wherein the value indicative of an interior angle close to 180° is the interior angle or a length of an early reflection trace.

26. The method of any one of claims 1 to 25, further comprising: The early reflection trace is output for use in rendering the three-dimensional audio scene.

27. The method of claim 26, wherein the rendering is performed by a virtual reality (VR), augmented reality (AR), mixed reality (MR), and / or extended reality (XR) device.

28. A method as claimed in any one of claims 1 to 27, wherein the early reflection trace represents a first order trace.

29. The method of claim 28, wherein the first-order trajectory is a reflection trajectory having a single reflection between the audio source position and the listener position.

30. The method of any one of claims 1 to 29, wherein the method is performed by a decoder or a renderer.

31. A method of processing frames of a three-dimensional audio scene, the method comprising: estimating an early reflection trajectory of the frame based on the method of any one of claims 1 to 30 and storing the estimated early reflection trajectory; or if: voxels containing listener positions, voxels containing audio source positions, and the geometry of the voxel-based representation of the three-dimensional audio scene have not changed between the frame and the previous frame, then An estimated early reflection trajectory of the previous frame estimated based on the method of any one of claims 1 to 30 is accessed.

32. An apparatus comprising a processor and a memory coupled to the processor, wherein the processor is adapted to perform the method according to any one of claims 1 to 31.

33. A program comprising instructions which, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 31.

34. A computer-readable storage medium storing the program according to claim 33.

Images