Rendering of occluded audio elements

Abstract

To provide a method for rendering occluded audio elements.SOLUTION: The method comprises a step s802 of determining a first point in the first extent that is not fully occluded, and a step s804 of determining a second extent for the audio piece. The determining includes determining a first edge of the second extent using the first point. The method also comprises, after determining the second extent, the steps of dividing s806 the second extent into a set of one or more sub-areas comprising at least the first sub-area, determining s808 a first gain value for a first sample point of the first sub-area, and rendering s810 the audio elements using the first gain value.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] Embodiments related to rendering occluded audio elements are disclosed. [Background technology]

[0002] Spatial audio rendering is a process used to present audio in extended reality (XR) scenes (e.g., virtual reality (VR), augmented reality (AR), and mixed reality (MR) scenes) that gives the listener the impression that the sound is coming from a physical source in the scene at a specific location and has a specific size and shape (i.e., extent). Presentation can be through headphone speakers or other speakers. When presentation is through headphone speakers, the process used is called binaural rendering, which uses spatial cues of human spatial hearing that allow one to identify which direction a sound is coming from. The cues involve interaural time delay (ITD), interaural level difference (ILD), and / or spectral differences.

[0003] The most common form of spatial audio rendering is based on the concept of point sources, where each sound source is defined as emanating from one specific point. Because each sound source is defined as emanating from one specific point, the sound sources do not have any size or shape. Different methods have been developed to render sound sources with extent (size and shape).

[0004] One such known method is to create multiple copies of a mono audio element at positions around the audio element. This configuration creates the perception of a spatially homogeneous object with a specific size. This concept is used, for example, in the "object spread" and "object divergence" features of the MPEG-H 3D Audio standard (see References [1] and [2]) and the "object divergence" feature of the EBU Audio Definition Model (ADM) standard (see Reference [4]). This idea using mono audio sources has been further developed as described in Reference [7], where the area-volume geometry of the sound object is projected onto a sphere around the listener, and the sound is rendered to the listener using a pair of head-related (HR) filters evaluated as the integral of all HR filters covering the geometric projection of the object on the sphere. For spherical volumetric sources, this integral has an analytical solution. However, for arbitrary area-volume source geometries, the integral is evaluated by sampling the projected source surface on the sphere using so-called Monte Carlo ray sampling.

[0005] Another rendering method is to render a spatially diffuse component in addition to the mono audio signal, which creates the perception of somewhat diffuse objects that, in contrast to the original mono audio elements, do not have a distinct pinpoint location. This concept is used, for example, in the "object diffuseness" feature of the MPEG-H 3D Audio standard (see reference [3]) and the "object diffuseness" feature of the EBU ADM (see reference [5]).

[0006] Combinations of the two methods above are also known, for example the "object extent" feature of the EBU ADM, which combines the creation of multiple copies of a mono audio element with the addition of a diffuse component (see reference [6]).

[0007] In many cases, the actual shape of an audio element can be adequately described by a basic shape (e.g., a sphere or a box), but the actual shape may be more complex and need to be described in a more detailed form (e.g., a mesh structure or a parametric description form).

[0008] For heterogeneous audio elements, as described in reference [8], the audio element includes at least two audio channels (i.e., audio signals) to describe the spatial variation over its range.

[0009] In some XR scenes, there may be objects that block at least a portion of an audio element in the XR scene, in such a scenario the audio element is said to be at least partially occluded.

[0010] That is, occlusion occurs when, from the perspective of a listener (e.g., a human listener) at a given listening position, an audio object is completely or partially obscured behind some object such that no, or very little, direct sound from the occluded portion of the object reaches the listener. Depending on the material of the occluding object, the occlusion effect can be either complete occlusion (e.g., when the occluding object is a thick wall) or non-complete occlusion (e.g., when the occluding object is made of a thin fabric such as a curtain) (also known as "soft" occlusion). Soft occlusion can often be well described by a filter with a specific frequency response that matches the acoustic properties of the occluding object's material. Summary of the Invention [Problem to be solved by the invention]

[0011] Currently, several challenges exist. For example, available occlusion rendering techniques handle point sources, where the occurrence of occlusion can be easily detected using ray tracing between the listener position and the point source position, but for audio elements that have an extent, the situation is more complicated because the occluding object may only occlude a portion of the audio element's extent. To achieve sufficient resolution of the occlusion effect, a large number of rays must be cast toward the extent, which adds a significant amount of processing complexity to the rendering of the audio object. [Means for solving the problem]

[0012] Thus, in one aspect, an improved method for rendering an audio element associated with a first extent is provided. In one embodiment, the method includes determining a first point in the first extent that is not fully occluded. The method also includes determining a second extent for the audio element, where determining includes determining a first edge of the second extent using the first point. The method also includes, after determining the second extent, dividing the second extent into a set of one or more subareas, including a first subarea. The method also includes determining a first gain value for a first sample point of the first subarea. The method further includes rendering the audio element using the first gain value (e.g., generating an output signal using the first gain value).

[0013] In another aspect, a computer program is provided that includes instructions that, when executed by a processing circuit of an audio renderer, cause the audio renderer to perform the methods disclosed herein. In one embodiment, a carrier is provided that includes the computer program, the carrier being one of an electrical signal, an optical signal, a wireless signal, and a computer-readable storage medium. In another aspect, a rendering device is provided that is configured to perform the methods disclosed herein. The rendering device may include a memory and a processing circuit coupled to the memory.

[0014] An advantage of the embodiments disclosed herein is that it allows for detecting occlusions to extents with good accuracy using a limited number of ray casts. [Brief explanation of the drawings]

[0015] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate various embodiments.

[0016] [Figure 1] Diagram showing two point sound sources (S1 and S2) and an occluding object (O).

[0017] [Figure 2] FIG. 10 illustrates an audio element having an extent that is partially occluded by an occluding object.

[0018] [Figure 3] FIG. 10 illustrates a process for determining the edges of a modified extent.

[0019] [Figure 4] Diagram showing changed extents.

[0020] [Figure 5] A diagram showing an audio element where the edges of the element's extent are not completely occluded.

[0021] [Figure 6A] Diagram showing the effect of a moving occluding object when occlusion detection is done on a uniform grid of raycasts.

[0022] [Figure 6B] A diagram showing the effect of a moving occluding object when occlusion detection is performed using a uniform grid of raycasts.

[0023] [Figure 7A] A diagram showing the effect of a moving occluding object when occlusion detection is performed using a diagonal grid of raycasts.

[0024] [Figure 7B] A diagram showing the effect of a moving occluding object when occlusion detection is performed using a distorted grid of raycasts.

[0025] [Figure 8] 1 is a flowchart showing a process according to an embodiment.

[0026] [Figure 9A] FIG. 1 illustrates a system according to some embodiments.

[0027] [Figure 9B] FIG. 1 illustrates a system according to some embodiments.

[0028] [Figure 10] FIG. 1 illustrates a system according to some embodiments.

[0029] [Figure 11] FIG. 2 illustrates a signal modifier according to one embodiment.

[0030] [Figure 12]1 is a block diagram of an apparatus according to some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0031] The occurrence of occlusion can be detected using ray tracing methods, in which the direct sound path (or "path" for short) between the listener's position and the position of an audio element is searched for any object occluding the audio element. Figure 1 shows an example of two point sound sources (S1 and S2), one of which (i.e., S2) is occluded by an object (O) (called the "occluding object") from the listener's perspective, and the other (i.e., S1) is not occluded from the listener's perspective. In this case, the occluded audio element S2 should be muted in a manner that corresponds to the acoustic characteristics of the occluding object's data. If the occluding object is a thick wall, the rendering of the direct sound from the occluded audio element should be more or less completely muted.

[0032] For a given frequency range, any given portion of an audio element may be fully occluded, partially occluded, or unoccluded. The frequency range may be the entire frequency range that can be perceived by humans, or a subset of that frequency range. In one embodiment, a portion of an audio element is fully occluded in a given frequency range when an occlusion gain factor (or "gain" for short) associated with the portion of the audio element satisfies a predetermined condition. For example, a portion of an audio element is fully occluded in a given frequency range when the occlusion gain (which may or may not be frequency-dependent) associated with the portion of the audio element is less than or equal to a threshold gain value (T), where the value T is a selected value (e.g., T=0 is one possibility). That is, for example, any one or more occluding objects that pass less than a certain amount of sound are considered fully occluded. In another embodiment, there is a frequency-dependent determination in which the amount of occlusion in different frequency bands is compared to a predetermined table of threshold values ​​for those frequency bands. In yet another embodiment, the current signal power of the audio signal representing the audio source is used to estimate the actual sound power passing to the listener, and then the sound power is compared to the hearing threshold. In essence, a fully occluded audio element (or part thereof) can be defined as a sound path in which the sound is so suppressed that it is perceptually insignificant. This includes cases where the sound is completely blocked by occlusion, meaning that no sound passes through at all, or cases where the occluding object passes only a small amount of the original sound energy, which does not contribute enough to have a perceptual impact on the overall rendering of the audio source.

[0033] For example, when there is a "hard" occluding object in the sound path, i.e., an imaginary straight line from the listening position to the portion of the audio element, the portion of the audio element will be completely occluded. An example of a hard occluding object is a thick brick wall. On the other hand, when there is a "soft" occluding object in the sound path, for example, the portion of the audio element may be partially occluded. An example of a soft occluding object is a thin curtain.

[0034] If one or several soft-occluding objects are in the sound path, the occlusion effect can be calculated as a filter that corresponds to the audio transmission characteristics of the data. This filter can be specified as a list of frequency ranges, and for each listed frequency range, a corresponding gain can be specified. If two or more soft-occluding objects are in the path, the filters of the data of those objects can be multiplied together to form one composite filter that corresponds to the audio transmission characteristics of that path.

[0035] Ray tracing can be initiated by specifying a start and end point, or by specifying the start point and direction of the ray in polar format, which implies horizontal and vertical angles, and optionally a length. Occlusion detection is repeated periodically in time, or whenever there is a scene update, so that the renderer has up-to-date occlusion information.

[0036] 2, the extent of the audio element 202 may be only partially occluded by an occluding object 206. This means that the rendering of the audio element 202 needs to be changed to reflect which parts of the extent are occluded and which parts are not. The extent 204 may be the actual extent of the audio element 202 as seen from the listener's position, or a projection of the audio element 202 as seen from the listener's position, where the projection may be, for example, a projection of the extent of the audio element onto a sphere around the listener, or a projection of the extent of the audio element onto a plane between the audio element and the listener.

[0037] 1.1. Aspects of Occlusion Detection and Rendering

[0038] The process of detecting occlusions in an extent, from the listener's perspective, typically involves checking the path between the listener's position (the "listening position") and each of a number of points on the extent for occluding objects. Some processing is required, both in the geometry operations involved in ray tracing and in the operations of the audio transmission filters. This means that the number of these paths (i.e., points on the extent) that are checked should be minimized.

[0039] Rendering the occlusion effect of audio objects with extents is perceptually the most important aspect. The human auditory system is very good at producing the angle of an audio object in the horizontal plane, often called azimuth, because it can exploit the timing difference between sound arriving at the right and left ear, respectively. Under ideal circumstances, humans can discern horizontal angle differences of just 1°. For vertical angles, often called elevation, there are no timing differences that can aid the auditory system. Instead, the only cue spatial hearing uses to distinguish different vertical angles is the difference in frequency response resulting from different filtering by the ears for different vertical angles. Therefore, precision in vertical angles is less perceptually important than horizontal angles.

[0040] Even if the vertical position of the top and bottom edges of an extent is not perceptually significant, detecting these edges can affect the overall energy of an audio object: the required resolution due to changes in energy can be higher than the resolution due to perceived changes in spatial position.

[0041] For audio objects with an extent, the outer edges are the most prominent feature, but the energy distribution across that extent also needs to be reflected reasonably well. It is important that any changes in position, energy, or filtering are smooth and do not change in discrete steps unless there is a sudden movement of the audio source, the listener, or some occluder (or any combination thereof).

[0042] The easiest way to avoid stepwise changes in occlusion detection is to use a large number of ray casts so that smooth changes in occlusion can be tracked with high resolution, making the steps small enough to be imperceptible. However, a large number of ray casts significantly increases the complexity of the algorithm.

[0043] Another solution is to add temporal smoothing to the occlusion detection and / or occlusion rendering, which evens out the abrupt steps and makes the occlusion effect behave more smoothly. The drawback to this is that the occlusion detection / rendering response is slower and does not react directly to fast motion. In general, there is a trade-off between resolution and temporal smoothing, in order to make the detection and rendering as fast as possible without producing audible steps.

[0044] 1.2 Two-stage occlusion detection

[0045] To achieve high resolution, the most important aspect of audio source occlusion detection, while minimizing the number of raycasts required, the process can be done in two stages, as follows.

[0046] First, it detects so-called cropping occlusions, which are occlusions that completely block at least one edge of the extent. In the case of cropping occlusions (i.e., when the entire edge of the extent is completely occluded), a modified extent is calculated in which the completely occluded part is discarded.

[0047] Second, using the modified extent with the fully occluded parts discarded, measure the amount of occlusion by sending out a set of raycasts (e.g., an evenly distributed set) and compute occlusion filters representing different subareas of the modified extent.

[0048] The first stage focuses on determining whether an edge of the extent is fully occluded, and if so, determining the corresponding edge of the modified extent (i.e., the occlusion edge). Here, an iterative search algorithm can be used to find the occlusion edge. Because the first stage only detects full occlusion, there is no need to compute an occlusion filter for each raycast. This stage is further described in Section 1.3.

[0049] The second stage operates on the modified extent, with fully occluded portions discarded. However, it is possible that the "modified" extent is actually the same as the extent, rather than a modified version of it (this is explained further below with respect to FIG. 5). In any case, the focus of this stage is to identify occlusions that occur within the so-called modified extent and to compute occlusion filters that correspond to occlusions in different subareas of the modified extent. This stage is further explained in Section 1.4.

[0050] 1.3 Optimized detection of cropping occlusion

[0051] Cropping occlusion detection examines the full occlusion of the edges of an audio object's extent. Because the auditory system is not very good at identifying the exact shape of an audio object, this can be simplified to identifying the width and height of the portion of the extent that is not fully occluded. This can be done using an iterative search algorithm, such as a binary search, to find the points on the extent that represent the highest and lowest horizontal and vertical angles that are not fully occluded. Using an iterative search algorithm allows for the identification of occlusion edges with high accuracy and as few ray casts as possible.

[0052] Together with the modified extent, a total gain factor is calculated that describes the overall gain of the modified extent compared to the original extent. If part of the original extent is occluded, it should be reflected in the total gain of the rendered audio element. The total gain factor g OV is expressed by the following equation:

number

number

[0053] Cropping occlusion detection begins by casting a sparse grid of rays towards the extent to get a first rough estimate of the edge. The points representing the highest and lowest horizontal and vertical angles that are not fully occluded are stored as starting points for an iterative search to find the exact edge.

[0054] 3 shows an example in which extent 304 (in this example, a rectangular extent) of audio element 302 is occluded by occluders 310 and 311. In one embodiment, occlusion detection is performed in two stages. In the first stage, cropping occlusion is detected, and a modified extent 404 (see FIG. 4 ) is determined, which in this example represents a portion of extent 304 (e.g., a rectangular portion of extent 304). That is, in this example, modified extent 404 is smaller than extent 304 because the entire edge 340 (i.e., the left edge) of extent 304 is fully occluded. More specifically, in this example, modified extent 404 has a different left edge than extent 304, but the right edge, top edge, and bottom edge are the same because none of them are fully occluded.

[0055] The ray tracing positions are visualized as black dots in Figure 3. In this example, as shown in Figure 3, the left edge 340 of extent 304 is completely occluded by object 310. Ray tracing point P1 is the point that represents the leftmost point of the unoccluded extent. A binary search between points P1 and P2 can be used to find the occlusion edge 350. This edge 350 is then used as the left edge of the modified extent ("cropped extent") that will be used in the next stage (see, for example, Figure 4). Occluder 311 does not occlude any edges and has no effect on the modified extent.

[0056] In one embodiment, after casting a grid of rays toward the extent, the non-occluded point P1 representing the lowest horizontal angle is stored as the min_azimuth_point. To find the exact edge of the occlusion, a binary search can be used. The binary search uses a lower and upper limit. In this case, the lower limit can be initialized to min_azimuth_point or P1. The upper limit is initialized to a point with a lower azimuth angle (the left side in this example) that is known to be either occluded or on the edge of the extent. In this case, this can be P2. Next, the search begins by evaluating occlusion at a point between the lower and upper limits. If this midpoint is occluded, the upper limit is set to this midpoint. If this midpoint is not occluded, the lower limit is set to this midpoint. This process can then be repeated until the distance between the lower and upper limits is below a certain threshold, or it can be repeated N times, where N is a predefined set value. The midpoint between the upper and lower limits is then used to describe the azimuth angle of the left edge of the modified extent.

[0057] 5 shows an example where none of the edges of extent 304 are fully occluded. Therefore, in this example, the determined modified extent is the same as extent 304.

[0058] Cropping occlusion detection does not detect the exact shape of the occlusion, but only the rectangular portion of the extent that is not fully occluded, as shown in Figure 4. However, this covers many typical cases, such as when an extent is partially obscured by a wall, or when viewing / listening to an audio object through a window. The cropping occlusion stage can be thought of as a way to define a frame around the part of the extent that is not fully occluded. Within this frame, partial or soft occlusion may occur. Outside the cropped extent, no further checks for occlusion need to be made.

[0059] In cases of occlusion where the shape of the occluding object is more complex, or where there is soft occlusion, a second stage is used to describe the effect of occlusion in the correction extent.

[0060] The density of the sparse grid of rays used as the starting point for the edge search does not directly affect the accuracy of edge detection. However, the grid of rays must be dense enough to detect at least one point of the unoccluded extent, which can then be used as the starting point for the iterative search for the edge of the modified extent. There may be situations where the iterative search cannot be performed properly if most of the extent is occluded and only a small portion is unoccluded, and the sparse grid does not identify the unoccluded portion of the extent. Sections 1.5-1.7 provide some examples of how to optimize the sampling grid so that even small unoccluded portions are detected without making the sampling grid very dense.

[0061] 1.4 Optimized detection of occlusions in cropped extents

[0062] The second stage of occlusion detection checks for occlusion within the modified ("cropped") extent, an example of which is shown in Figure 4.

[0063] This is done by dividing the modified extent 404 into one or more subareas and computing an occlusion filter for each subarea of ​​the modified extent, as shown in Figure 4. The occlusion filter for a subarea describes the amount of occlusion in different frequency bands for the subarea.

[0064] Therefore, in one embodiment, the modified extent (i.e., extent 304 or 404) is divided into several sub-areas. The number of sub-areas may vary and be adaptive, for example, depending on the size of the extent. Typically, the number of sub-areas required is related to how the extent will be rendered later. If the rendering is based on virtual speakers and the number of virtual speakers is small, there is little need to have many sub-areas because the rendering is performed using a virtual speaker setup with limited spatial resolution. If the extent is very small, no division is required and only one sub-area is defined, which is equal to the entire modified extent.

[0065] Below are some examples of typical subarea divisions with different numbers of subareas:

[0066] 1 subarea: no division,

[0067] Two sub-areas: left, right,

[0068] Three sub-areas: left, center, right,

[0069] 4 sub-areas: top left, top right, bottom left, bottom right,

[0070] Five sub-areas: top left, top right, center, bottom left, bottom right,

[0071] Six sub-areas: top left, top center, top right, bottom left, bottom center, bottom right.

[0072] The sub-areas do not necessarily have to be the same size, but if they are the same size it simplifies rendering as each sub-area has the same energy contribution.

[0073] For each subarea, a set of rays is cast to obtain an estimate of how much occlusion exists for this particular portion of the correction extent. For each ray cast, an occlusion filter is formed from the acoustic transmission parameters of any materials the ray passed through. The filter can be represented as a list of gain coefficients for different frequency bands. If the ray passes through two or more occluders, the occlusion filter is calculated by multiplying the gains of the different materials in each frequency band. If the ray is fully occluded, the occlusion filter can be set to 0.0 for all frequencies. If the ray does not pass through any occluding objects, the occlusion filter can be counted as having a gain of 1.0 for all frequencies. If a ray does not hit the audio object's extent, it can be treated as a fully occluded ray or simply discarded.

[0074] For each subarea, the occlusion filters for each raycast are accumulated to form one occlusion filter that represents the occlusion in that subarea. The cumulative gain per frequency band for that subarea can then be calculated using, for example,

number

number

[0075] In a specific example, assume two rays are cast toward a subarea of ​​an extent, with the first ray passing through a thin occluding object made of a first material (e.g., cotton) and the second ray passing through a thick occluding object made of a second material (e.g., brick). That is, points within the extent through which the first ray passes are occluded by the thin occluding object, and points within the extent through which the second ray passes are occluded by the thick occluding object. Also assume that each material is associated with a different filter (i.e., a set of frequency ranges and gain coefficients for each frequency range), as shown in the table below: [Table 1]

[0076] Here, G SA,F1 =sqrt((g11+g21) / 2); G SA,F2 =sqrt((g12+g22) / 2); G SA,F3 = sqrt((g13 + g23) / 2). That is, the subarea has three different cumulative gain values ​​(G SA,F1 , G SA,F2 , G SA,F3 )

[0077] The distribution pattern of rays across each subarea should preferably be uniform. The simplest form of a uniform distribution pattern would be a regular grid. However, a regular grid pattern means that many sample points are made at the same horizontal angle and many sample points are made at the same vertical angle. Because many occlusion situations involve occluders with straight vertical or horizontal edges, such as walls, doorways, and windows, this can exacerbate the problem of stepwise behavior. This problem is illustrated in Figures 6A and 6B.

[0078] Figures 6A and 6B show an example of occlusion detection using a uniform grid of 24 rays. An extent 604 is shown as seen from the listening position, and an occluder 610 is moved from left to right to cover more of the extent. In Figure 6A, the occluder 610 occludes 12 rays (rays are shown as black dots). In Figure 6B, the occluder has moved further to the right, where it now occludes 15 rays. As the occluder moves further, the amount of occlusion changes in discrete steps, resulting in audible momentary changes in audio level.

[0079] Instead of using a regular grid as shown in Figures 6A and 6B, some form of random sampling distribution may be used, such as completely random sampling, clustered random sampling, or regular sampling with random offsets. Generally, a good distribution pattern is one in which sample points do not repeat the same vertical or horizontal angle. Such a pattern can be constructed from a regular grid in which increasing offsets are added to the vertical positions of samples within each horizontal row and increasing offsets are added to the horizontal positions of samples within each vertical column. Such a skewed grid pattern distributes the sample points so that the horizontal and vertical positions of all sample points are as evenly distributed as possible. Figures 7A and 7B show an example of a grid with increasing offsets to the horizontal positions of the sample points. As can be seen, when the occluder moves, only one extra ray is occluded. This means that the detection resolution is improved by a factor of three compared to using a uniform grid with the same number of sample points, as shown in Figures 6A and 6B.

[0080] 1.5 Time-varying raycast sampling grid

[0081] The number of raycasts used for the two stages of detection can be adaptive, e.g., so that fewer rays are used if there are many other processes active in the renderer, the number of raycasts can be adapted so that the resolution remains constant regardless of the extent size, or the number of rays can be adapted depending on the current renderer load.

[0082] Another way to vary the number of rays is to use past detection results, so that the resolution is increased for a period of time after an occlusion is detected. In this way, a sparser set of rays can be used to detect whether an occlusion is present, and whenever an occlusion is detected, the resolution of the next update of the occlusion state can be increased. The increased resolution can then be maintained for an extra period of time, as long as some occlusion is still detected.

[0083] Yet another way to vary the raycast sampling grid over time is to use a series of grids that complement each other, so that spatial resolution can be increased by using the cumulative results of two or more successive grids. This means that the results are averaged over a longer time frame, and therefore the response of occlusion detection will be slower, similar to when applying temporal smoothing. One way to overcome this is to use only sequential grids for a period of time when no occlusion was previously detected, and then turn off the sequential grids and use a single sampling grid with high resolution instead when an occlusion is detected. Such sequential grids can be pre-calculated or generated on the fly by adding an offset to one predefined grid.

[0084] 1.6 Reusing ray tracing information from stage 1 in stage 2

[0085] It is possible to reuse ray tracing information from stage 1 in stage 2, where the occlusion filter for each ray cast in stage 1 can be evaluated, stored, and included in the calculation of the cumulative occlusion filter for each sub-area.

[0086] 1.7 Reusing occlusion information from previous occlusion detection updates

[0087] Because scene updates are often smooth, occlusion changes are usually gradual as well. In many cases, information from a previous occlusion detection can be used as a good starting point for the next update. One way to utilize previous detections is to add points from within the correction extent of a past update when performing first-stage detection of cropping occlusion. For example, the center point of the correction extent of a previous occlusion detection update can be added as an additional sample point in the first stage. For example, combining a sparse continuous grid of sample points with extra sample points from the previous correction extent can provide a very efficient way to detect cropping occlusion.

[0088] 8 is a flow chart illustrating a process 800 for rendering an audio element associated with a first extent, according to one embodiment. The first extent may be the actual extent of the audio element as seen from the listener's position, or a projection of the audio element as seen from the listener's position, where the projection may be, for example, a projection of the extent of the audio element onto a sphere around the listener, or a projection of the extent of the audio element onto a plane between the audio element and the listener. International Patent Application Publication No. WO2021180820 describes a technique for projecting audio objects in complex shapes. For example, this publication describes a method for representing an audio object relative to a listening position of a listener in an augmented reality scene, the method including: obtaining first metadata describing a first three-dimensional (3D) shape associated with the audio object; and transforming the obtained first metadata to generate transformed metadata describing a two-dimensional (2D) plane or a one-dimensional (1D) line, the 2D plane or 1D line representing at least a portion of the audio object; and transforming the obtained first metadata to generate the transformed metadata; determining a set of description points; and determining the 2D plane or 1D line using the description points, the 2D plane or 1D line passing through an anchor point. The anchor point may be i) a point on the surface of the 3D shape closest to the listener's listening position in the augmented reality scene, ii) a spatial average of points on or within the 3D shape, or iii) a centroid of a portion of the shape visible to the listener, and the set of description points further includes a first point on the first 3D shape representing a first edge of the first 3D shape relative to the listener's listening position, and a second point on the first 3D shape representing a second edge of the first 3D shape relative to the listener's listening position.

[0089] Process 800 begins at step s802, which involves determining a first point in the first extent that is not fully occluded. This step corresponds to the first stage of the two-stage process described above, and the first point may correspond to point P1 in FIG. 3.

[0090] Step s804 includes determining a second extent (referred to above as a modified extent) for the audio element, where determining includes determining a first edge of the second extent using the first point. This step is also a step of the first stage described above. The first edge of the second extent may be edge 350 if edge 340 is fully occluded as shown in FIG. 3, or edge 340 if edge 340 is not fully occluded as shown in FIG. 5.

[0091] Step s806 includes, after determining the second extent, dividing the second extent into a set of one or more sub-areas including the first sub-area.

[0092] Step s808 includes determining a first gain value for a first sample point of a first sub-area (eg, for a first frequency).

[0093] Step s810 includes rendering the audio element using the first gain value (eg, generating an output signal using the first gain value).

[0094] Usage example

[0095] 9A illustrates an XR system 900 to which embodiments may be applied. The XR system 900 includes speakers 904 and 905 (which may be speakers of headphones worn by a listener) and a display device 910 configured to be worn by the listener. As shown in FIG. 9B, the XR system 910 may include an orientation detector 901, a position detector 902, and a processing device 903 coupled (directly or indirectly) to an audio renderer 951 for generating output audio signals (e.g., a left audio signal 981 for the left speaker and a right audio signal 982 for the right speaker as shown). The audio renderer 951 generates the output signals based on the input audio signals, metadata about the XR scene the listener is experiencing, and information about the listener's position and orientation. The metadata for the XR scene includes metadata for each object and audio element included in the XR scene, and the metadata for an object may include information about the size of the object and the occlusion gain for the object (e.g., the metadata may specify a set of occlusion gains (or a set of occlusion factors from which the occlusion gains may be derived), where each occlusion gain is applicable to a different frequency or frequency range). The audio renderer 951 may be a component of the display device 910 or may be remote to the listener (e.g., the renderer 951 may be implemented in the "cloud").

[0096] The orientation sensing unit 901 is configured to detect changes in the listener's direction and provide information regarding the detected changes to the processing unit 903. In some embodiments, the processing unit 903 determines an absolute orientation (with respect to some coordinate system) taking into account the detected changes in orientation detected by the orientation sensing unit 901. Also, different systems for determining orientation and position may exist, such as systems that use lighthouse trackers (lidars). In one embodiment, the orientation sensing unit 901 may determine an absolute orientation (with respect to a coordinate system) given the detected changes in orientation. In this case, the processing unit 903 may simply multiplex the absolute orientation data from the orientation sensing unit 901 with the position data from the position sensing unit 902. In some embodiments, the orientation sensing unit 901 may comprise one or more accelerometers and / or one or more gyroscopes.

[0097] 10 shows an example implementation of an audio renderer 951 for generating sound for an XR scene. The audio renderer 951 includes a controller 1001 and a signal modifier 1002 for modifying an audio signal 961 (e.g., an audio signal of a multi-channel audio element) based on control information 1010 from the controller 1001. The controller 1001 may be configured to receive one or more parameters and trigger the signal modifier 1002 to perform a modification to the audio signal 961 (e.g., increase or decrease the volume level) based on the received parameters. The received parameters include information 963 regarding the listener's position and / or orientation (e.g., direction and distance to the audio element), metadata 962 regarding the audio element (e.g., audio element 602) in the XR scene, and metadata regarding objects occluding the audio element (in some embodiments, the controller 1001 itself generates the metadata 962). Using the metadata and position / orientation information, the controller 1001 may calculate one or more gain factors (e.g., total gain factors and cumulative gains per sub-area) for audio elements in an XR scene that are at least partially occluded as described above.

[0098] 11 shows an exemplary implementation of a signal modifier 1002 according to one embodiment. The signal modifier 1002 includes a directional mixer 1104, a filter 1106, and a speaker signal generator 1108.

[0099] The directional mixer 1104 receives the audio input 961, which in this example includes pairs of audio signals 1101 and 1102 associated with the audio elements, and generates a set of k virtual speaker signals (VS1, VS2, ..., VSk) based on the audio input and control information 1171. In one embodiment, the signal for each virtual speaker may be derived by appropriate mixing of signals including, for example, the audio input 961. For example, VS1 = α × L + β × R, where L is the input audio signal 1101, R is the input audio signal 1102, and α and β are coefficients that depend, for example, on the position of the listener relative to the audio elements and the position of the virtual speaker to which VS1 corresponds.

[0100] In an example where an audio element is associated with three virtual speakers (SpL, SpC, R), k may be equal to 3 for the audio element, with VS1 corresponding to SpL, VS2 corresponding to SpC, and VS3 corresponding to SpR. The control information 1171 used by the directional mixer to generate the virtual speaker signals may include the position of each virtual speaker relative to the audio element. In some embodiments, when an audio element is occluded, the controller 1001 adjusts the position of one or more of the virtual speakers associated with the audio element and provides the position information to the directional mixer 1104, which then uses the updated position information to generate signals for the virtual speakers (i.e., VS1, VS2, ..., VSk).

[0101] The filter 1106 may adjust the gain of any one or more of the virtual speaker signals based on control information 1172, which may include the above-mentioned cumulative and total gain factors calculated by the controller 1001. That is, for example, when an audio element is at least partially occluded, the controller 1001 may control the filter 1106 to adjust the gain of one or more of the virtual speaker signals by providing one or more gain factors to the filter 1106. For example, when the entire left portion of the audio element is occluded, the controller 1001 provides the control information 1172 to the filter 1106, which causes the filter 1106 to reduce the gain of VS1 by 100% (i.e., the gain factor=0 so that VS1′=0). As another example, if only 50% of the left portion of an audio element is occluded and 0% of the center portion is occluded, then controller 1001 may provide control information 1172 to filter 1106 to cause filter 1106 to reduce the gain of VS1 by 50% (i.e., VS1′=50% VS1) and not reduce the gain of VS2 at all (i.e., gain factor=1 so that VS2′=VS2). As another example, if VS1 is a signal associated with a particular sub-area and the cumulative gain of this sub-area is g SA1 and the total gain is g OV Assuming that, in one embodiment, VS1′=VS1×g SA1 ×g OV is.

[0102] Using the virtual speaker signals VS1', VS2', ..., VSk', speaker signal generator 1108 generates output signals (e.g., output signals 981 and 982) for driving speakers (e.g., headphone speakers or other speakers). In an embodiment in which the speakers are headphone speakers, speaker signal generator 1108 can perform conventional binaural rendering to generate the output signals. In an embodiment in which the speakers are not headphone speakers, speaker signal generator 1108 can perform conventional speaker panning to generate the output signals.

[0103] FIG. 12 is a block diagram of an audio rendering device 1200 according to some embodiments for performing methods disclosed herein (e.g., audio renderer 951 may be implemented using audio rendering device 1200). As shown in FIG. 12, audio rendering device 1200 includes a processing circuit (PC) 1202. Processing circuit (PC) 1202 may include one or more processors (P) 1255 (e.g., a general-purpose microprocessor and / or one or more other processors, such as an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), etc.). Multiple processors may be co-located within a single housing or a single data center, or may be geographically distributed (i.e., device 1200 may be a distributed computing device). Audio rendering device 1200 may include at least one network interface 1248. At least one network interface 1248 may comprise a transmitter (Tx) 1245 and a receiver (Rx) 1247 to enable the device 1200 to transmit and receive data to and from other nodes connected to the network 110 (e.g., an Internet Protocol (IP) network) to which the network interface 1248 is connected (directly or indirectly). In that case, the network interface 1248 may be wirelessly connected to the network 110. The network interface 1248 is connected to an antenna arrangement and to a storage unit (also referred to as a “data storage system”) 1208, which may include one or more non-volatile storage devices and / or one or more volatile storage devices. In embodiments in which the PC 1202 includes a programmable processor, a computer program product (CPP) 1241 may be provided. The CPP 1241 includes a computer-readable medium (CRM) 1242 that stores a computer program (CP) 1243 comprising computer-readable instructions (CRI) 1244. The CRM 1242 may be a non-transitory computer-readable medium such as a magnetic medium (eg, a hard disk), an optical medium, or a memory device (eg, a random access memory, a flash memory).In some embodiments, when CRI 1244 of computer program 1243 is executed by PC 1202, the CRI is configured to cause audio-rendering device 1200 to perform the steps described herein (e.g., steps described herein with reference to flowcharts). In other embodiments, audio-rendering device 1200 may be configured to perform the steps described herein without the need for code. That is, for example, PC 1202 may consist solely of one or more ASICs. Thus, features of the embodiments described herein may be implemented by hardware and / or software.

[0104] Overview of Various Embodiments

[0105] A1. A method 800 for rendering an audio element 302 associated with a first extent 304, comprising: determining a first point (e.g., P1) in the first extent that is not fully occluded in step s802; determining a second extent 304, 404 for the audio element in step s804; wherein determining the second extent includes determining a first edge of the second extent using the first point; after determining the second extent, dividing the second extent into a set of one or more subareas including a first subarea in step s806; determining a first gain value for a first sample point of the first subarea in step s808; and rendering the audio element using the first gain value in step s810.

[0106] A2. The method described in embodiment A1, wherein the step of determining the first edge of the second extent using the first point includes a step of determining whether the first point is on the first edge of the first extent or within a threshold distance of the first edge of the first extent.

[0107] A3. The method described in embodiment A2, wherein the step of determining the first edge of the second extent further includes setting the first edge of the second extent equal to the first edge of the first extent as a result of determining that the first point is on the first edge of the first extent or within the threshold distance of the first edge of the first extent.

[0108] A4. The method described in embodiment A2, wherein the step of determining the first edge of the second extent further includes the steps of determining a third point between the first point and a second point that is fully occluded, and determining whether the third point is fully occluded or using the third point to define the first edge of the second extent.

[0109] A5. The method described in embodiment A4, wherein the step of determining the first edge of the second extent further includes the steps of determining whether the third point is fully occluded, and determining a fourth point between the first point and the third point if it is determined that the third point is fully occluded, or determining a fourth point between the second point and the third point if it is determined that the third point is not fully occluded.

[0110] A6. The method of embodiment A5, wherein determining the first edge of the second extent further comprises using the fourth point to define the first edge of the second extent.

[0111] A7. A method according to any one of embodiments A1 to A6, wherein the step of determining the second extent further includes the steps of determining a fifth point in the first extent that is not fully occluded, and determining a second edge of the second extent using the fifth point.

[0112] A8. The method described in embodiment A7, wherein the step of determining the second edge of the second extent using the fifth point includes a step of determining whether the fifth point is on the second edge of the first extent or within a threshold distance of the second edge of the first extent.

[0113] A9. The method described in embodiment A8, wherein the step of determining the second edge of the second extent further includes a step of setting the second edge of the second extent equal to the second edge of the first extent as a result of determining that the fifth point is on the second edge of the first extent or within the threshold distance of the second edge of the first extent.

[0114] A10. A method according to any one of embodiments A1 to A9, wherein the step of determining the first gain value for the first sample point in the first sub-area includes a step of determining, for an imaginary line extending from the listening position to the first sample point, whether the imaginary line passes through one or more objects.

[0115] A11. The method described in embodiment A10, wherein the virtual line passes through at least a first object, and the step of determining the first gain value for the first sample point of the first sub-area further includes the steps of obtaining first metadata associated with the first object and determining the first gain value using the first metadata.

[0116] A12. The method described in embodiment A11, wherein the virtual line further passes through a second object, and the step of determining the first gain value for the first sample point of the first sub-area further includes the step of obtaining second metadata associated with the second object, and the step of further using the second metadata to determine the first gain value.

[0117] A13. A method according to any one of embodiments A1 to A12, wherein the step of rendering the audio element using the first gain value includes a step of calculating a first cumulative gain value for the first sub-area using the first gain value, and rendering the audio element using the first cumulative gain value.

[0118] A14. The method described in embodiment A13, wherein the step of rendering the audio element using the first cumulative gain value includes the step of modifying an audio signal associated with the first sub-area based on the first cumulative gain value to generate a modified audio signal, and rendering the audio element using the modified audio signal.

[0119] A15. A method according to any one of embodiments A1 to A14, wherein the step of determining the first gain value for the first sample point of the first subarea includes a step of casting a diagonal grid of rays towards the first subarea, one of the rays intersecting the subarea at the first sample point.

[0120] A16. Total gain factor g OV and the step of rendering the audio element using the first gain value further comprises calculating the first gain value and g OV The method of any one of embodiments A1 to A15, comprising rendering the audio element using:

[0121] A17. The first extent has a first area A1, and the second extent has a second area A2, and A2 <A1であり、g OV 17. The system of embodiment 16, wherein the step of calculating includes the step of calculating A2 / A1.

[0122] A18. g OV 18. The system of embodiment 17, wherein the step of calculating further comprises the step of taking the square root of A2 / A1.

[0123] B1. A computer program comprising instructions that, when executed by a processing circuit of an audio rendering device, cause the audio rendering device to perform the method of any one of embodiments A1 to A18.

[0124] B2. A carrier comprising the computer program of embodiment B1, the carrier being one of an electrical signal, an optical signal, a radio signal, and a computer-readable storage medium.

[0125] C1. An audio rendering device configured to perform the method of any one of embodiments A1 to A18.

[0126] C2. The audio rendering device of embodiment C1, wherein the audio rendering device comprises a memory and a processing circuit coupled to the memory.

[0127] While various embodiments have been described herein, it should be understood that they are presented by way of example only, and not limitation. Thus, the breadth and scope of the present disclosure should not be limited by any of the above-described exemplary embodiments. Moreover, any combination of the above-described objects in all possible variations thereof is encompassed by the present disclosure unless otherwise indicated herein or clearly contradicted by context.

[0128] Additionally, while the processes described above and illustrated in the figures are shown as a series of steps, this is done for illustrative purposes only, and as such, some steps may be added, some steps may be omitted, the order of steps may be rearranged, and some steps may be performed in parallel.

[0129] References

[0130] [1] MPEG-H 3D Audio, Clause 8.4.4.7: “Spreading”

[0131] [2] MPEG-H 3D Audio, Clause 18.1: “Element Metadata Preprocessing”

[0132] [3] MPEG-H 3D Audio, Clause 18.11: “Diffuseness Rendering”

[0133] [4] EBU ADM Renderer Tech 3388, Clause 7.3.6: “Divergence”

[0134] [5] EBU ADM Renderer Tech 3388, Clause 7.4: “Decorrelation Filters”

[0135] [6] EBU ADM Renderer Tech 3388, Clause 7.3.7: “Extent Panner”

[0136] [7] Schissler, C., et. al., “Efficient HRTF-based Spatial Audio for Area and Volumetric Sources,” IEEE Transactions on Visualization and Computer Graphics, Vol. 22, No. 4, pp. 1356-1366, April 2016.

[0137] [8] Patent Publication WO2020144062, “Efficient spatially-heterogeneous audio elements for Virtual Reality.”

[0138] [9] Patent Publication WO2022218986, “RENDERING OF OCCLUDED AUDIO ELEMENTS,” (Application No. PCT / EP2022 / 059762)

[0139]

[10] Patent Publication WO2021180820, “RENDERING OF AUDIO OBJECTS WITH A COMPLEX SHAPE”

Claims

1. A method (800) for rendering an audio element (302) associated with a first extent (304), comprising: determining (s802) a first point (P1) in the first extent that is not fully occluded; determining (s804) a second extent (304, 404) for the audio element, wherein determining the second extent includes determining a first edge (350) of the second extent using the first point; After determining the second extent, dividing the second extent into a set of one or more sub-areas including a first sub-area (s806); determining (s808) a first gain value for a first sample point of the first sub-area; Rendering the audio element using the first gain value (s810); A method comprising:

2. 2. The method of claim 1, wherein using the first point to determine the first edge (350) of the second extent comprises determining whether the first point is on or within a threshold distance of the first edge of the first extent.

3. 3. The method of claim 2, wherein determining the first edge of the second extent further comprises setting the first edge of the second extent equal to the first edge of the first extent as a result of determining that the first point is on the first edge of the first extent or within the threshold distance of the first edge of the first extent.

4. The step of determining the first edge of the second extent comprises: determining a third point between said first point and a second point (P2) that is completely occluded; determining whether the third point is completely occluded or using the third point to define the first edge of the second extent; The method of claim 2 further comprising:

5. The step of determining the first edge of the second extent comprises: determining whether the third point is fully occluded; determining a fourth point between the first point and the third point if the third point is determined to be fully occluded; or determining a fourth point between the second point and the third point if the third point is determined to be not fully occluded; The method of claim 4 further comprising:

6. The step of determining the first edge of the second extent comprises: The method of claim 5 further comprising using the fourth point to define the first edge of the second extent.

7. The step of determining the second extent comprises: determining a fifth point in the first extent that is not fully occluded; determining a second edge of the second extent using a fifth point; The method of claim 1 , further comprising:

8. 8. The method of claim 7, wherein determining the second edge of the second extent using the fifth point includes determining whether the fifth point is on the second edge of the first extent or within a threshold distance of the second edge of the first extent.

9. 9. The method of claim 8, wherein determining the second edge of the second extent further comprises setting the second edge of the second extent equal to the second edge of the first extent as a result of determining that the fifth point is on the second edge of the first extent or within the threshold distance of the second edge of the first extent.

10. The step of determining the first gain value for the first sample point of the first sub-area comprises:

10. A method according to any one of claims 1 to 9, comprising determining, for an imaginary line extending from a listening position to the first sample point, whether the imaginary line passes through one or more objects.

11. The virtual line passes through at least a first object, and determining the first gain value for the first sample point of the first sub-area comprises: obtaining first metadata associated with the first object; determining the first gain value using the first metadata; The method of claim 10 further comprising:

12. The virtual line further passes through a second object, and determining the first gain value for the first sample point of the first sub-area comprises: obtaining second metadata associated with the second object; determining the first gain value further using the second metadata; The method of claim 11 further comprising:

13. 13. The method of claim 1, wherein rendering the audio element using the first gain value comprises calculating a first cumulative gain value for the first sub-area using the first gain value, and rendering the audio element using the first cumulative gain value.

14. 14. The method of claim 13, wherein rendering the audio element using the first cumulative gain value comprises modifying an audio signal associated with the first sub-area based on the first cumulative gain value to generate a modified audio signal, and rendering the audio element using the modified audio signal.

15. The step of determining the first gain value for the first sample point of the first sub-area comprises:

15. The method of claim 1, comprising casting a diagonal grid of rays towards the first sub-area, one of the rays intersecting the sub-area at the first sample point.

16. Total gain factor g OV and the step of rendering the audio element using the first gain value further comprises calculating the first gain value and g OV 16. The method of claim 1, comprising rendering the audio element using

17. the first extent has a first area A1; the second extent has a second area A2, where A2<A1; g OV 17. The method of claim 16, wherein the step of calculating comprises the step of calculating A2 / A1.

18. g OV 18. The method of claim 17, wherein the step of calculating further comprises taking the square root of A2 / A1.

19. A computer program (1243) comprising instructions (1244) which, when executed by a processing circuit (1202) of an audio rendering device (1200), cause said audio rendering device to perform the method of any one of claims 1 to 18.

20. 20. A carrier containing the computer program of claim 19, wherein the carrier is one of an electrical signal, an optical signal, a radio signal, and a computer-readable storage medium (1242).

21. An audio rendering device (1200) for rendering an audio element (302) associated with a first extent (304), said audio rendering device (1200) comprising: determining a first non-fully occluded point (P1) in the first extent (s802); determining (s804) a second extent (304, 404) for the audio element, wherein determining the second extent includes determining a first edge (350) of the second extent using the first point; After determining the second extent, dividing the second extent into a set of one or more sub-areas including a first sub-area (s806); determining (s808) a first gain value for a first sample point of the first sub-area; Rendering the audio element using the first gain value (s810); 1. An audio rendering device configured to perform a method comprising:

22. 22. The audio rendering device of claim 21, wherein the audio rendering device is further configured to perform a method according to any one of claims 2 to 18.

Images