Rendering audio objects with complex shapes

Abstract

To provide a method, an apparatus, and a program for simplifying complex descriptions of audio objects so that a plausible impression of the audio objects can be rendered to a listener with a low computational cost.SOLUTION: A method (900) for representing an audio object in an extended reality scene includes obtaining (s902) first metadata describing a first three-dimensional (3D) shape associated with the audio object, and transforming (s904) the obtained first metadata to produce transformed metadata describing a two-dimensional (2D) plane or a one-dimensional (1D) line that represents at least a portion of the audio object.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] Embodiments relating to the rendering of audio objects with complex shapes are disclosed. [Background technology]

[0002] Spatial audio rendering is a process used to present audio within an extended reality (XR) scene (e.g., a virtual reality (VR), augmented reality (AR), or mixed reality (MR) scene) to give the listener the impression that the sound is coming from a physical source within the scene, located at a specific position and possessing a certain size and shape (i.e., extent). The presentation can be done through headphones or other speakers. When the presentation is done through headphones, the process used is called binaural rendering and utilizes spatial cues of human spatial hearing, which allow for the determination of the direction from which the sound is coming. These cues involve inter-aural time delay (ITD), inter-aural level difference (ILD), and / or spectral difference.

[0003] The most common form of spatial audio rendering is based on the concept of point sources, where each sound source is defined as emitting sound from a single specific point. Because each sound source is defined as emitting sound from a single specific point, the sound sources have no size or shape. Different methods have been developed to render sound sources that have range (size and shape).

[0004] One such known method is to create multiple copies of a mono audio object at positions around the audio object. This configuration results in the perception of a spatially uniform object of a certain size. This concept is used, for example, in the "object spread" and "object divergence" features of the MPEG-H 3D audio standard (see references [1] and [2]), and in the "object divergence" feature of the EBU Audio Definition Model (ADM) standard (see reference [4]). This idea using a mono audio source has been further developed, as described in reference [7], where the area-volume geometry of the sound object is projected onto a sphere around the listener, and the sound is rendered to the listener using a pair of HR filters, which are evaluated as the integral of all head-related (HR) filters covering the geometric projection of the object on the sphere. For a sphere volume source, this integral has an analytical solution. However, for any area-volume source geometry, the integral is evaluated by sampling the projected source surface on the sphere using so-called Monte Carlo ray sampling.

[0005] Another rendering method renders a spatially diffuse component in addition to the mono audio signal, resulting in the perception of a somewhat diffuse object that does not have a distinct pinpoint location, in contrast to the original mono audio object. This concept is used, for example, in the "object diffuseness" feature of the MPEG-H 3D audio standard (see reference [3]) and the "object diffuseness" feature of EBU ADM (see reference [5]).

[0006] Combinations of the two methods described above are also known. For example, the "object extent" feature of EBU ADM combines the creation of multiple copies of a mono audio object with the addition of a diffuse component (see reference [6]).

[0007] In many cases, the actual shape of an audio object can be described sufficiently well using a basic shape (e.g., a sphere or a box). However, sometimes the actual shape is more complex and needs to be described in a more detailed form (e.g., a mesh structure or a parametric description format). In such cases, rendering the audio object requires real-time evaluation of how the audio object will be rendered based on the listener's current relative position to the audio object. This real-time evaluation can be processing-intensive. [Overview of the project]

[0008] Existing methods for rendering volumetric audio objects generally use a single point source to represent the audio object, setting the position of that point source as the center of the audio object (see, for example, Figure 1). This means that the height and width of the audio object are not represented. More sophisticated methods exist for rendering volumetric audio objects. Such methods attempt to represent the shape of the audio object, but these methods are limited to a mono source. This results in the audio object being rendered as either spatially uniform or spatially diffused. In either case, the rendering is based on a mono recording without spatial information.

[0009] Another option for rendering volumetric audio objects is to represent them using a number of point sources that together constitute the sound emanating from the audio object. However, this method is highly inefficient due to the number of point sources required for an accurate representation of the object's shape from different angles. Furthermore, the number of point sources must increase proportionally to the size of the object.

[0010] Accordingly, this disclosure supports the use of audio objects of a defined size and shape, represented by a multi-channel audio signal in which different channels represent spatial information of the audio object in different dimensions. An example is a stereo recording representing spatial information along the horizontal dimension of an audio object. More specifically, this disclosure provides a method for simplifying the three-dimensional (3D) shape of an audio object to a one-dimensional (1D) or two-dimensional (2D) representation that describes the width and / or height of the audio object as seen from the listener's position. The multi-channel audio signal is then rendered so that the perceived spatial range matches that of the simplified range.

[0011] Accordingly, in one embodiment, a method is provided for representing an audio object in an extended reality scene. The method includes obtaining first metadata describing a first three-dimensional (3D) shape associated with the audio object. The method also includes transforming the obtained first metadata to produce transformed metadata describing a two-dimensional (2D) plane or a one-dimensional (1D) line. The 2D plane or 1D line represents at least a portion of the audio object.

[0012] In another embodiment, a computer program is provided that, when executed by a processing circuit, comprises instructions causing the processing circuit to perform the method described above. In one embodiment, a carrier containing the computer program is provided, the carrier being one of an electronic signal, an optical signal, a radio signal, and a computer-readable storage medium.

[0013] In another aspect, an apparatus for representing an audio object in an extended reality scene is provided. The apparatus is adapted to obtain first metadata describing a first three-dimensional (3D) shape associated with the audio object. The apparatus is further adapted to transform the obtained first metadata to generate transformed metadata describing a two-dimensional (2D) plane or a one-dimensional (1D) line. The 2D plane or 1D line represents at least a portion of the audio object. In one embodiment, the apparatus comprises processing circuitry and a storage unit storing instructions for configuring the apparatus to perform any of the processes disclosed herein.

[0014] Compared with existing methods for rendering audio objects, the embodiments described herein provide a more efficient way to dynamically represent both the width and height of an audio object when a listener moves around in a 6 degrees of freedom (6DoF) XR scene.

[0015] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate various embodiments. [Brief Description of the Drawings]

[0016] [Figure 1] FIG. 2 is a diagram illustrating an exemplary representation of an audio object. [Figure 2] FIG. 3 is a diagram illustrating an example of a simplified range. [Figure 3] FIGS. 4A and 4B are diagrams illustrating an exemplary process of finding description points. [Figure 4] FIG. 5 is a diagram illustrating an exemplary process of defining a 2D plane. [Figure 5] FIG. 6 is a diagram illustrating a scenario where a very small movement of a listener triggers a jump in the position of an anchor point. [Figure 6] FIGS. 7A to 7C are diagrams illustrating examples of rendering settings. [Figure 7] FIG. 8 is a diagram illustrating a system in accordance with some embodiments. [Figure 8] Figures A and B show the system in several embodiments. [Figure 9] This figure shows the process according to several embodiments. [Figure 10] This figure shows an apparatus according to several embodiments. [Modes for carrying out the invention]

[0017] To render audio objects with complex ranges (i.e., complex shapes) in a 6DoF virtual environment, the object's shape needs to be evaluated in real time as the listener's listening position changes. The object's shape relative to the listener's listening position should affect the apparent position, width, and height of the audio object to match the behavior of a real sound source.

[0018] The level of range detail that needs to be considered when rendering the audio of an audio object without affecting the perceived spatial quality will vary, for example, depending on the distance from the listener's position to the audio object. As the listener moves further away from the audio object, the shape details become less important. This can be leveraged by efficient implementations that gradually simplify the range shape description.

[0019] Generally, volumetric audio objects are described either by a basic shape (e.g., a sphere or a box) or by a more elaborate description based on a combination of basic shapes, mesh structures, or other forms of shape descriptions. To efficiently render these audio objects with complex shape descriptions in real time, it is desirable to simplify the description of the object's shape in a way that preserves the important cues perceived by the auditory system but does not require excessive complexity. Often, such simplification is possible because extremely fine shape details are generally not perceptually important to audio rendering, and therefore, considering these fine shape details in the rendering process would be a waste of computational resources.

[0020] This disclosure describes a method for simplifying the complex description of audio objects so that a plausible impression of the audio object can be rendered to the listener with low computational cost.

[0021] In one embodiment, the method is divided into three basic steps.

[0022] (1) An optional preprocessing step to simplify a complex description of the scope (i.e., shape) of an audio object into a simpler description. For example, the simpler description may be a mesh structure with fewer vertices, a basic shape (e.g., a box, sphere, ellipsoid, or cylinder), or a combination of several basic shapes. This simplified shape description may be used instead of the original description either always, or only, for example, when the distance to the listener's listening position is sufficiently long. This step is described in Section 1.

[0023] (2) A real-time simplification step in which the range is represented using a two-dimensional plane or a one-dimensional line. Here, the listener's instantaneous listening position is taken into consideration. This step is described in Section 2.

[0024] (3) Rendering of the audio object based on the simplified geometric range of the audio object determined in step 2. Additional processing may be added during the rendering stage to add perceptual effects, such as distance gain. This step is described in section 3.

[0025] 1. Simplification of pre-rendering

[0026] The pre-rendering simplification step is performed before rendering begins, or alternatively, as a parallel process while rendering is in progress, but the results are not used until the step is complete. The purpose of this step is to find one or more simplified descriptions of the range that can be used depending on the amount of detail required at a particular moment. Several alternative range descriptions may be stored in a list so that the real-time rendering step can select the most appropriate range description at any given time. Since the pre-rendering simplification step is not performed in real time, it may involve complex calculations and optimizations as needed. This step may even be performed as part of a content creation step, where several alternative representations are pre-calculated. Alternatively, an encoding node or a pre-rendering node may perform this step to offload it to a listener's device.

[0027] The amount of detail required in describing the range needed to achieve high-quality rendering for listeners in an Extended Reality scene depends on many things that can fluctuate in real time, such as i) the distance of the audio object to the listener's listening position in the Extended Reality scene, ii) the size of the audio object, iii) how many other audio objects are currently active, iv) the current audio energy level of the audio object, and v) the current CPU load of the renderer that will be used to render the audio object.

[0028] When the distance is sufficiently large, any audio object can be represented quite well using a simple point source. However, the closer the listener is to the audio object, the more detail is needed to provide a subjectively accurate audio rendering of the audio object.

[0029] By predetermining one or more simplified descriptions of the range, the description that provides the most efficient trade-off between detail and complexity can be selected at any given time. As an example, a complex mesh structure describing the range of audio objects may be simplified into two alternative simplified shapes, one which is still a mesh structure but has a reduced number of vertices, and the second which is just a box shape with dimensions set to match the original shape. An example of such simplification is shown in Figure 2, which shows the complex mesh structure 202, a first less complex version of the complex mesh structure 202 204, and a second less complex version of the complex mesh structure 202 206.

[0030] As shown in Figure 2, the mesh structure 202 has a relatively large number of vertices. In the illustrated example, the mesh structure 202 represents a forest viewed from above. In this example, a first less complex version 204 of the complex mesh structure 202 is itself a mesh structure, but mesh structure 204 has fewer vertices than mesh structure 202, and a second less complex version 206 of the complex mesh structure 202 is a simple box shape used to represent the shape of the forest. That is, in the example of Figure 2, two simplified shapes are created, one created by reducing the number of vertices of the original mesh structure 202, and the other created based on the basic box shape 206. The two alternative shape descriptions 204 and 206 represent different levels of simplification, and depending on the amount of detail required, either of these representations can be selected as the basis for a real-time simplification step. With respect to the box shape 206, the box shape is rotated so that one dimension represents the largest dimension of the forest (i.e., the mesh structure 202), and the width and height of the box are adjusted so that the box fits as closely as possible to the mesh structure 202. Even simple shapes such as spheres or boxes can be simplified to point sources when the distance to the listener is sufficiently far.

[0031] Many methods are known for reducing the number of vertices in a mesh (see, for example, reference [8]). The problem of decomposing an arbitrary shape into one or a few basic shapes, such as a box, is also a well-known problem that can be solved, for example, by using minimal volume bounding box decomposition (see reference [9]). As described herein, several alternative representations are pre-computed, along with a parameterized way of selecting which representation to use at any given time during real-time rendering.

[0032] As a basis for real-time simplification, step 2, the parameterized selection of which of these representations to use can be based on the distance of the audio object from the listener's listening position. An example of this is illustrated in Table 1. TIFF0007936903000001.tif81170

[0033] Here, D1 and D2 are threshold distance parameters that can be either constants or variables, controlled, for example, by the number of currently active audio objects or the current CPU load. The design parameters may depend on the size of the audio objects. For example, the listener's listening position may need to be further away from a large audio object for a single-point representation of a large audio object to be appropriate. In the example shown in Table 1, there are three different representations for the audio object, and two parameters (i.e., D1 and D2) are used to tell the renderer when to switch between the three different representations. Any additional representations, such as the simplified mesh structure 204 representation in Figure 2, would require additional parameters specifying when such representations should be used.

[0034] The following is an example of how D1 and D2 can be selected: D1 = 2S max Therefore, D2 = 20S max And here, S max This is the size of the range in its largest dimension (for example, length, width, height, or depth).

[0035] In some cases (for example, when the computational resources available for rendering fall below a certain lower limit), D1 can be extremely small or even zero (i.e., the original mesh is always simplified to at least the first level of simplification, i.e., to a box shape as in the case of Table 1).

[0036] 2. Real-time simplification step

[0037] A real-time simplification step can be performed as part of the rendering process when the listener's current listening position is known. This step can generally be performed at a uniform rate related to the frame size used by the rest of the renderer (for example, it may be performed each time a new audio frame is received or processed). Alternatively, this step can be triggered only when there is a change in the position or rotation of either the audio object or the listener.

[0038] Since this step is performed in real time, one of the different representations determined in step 1 may be selected according to parameterized selection rules and the listener's current listening position. Other parameters that may be taken into consideration in the selection of one of the representations include the number of active audio objects, the renderer's CPU load, and the current audio energy level of the rendered audio object.

[0039] After one of the representations (i.e., the original description or one of the simplified descriptions) is selected, the selected representation may be further simplified to a two-dimensional (2D) plane or a one-dimensional (1D) line. In this further simplification, the depth of the audio object is not considered, since the depth of an audio object placed in front of the listener of the audio object cannot be subjectively perceived in the same way as the width and height of the audio object.

[0040] When an audio object is placed directly in front of the listener, the perceived horizontal width is primarily determined by the time and level difference cues. However, the perceived vertical height of the object is primarily determined by the less precise spectral cues.

[0041] Therefore, listeners can generally estimate the horizontal width of an audio object more accurately than its vertical height. Perceiving the depth of an audio object is even less accurate, and usually simply impossible. Therefore, the depth dimension of an audio object generally does not need to be included during the rendering of the audio object.

[0042] Therefore, a 2D plane representing both the width and height of an audio object captures the majority of the important spatial cues. Also, in many cases, a 1D linear representation of only the position and horizontal width of an audio object is an efficient representation that captures the most important spatial cues of the original audio object.

[0043] This simplification step simplifies the geometric shape of the range, which does not necessarily directly correspond to the perceived width and height experienced by the listener. These perceptual effects should be handled by a rendering step in which the perceived width and height are calculated from the geometric size and shape of the range.

[0044] 2.1 Finding the points to describe

[0045] The process of finding a two-dimensional plane to represent an audio object begins by finding anchor points on, within, or near a range, as well as the outer edges of the range, as seen from the listener's listening position. Therefore, the following steps may be performed:

[0046] (1) Find an anchor point on, within, or near the range. The two-dimensional plane that will be defined in the following steps will be fitted so that the two-dimensional plane passes through the anchor point. The line between the listener's listening position and this point will act as a reference direction, from which the angle of subsequent points will be calculated.

[0047] (2) Find the range point that represents the right edge of the range from the listener's listening position. This corresponds to the point with the highest azimuth angle relative to the reference direction.

[0048] (3) Find the range point that represents the left edge of the range from the listener's listening position. This corresponds to the point with the lowest azimuth angle relative to the reference direction.

[0049] (4) Find the range point that represents the upper edge of the range from the listener's listening position. This corresponds to the point with the highest elevation angle relative to the reference direction.

[0050] (5) Find the range point that represents the lower edge of the range from the listener's listening position. This corresponds to the point with the lowest elevation angle relative to the reference direction.

[0051] The five points found during this process will be shown as the descriptive points of the range representation. Figure 3A shows an example of the process for finding the descriptive points. That is, Figure 3A shows how the descriptive points are found for a mesh-based range 302. For simplicity, Figure 3A is shown in a bird's-eye view, which means that the elevation axis is not shown. The line 310 between the listener's listening position and the anchor point (A) on the range 302 is used as the basis for angle measurements, where the azimuth is 0 in the direction of the anchor point, increases to the right, and decreases below zero to the left. The right edge (B) and left edge (C) are then found as the points on the range with the largest respective minimum azimuth angles. The upper edge and lower edge are the points with the largest respective minimum elevation angles.

[0052] 2.1.1 Finding Anchor Points

[0053] An anchor point is a point that will act as a base point for the resulting two-dimensional plane or one-dimensional line. The plane or line will be defined such that they traverse this point, and therefore, to a large extent, the anchor point will define the distance and angle to the listener. Different methods for finding anchor points are described below.

[0054] 2.1.1.1 Use the nearest point as the anchor point.

[0055] A simple way to find an anchor point is to use the point on the range that is closest to the listener's listening position. The exact method for finding the closest point on the range will vary depending on the type of representation used. For example, a basic cylindrical shape has no vertices, and therefore the closest point on its surface must be found using a parametric model based on its radius, length, position, and rotation. A box shape has vertices, but the closest point may be somewhere on one of its surfaces, and not exactly on one of its vertices. For some shapes, the closest point is easy to calculate, such as a sphere, but for other shapes, approximations may be necessary, and identifying the exact closest point can be too complex a process.

[0056] For complex mesh shapes, a limited number of points representing both vertices and points on the surface can be evaluated. Then, each of these points is evaluated, and its Euclidean distance D to the listening position is calculated, i.e., D = |P|. k -P L By calculating |, we can identify which point is closest, where P k is the 3D position of point k on the range, and P L This indicates the listener's listening position.

[0057] 2.1.1.2 Alternative forms for determining anchor points

[0058] As an alternative to using the nearest point as the anchor point, a spatial average of points on a range (or, in some embodiments, within a range) may be used to select the anchor point.

[0059] In one particular embodiment, the anchor point is found by determining a portion of the geometric "centroid" of a range that is "visible" from the listener's listening position.

[0060] The “visible” part of a range can be defined by the largest contour on the extent surface of an object that has a direct line of sight from the listener’s current listening position. For example, in the case of a sphere range, the contour would be a circle with the same radius as the sphere, and the “visible” part of the range would be the frontal hemisphere of the sphere range relative to the listener’s listening position.

[0061] For a region of a cube with length L, where one of the cube's perpendicular edges faces the listener, the contour is: The range will be a rectangle the size of TIFF0007936903000002.tif10170, and the visible portion of the range will be half of the cubic surface facing the listener. For arbitrarily shaped ranges, the contour can also have any shape, and the visible portion of the range surface will generally not be simply half of the total surface of the range.

[0062] Once the “visible” portion is determined, the anchor point can be found by finding the geometric centroid of the visible portion (for example, by spatially averaging the position vectors of all points on the visible portion of the range). In the example of a spherical range, the anchor point will therefore be located somewhere between the surface and center of the spherical range, within the range on the line between the listener’s listening position and the center of the range. Thus, in this example, the anchor point is located on the same line from the listener as the nearest point used in the first embodiment, but at a slightly greater distance from the listener. For ranges of arbitrary shape, this is usually not the case; that is, for ranges of arbitrary shape, the anchor point derived as described above and the nearest point will generally not be located on the same line from the listener’s listening position.

[0063] Figure 3B shows the difference between using the nearest point on the ellipse (A) as the anchor point and using the geometric centroid of the visible portion of the ellipse (B) as the anchor point, in the example of an ellipse with a range of 304. Note that the visible portion of the ellipse is indicated in the figure by the thick line segment.

[0064] If the range is defined in a separate manner (for example, through a set of mesh vertices) and the distribution of points (as seen from the listener's listening position) is not uniform, this non-uniformity of the spatial sampling of the range surface can be accounted for by appropriate weighting of the position vectors in the spatial averaging procedure used to determine the anchor points.

[0065] In other alternative embodiments, the anchor point may be determined as the geometric centroid of the entire spatial range, as opposed to only the “visible” portion of the range. This results in an anchor point located at the spatial centroid of the volume enclosed by the range. In a simple example of a spherical range, the anchor point would be located at the center of the sphere. Figure 3B also shows this alternative anchor point (C) for an exemplary elliptical range. While this alternative embodiment may be suitable for smaller volumetric sound sources, embodiments using the “visible” portion of the range may be more suitable for larger volumetric sound sources.

[0066] In another alternative embodiment, the anchor point is determined as the geometric centroid of a limited area within a range around the nearest point determined according to the method described in Section 2.1.1.1.

[0067] The advantage of the alternative embodiments described in this section is that the position of the anchor point changes more smoothly with changes in the listener's listening position and / or object position / orientation, especially in the range of irregular shapes, than when the nearest point is used as the anchor point. As a result, the position and orientation of the adapted 2D plane also change more smoothly with changes in the listener's position and / or object position / orientation. In particular, these alternative embodiments do not exhibit the problem of the adapted 2D plane changing abruptly due to discontinuous jumps of anchor points, as will be described in more detail below. This advantage comes at the cost of somewhat more complex calculations of the anchor point positions, on the other hand, the embodiments may not require the smoothing process described below in Section 2.3.

[0068] 2.1.2 Finding description points based on anchor points

[0069] Assuming anchor points exist, numerous methodologies can be used to determine other description points, one of which is described below, for example, using a mesh shape.

[0070] The right edge and the left edge of the range are the azimuth angle φ of the vector between the listening position and the point to be evaluated relative to the vector between the listening position and the anchor point of the range, that is, the azimuth angle TIFF0007936903000003.tif15170 can be found by evaluating, where v k is the vector between the listening position and point k on the range, and v c is the vector between the listener's listening position and the anchor point of the range, and both vectors have an elevation angle component set to 0. In this example, as shown in FIG. 3A, the azimuth angle is defined to be 0 in the direction of the anchor point, increases toward the right, and decreases toward the left.

[0071] Similarly, the upper edge and the lower edge are found by evaluating the elevation angle of the vector from the listening position and the point to be evaluated relative to the horizontal plane of the scene. For a range v with coordinates (x, y, z) relative to the listening position k The elevation angle θ of the upper point k is the elevation angle TIFF0007936903000004.tif13164 can be calculated as, where the coordinate system is defined such that the z component represents the upward direction in the scene.

[0072] 2.2 Defining a two-dimensional plane or one-dimensional line representation

[0073] After determining the description points, the next step may include defining a two-dimensional plane. Figure 4 shows an example of the process. For simplicity, Figure 4 is shown in a bird's-eye view, which means that the elevation axis is not shown. As shown in Figure 4, plane 402 is defined such that plane 402 crosses the anchor point at a horizontal angle α that coincides with the angle between the line from the listener's listening position to the anchor point (A) of range 302 and the line from the right edge (B) to the left edge (C) of the range. The outer edges of the plane should coincide with the lowest and highest azimuth angles of each outer edge of the range. The same technique is used when calculating the height and vertical inclination of the plane using the upper and lower description points as the basis.

[0074] In another embodiment, a simplified one-dimensional representation is used, where the height side of the plane is ignored, and the representation is a line passing through the anchor point at a horizontal angle α that matches the angle between the line (A) from the listener's listening position to the anchor point and the line from the right edge (B) to the left edge (C) of the range. The ends of the line should coincide with the lowest and highest azimuth angles of each outer edge of the range. In this simplified representation, upper edge description points and lower edge description points are not required.

[0075] If the range representation selected from the simplified representation derived in the pre-rendering simplification step is a point source, the calculation of the 2D plane or 1D line representation can be bypassed.

[0076] Alternatively, the two-dimensional plane can be adapted as passing through the anchor point and oriented perpendicular to the line from the listener's listening position to the anchor point. In this case, the edge point described above is still used to determine the "opening angle" of the range (as seen from the listener's listening position) and its two-dimensional representation, but the orientation of the 2D plane is always perpendicular to the line from the listener to the anchor point. This representation may be preferable for some range shapes.

[0077] 2.3 Movement Synchronization Smoothing of Range Descriptions

[0078] One potentially important aspect of real-time simplification is that changes in position and range may need to be smooth so that listeners do not perceive unexpected abrupt changes in the spatial representation of audio objects.

[0079] When a listener or audio object moves around or rotates, the description points of the range representation sometimes jump from one point to another. This can also occur when switching between different representations defined in the pre-rendering simplification step. Such abrupt jumps need to be smoothed to avoid generating unnatural audio artifacts. However, normal temporal smoothing is not preferable here because it can make the audio object appear to be moving even when it is not.

[0080] A better approach is to use smoothing synchronized with the relative movement of the listener and audio object. This means that if the listener moves, or if the audio object moves or rotates, only the change in the range description is updated. This avoids situations that can occur with temporal smoothing, such as when the range of the audio object changes while the listener is stationary, or when very small listener movements result in large changes in the position or shape of the audio object.

[0081] Figure 5 shows an example where a very small movement of the listener triggers a jump in the anchor point's position. Changes in the description points can be synchronized with this change by measuring the change in the relative distance between the listener and the center position of the audio object. This is done by applying constraints to the changes in the description points so that they never move more than the change in relative distance. In the example in Figure 5, the nearest point is used as the anchor point. If the listener moves, t n-1 From the position at t n When moving to a position within the range, the nearest point in the range makes a sharp jump from one edge to another (i.e., from the "previous anchor point" to the "target anchor point"). In this case, a very small change in listening position creates a large difference in the range's position, which is perceived as unnatural. This problem cannot be solved by using time smoothing techniques. Smoothing is applied to the change in relative distance dD between the listener and the audio object. R By basing it on this, more natural behavior can be achieved. In this example, only the listener is moving, and therefore the change in relative position is the same as the change in the listener's position. Smoothing moves the anchor point toward the target position, but the change in relative distance dD R This will limit movement to a specific distance.

[0082] Listener rotation does not affect the range description, but audio object rotation can affect it. Therefore, when limiting changes in the range description, audio object rotation should also be taken into consideration.

[0083] Next, a specific example of smoothing proportional to the change in relative position is described. D R This represents the relative distance between the audio object and the listener's listening position, and dD R Let this be the delta change of the relative distance. dD R is dD R =|(p AEn -p Ln )|-|(p AEn-1 -p Ln-1 )| is calculated as, where p AEn and p Ln is the 3D position of the audio object and listener at time n, and p AEn-1 and p Ln-1 This is the corresponding position from the last point in time.

[0084] The five different description points of the range representation all have a maximum change in the position of each point of dD. R It should be interpolated / smoothed so as to be limited to dD. This is done by first determining the delta change of each point, dD vSmooth =min(dD v ,αdD R This can be done by calculating as dD v This indicates the distance between the old location of the description point and the new target point, dD vSmooth represents the smoothed distance at which the description point is updated, and α is the update rate and dD. R This is a design constant that controls the relationship between [the two factors]. In most cases, a value between 1.0 and 2.0 is preferred.

[0085] Next, the points to describe are It was updated as TIFF0007936903000005.tif14170, and here, p pnis the target description point, p pn-1 is the previous description point. Therefore, the change in the description point occurs in the direction of the new target description point, but the distance is limited to being at most the same magnitude as the change in the relative position between the listener and the audio object. See Figure 5, for example.

[0086] To include the rotation of the audio object as part of the smoothing process, use dD R An additional term may be added to the calculation, where the rotation is calculated as a circular movement centered on the midpoint of the range in the radius of the anchor point. Then dD R However, dD R =|(p AE1 -p L1 )|-|(p AE1 -p L1 It is calculated as ||dθr|, where dθ is the change in rotation since the last iteration of the method, and r is the distance from the center point of the range to the anchor point at the listener's listening position. To capture the rotation in all directions, dθ can be calculated as the sum of the rotations around all axes.

[0087] 3. Rendering

[0088] The two previously described steps were performed to simplify the three-dimensional geometric shape of the audio object to a two-dimensional plane or one-dimensional line. The rendering step uses this simplified description of the geometric shape and employs spatial rendering techniques to render the audio object to the listener. That is, the rendering step uses the simplified description along with spatial rendering techniques to render the audio signal (e.g., a multi-channel audio signal) representing the audio object. Different spatial rendering techniques may be used depending on the speaker setup used; for example, binaural rendering is generally used for headphones, and speaker panning techniques are generally used for other speaker setups. In some cases, rendering is performed in an intermediate format, such as ambisonic or some specific multi-channel audio format.

[0089] To convey spatial information from a multi-channel audio object, different channels should be rendered so that they represent the object's intended dimensions. For a stereo audio object, two channels should generally represent spatial information in the horizontal plane and therefore be rendered as the left and right sides of the object. This can be done by using two sources, each representing the right and left sides of a simplified range. In the simplest embodiment, these may be point sources, but in more advanced implementations, these sources can use head-related transfer functions (HRTFs) that better describe the range in a two-dimensional plane. For a multi-channel source that includes a channel describing spatial information in the elevation dimension, more sources are required to represent the elevation dimension as well.

[0090] Examples of the rendering settings described above are shown in Figures 6A, 6B, and 6C. These figures illustrate three different ways of rendering an audio object represented by a two-dimensional plane using multiple virtual sound sources. In the example in Figure 6A, two point sources are used to render an audio object with a stereo audio signal. In the example in Figure 6B, two virtual sound sources with associated HRTFs representing sub-areas of the two-dimensional plane are used to render an audio object with a stereo audio signal. More specifically, Figure 6B shows how the two-dimensional plane can be divided into segments that together cover the entire two-dimensional plane, and each segment can then be rendered through a virtual sound source with an associated HRTF representing that entire segment. In the example in Figure 6C, four virtual point sources are used to render an audio object with a four-channel audio signal, where different channels represent spatial information in both the horizontal and vertical planes.

[0091] For example, to achieve a perceptually correct spatial distribution across a plane, additional processing of the virtual sound source signals before binaural rendering may be required, using techniques such as mid / side mixing. If a shape is relatively long in one dimension, it may be rendered as a line source, taking into account the specific behavior of line sources.

[0092] 4. Exemplary Implementation Forms

[0093] Figure 7 shows an exemplary system 700 (also known as a renderer) for creating sound for an XR scene. System 700 includes a controller 701, a signal modifier 702 for modifying an audio signal 751 (e.g., a multi-channel audio signal), a left speaker 704, and a right speaker 705. One audio signal and two speakers are shown in Figure 7, but this is for illustrative purposes only and does not limit any embodiments of the present disclosure in any way.

[0094] The controller 701 may be configured to receive one or more parameters and trigger a modifier 702 to perform modifications on the audio signal 751 based on the received parameters (for example, increasing or decreasing the volume level). The received parameters include (1) information 753 about the listener's position (for example, direction and distance to the audio source), and (2) metadata 754 about the audio object, such as converted metadata describing a 2D plane or 1D line as described herein (in some embodiments, the controller 701 itself generates the metadata 754).

[0095] In some embodiments of this disclosure, information 753 may be provided from one or more sensors included in the XR system 800 shown in Figure 8A. As shown in Figure 8A, the XR system 800 is configured to be mounted by a listener. As shown in Figure 8B, the XR system 800 may comprise an orientation detection unit 801, a position detection unit 802, and a processing unit 803 coupled to a controller 851 of the system 800. The orientation detection unit 801 is configured to detect changes in the listener's orientation and provide the processing unit 803 with information about the detected changes. In some embodiments, the processing unit 803 determines the absolute orientation (with respect to some coordinate system) based on the detected changes in orientation detected by the orientation detection unit 801. There may also be systems using different systems for determining orientation and position, such as a lighthouse tracker (lidar). In one embodiment, the orientation detection unit 801 may determine the absolute orientation (with respect to some coordinate system) based on the detected changes in orientation. In this case, the processing unit 803 can simply multiplex the absolute orientation data from the orientation detection unit 801 and the absolute position data from the position detection unit 802. In some embodiments, the orientation detection unit 801 may include one or more accelerometers and / or one or more gyroscopes.

[0096] Figure 9 is a flowchart of process 900 for representing audio objects in an extended reality scene. Process 900 may begin with step s902.

[0097] Step s902 includes obtaining first metadata that describes a first three-dimensional (3D) shape associated with the audio object.

[0098] Step s904 involves transforming the acquired first metadata to produce transformed metadata that describes a two-dimensional (2D) plane or a one-dimensional (1D) line. The 2D plane or 1D line represents at least a portion of the audio object.

[0099] In some embodiments, obtaining the first metadata involves selecting the first metadata from a set of metadata comprising the first metadata and the second metadata describing a second 3D shape associated with an audio object, wherein the second 3D shape described by the second metadata is one of the following forms: a mesh structure, a box shape, a sphere shape, an ellipsoid shape, or a cylindrical shape, having fewer vertices than the first 3D shape described by the first metadata.

[0100] In some embodiments, the metadata set further comprises a third set of metadata describing a single point in an extended reality scene.

[0101] In some embodiments, the selection is based on one or more of the following: the distance between the audio object and the listener's listening position in the extended reality scene and at least one threshold distance parameter (e.g., D1 or D2), the size of the audio object, the number of currently active audio objects in the extended reality scene, the current load of the renderer that will be used to render the audio object, or the current audio energy level of the audio object.

[0102] In some embodiments, transforming the acquired first metadata to produce transformed metadata includes determining a set of description points such that the set of description points comprises anchor points, and determining a 2D plane or 1D line using the description points such that the 2D plane or 1D line passes through the anchor points. For many 3D shapes, the anchor points are within or on the 3D shape, but for some other 3D shapes (e.g., concave shapes), the anchor points are not on or within the 3D shape, but are usually close to the 3D shape.

[0103] In some embodiments, the anchor point is a point on the surface of a 3D shape that is closest to the listener's listening position in the extended reality scene.

[0104] In some embodiments, the anchor point is the spatial average of points on or within a 3D shape.

[0105] In some embodiments, a portion of the shape is visible to the listener in the extended reality scene, and the anchor point is the centroid of the portion of the shape that is visible to the listener.

[0106] In some embodiments, the set of description points further comprises a first point on the first 3D shape representing a first edge of the first 3D shape with respect to the listener's listening position, and a second point on the first 3D shape representing a second edge of the first 3D shape with respect to the listening position.

[0107] In some embodiments, the method also includes determining the dimensions (e.g., width, height, length) of a 2D plane or a 1D line based on a first point and a second point.

[0108] In some embodiments, the method further includes determining the horizontal angle of a 2D plane or a 1D line based on a first point and a second point.

[0109] In some embodiments, the set of description points further comprises a third point on the first 3D shape representing a third edge of the first 3D shape with respect to a listening position, and a fourth point on the first 3D shape representing a fourth edge of the first 3D shape with respect to a listening position.

[0110] In some embodiments, the method also includes determining the dimensions (e.g., width, height, length) of a 2D plane or 1D line based on a third point and a fourth point.

[0111] In some embodiments, the method further includes determining the vertical angles of a 2D plane or a 1D line based on a third point and a fourth point.

[0112] In some embodiments, the audio object is represented by a multi-channel audio signal, and the method further includes rendering the multi-channel audio signal using the transformed metadata.

[0113] Figure 10 is a block diagram of the apparatus 1000 according to several embodiments for carrying out the methods disclosed herein (for example, the renderer 700 may be implemented using the apparatus 1000). As shown in Figure 10, the apparatus 1000 comprises a processing circuit (PC) 1002 which may include one or more processors (P) 1055 (for example, one or more other processors such as general-purpose microprocessors and / or application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), etc.), wherein the processors may be co-site in a single housing or in a single data center or geographically distributed (i.e., the apparatus 1000 may be a distributed computing device), and at least one network interface 1048, wherein the apparatus 1000 is connected to the network interface 1048 The PC may include at least one network interface 1048 and a storage unit (also known as a “data storage system”) 1008 which may include one or more non-volatile storage devices and / or one or more volatile storage devices. In embodiments in which the PC 1002 includes a programmable processor, a computer program product (CPP) 1041 may be provided. The CPP 1041 includes a computer-readable medium (CRM) 1042 which stores a computer program (CP) 1043 comprising computer-readable instructions (CRI) 1044. CRM1042 can be a non-temporary computer-readable medium, such as a magnetic medium (e.g., a hard disk), an optical medium, or a memory device (e.g., random access memory, flash memory).In some embodiments, the CRI 1044 of the computer program 1043, when executed by the PC 1002, is configured to cause the device 1000 to perform the steps described herein (for example, the steps described herein with reference to a flowchart). In other embodiments, the device 1000 may be configured to perform the steps described herein without the need for code. That is, for example, the PC 1002 may consist of simply one or more ASICs. Thus, the features of the embodiments described herein may be implemented in hardware and / or software.

[0114] 5. Overview of Various Embodiments

[0115] A1. A method (900) for representing an audio object relative to a listener's listening position in an extended reality scene, the method comprising: obtaining first metadata describing a first three-dimensional (3D) shape associated with the audio object (s902); and transforming the obtained first metadata (s904) to produce transformed metadata describing a two-dimensional (2D) plane or a one-dimensional (1D) line, wherein the 2D plane or 1D line represents at least a portion of the audio object.

[0116] A2a. The method of Embodiment A1, wherein obtaining the first metadata includes selecting the first metadata from a set of metadata comprising the first metadata and the second metadata describing a second 3D shape associated with an audio object (in one embodiment, the second 3D shape described by the second metadata has fewer vertices than the first 3D shape described by the first metadata).

[0117] A2b. The method according to embodiment A2a, wherein the second 3D shape is one of the following forms: a mesh structure, a box shape, a sphere shape, an ellipsoid shape, or a cylindrical shape, having fewer vertices than the first 3D shape described by the first metadata.

[0118] A3. The method according to embodiment A2a or A2b, wherein the set of metadata further comprises a third set of metadata describing a single point in an extended reality scene.

[0119] A4. The method according to Embodiments A2a, A2b, or A3, wherein the selection is based on one or more of the following: the distance between the audio object and the listener's listening position in the Extended Reality scene and at least one threshold distance parameter, the size of the audio object, the number of currently active audio objects in the Extended Reality scene, the current load of the renderer that will be used to render the audio object, or the current audio energy level of the audio object.

[0120] A5. The method according to any one of Embodiments A1 to A4, wherein transforming the acquired first metadata to produce transformed metadata includes determining a set of description points, wherein the set of description points comprises anchor points, and determining a 2D plane or 1D line using the description points, wherein the 2D plane or 1D line passes through the anchor points.

[0121] A6. The method according to Embodiment A5, wherein the anchor point is a point on the surface of a 3D shape that is closest to the listener's listening position in the Extended Reality Scene.

[0122] A7. The method according to Embodiment A5, wherein the anchor point is the spatial average of points on or within a 3D shape.

[0123] A8. The method according to Embodiment A5, wherein a portion of the shape is visible to the listener in an extended reality scene, and the anchor point is the centroid of the portion of the shape visible to the listener.

[0124] A9. The method according to any one of embodiments A5 to A8, wherein the set of description points further comprises a first point on the first 3D shape representing a first edge of the first 3D shape with respect to the listener's listening position, and a second point on the first 3D shape representing a second edge of the first 3D shape with respect to the listening position.

[0125] A10. The method according to Embodiment A9, further comprising determining the dimensions (e.g., width, height, length) of a 2D plane or a 1D line based on a first point and a second point.

[0126] A11. The method according to embodiment A9 or A10, further comprising determining the horizontal angle of a 2D plane or a 1D line based on a first point and a second point.

[0127] A12. The method according to any one of embodiments A5 to A11, wherein the set of description points further comprises a third point on the first 3D shape representing a third edge of the first 3D shape with respect to a listening position, and a fourth point on the first 3D shape representing a fourth edge of the first 3D shape with respect to a listening position.

[0128] A13. The method according to Embodiment A12, further comprising determining the dimensions (e.g., width, height, length) of a 2D plane or a 1D line based on the third and fourth points.

[0129] A14. The method according to embodiment A12 or A13, further comprising determining vertex angles of a 2D plane based on a third point and a fourth point.

[0130] A15. The method according to any one of embodiments A5 to A14, wherein the position of at least one description point is smoothed using a technique such that the magnitude of the positional change between two time instances is at most proportional to the magnitude of the change in the relative distance between the audio object and the listener between the same two time instances.

[0131] A16. The method according to Embodiment A16, wherein changes in the rotation of the range of the audio object are taken into account by calculating the expected position changes of the anchor points, and the position changes of the description points are limited to at most the sum of the magnitude of this position change and the magnitude of the change in the relative position between the audio object and the listener.

[0132] A17. The method according to any one of Embodiments A1 to A16, wherein the audio object is represented by a multi-channel audio signal, and the method further comprises rendering the multi-channel audio signal using the converted metadata.

[0133] A18. The method according to embodiment A17, wherein a multichannel audio signal is rendered using a virtual sound source representing a 2D shape or a 1D line edge.

[0134] A19. The method according to Embodiment A17 or A18, wherein a multichannel audio signal is rendered using a virtual sound source having associated head-related transfer functions (HRTFs) that represent sub-areas of 2D shape or 1D line.

[0135] B1. A device (1000) for representing an audio object in an extended reality scene, wherein the device is configured to acquire (s902) first metadata describing a first three-dimensional (3D) shape associated with the audio object, and to transform (s904) the acquired first metadata to produce transformed metadata describing a two-dimensional (2D) plane or a one-dimensional (1D) line, the 2D plane or 1D line representing at least a portion of the audio object.

[0136] B2. The apparatus according to Embodiment B1, wherein the apparatus is further configured to carry out the method described in any one of Embodiments A2 to A19.

[0137] B3. A device (1000) for representing an audio object in an extended reality scene, the device comprising a memory unit (1008) and a processing circuit (1002) coupled to the memory unit, the device being configured to acquire (s902) first metadata describing a first three-dimensional (3D) shape associated with an audio object, and to transform (s904) the acquired first metadata to produce transformed metadata describing a two-dimensional (2D) plane or a one-dimensional (1D) line, wherein the 2D plane or 1D line represents at least a portion of the audio object.

[0138] B4. The apparatus according to Embodiment B3, wherein the memory unit (1008) includes a memory (1042) for storing instructions for configuring the apparatus to carry out the method described in any one of Embodiments A1 to A19.

[0139] C1. A computer program (1043) that, when executed by a processing circuit (1002), provides instructions to cause the processing circuit to carry out the method described in any one of embodiments A1 to A19.

[0140] C2. A carrier comprising the computer program described in Embodiment C1, wherein the carrier is one of an electronic signal, an optical signal, a radio signal, or a computer-readable storage medium (1042).

[0141] While various embodiments have been described herein, it should be understood that these embodiments are presented only as examples and not as limitations. Therefore, the breadth and scope of this disclosure should not be limited by any of the exemplary embodiments described above. Furthermore, unless otherwise indicated herein or expressly rejected by the context, any combination of all possible variations of the objects described above is encompassed by this disclosure.

[0142] Furthermore, the process described above and shown in the diagrams is presented as a sequence of steps, but this is done for illustrative purposes only. Therefore, it is intended that some steps may be added, some steps may be omitted, the order of the steps may be rearranged, and some steps may be performed in parallel.

[0143] References

[0144] [1]MPEG-H 3D Audio,Clause 8.4.4.7:“Spreading”

[0145] [2]MPEG-H 3D Audio,Clause 18.1:“Element Metadata Preprocessing”

[0146] [3]MPEG-H 3D Audio,Clause 18.11:“Diffuseness Rendering”

[0147] [4]EBU ADM Renderer Tech 3388,Clause 7.3.6:“Divergence”

[0148] [5]EBU ADM Renderer Tech 3388,Clause 7.4:“Decorrelation Filters”

[0149] [6]EBU ADM Renderer Tech 3388,Clause 7.3.7:“Extent Panner”

[0150] [7]Efficient HRTF-based Spatial Audio for Area and Volumetric Sources“, IEEE Transactions on Visualization and Computer Graphics 22(4):1-1 January 2016

[0151] [8] "Surface Simplification Using Quadric Error Metrics", ACM SIGGRAPH Computer Graphics, July 1997

[0152] [9] "Minimum volume bounding box decomposition for shape approximation in robot grasping", 2008 IEEE International Conference on Robotics and Automation.

Claims

1. A method for representing an audio object relative to the listener's listening position in an extended reality scene, wherein the method is: Obtaining first metadata that describes a first three-dimensional (3D) shape associated with the audio object, Transforming the acquired first metadata to produce transformed metadata describing a two-dimensional (2D) plane or a one-dimensional (1D) line, the acquired first metadata describes a first three-dimensional shape related to the audio object, The 2D plane or the 1D line represents at least a portion of the audio object, To produce the converted metadata, converting the acquired first metadata includes determining an anchor point, where the anchor point is a point on the first 3D shape that is considered to be closest to the listener's listening position in the extended reality scene. The 2D plane or the 1D line passes through the anchor point, To produce the converted metadata, the acquired first metadata is further converted. Determining a first point on the first 3D shape that represents the first edge of the first 3D shape with respect to the listening position of the listener, Determining a second point on the first 3D shape that represents the second edge of the first 3D shape with respect to the listening position of the listener, This includes determining a first characteristic of the 2D plane or the 1D line based on the first point and the second point, The aforementioned method further Before determining the first point on the first 3D shape and the second point on the first 3D shape, determine the line between the listening position of the listener and the anchor point. The method includes using the determined line to determine the first point on the first 3D shape and the second point on the first 3D shape, The first point on the first 3D shape has the highest azimuth angle with respect to the line between the listener's listening position and the anchor point. A method wherein the second point on the first 3D shape has the lowest azimuth angle with respect to the line between the listening position of the listener and the anchor point.

2. Obtaining the first metadata includes selecting the first metadata from a set of metadata comprising the first metadata describing the first 3D shape associated with the audio object and the second metadata describing the second 3D shape associated with the audio object, wherein the second 3D shape is different from the first 3D shape. The method according to claim 1.

3. The second 3D shape is A mesh structure having fewer vertices than the first 3D shape described by the first metadata, Box shape, spherical shape, Ellipsoidal shape, Cylindrical shape The method according to claim 2, which is one of the embodiments thereof.

4. The aforementioned selection is, The distance between the audio object and the listener's listening position in the extended reality scene, and at least one threshold distance parameter, The size of the aforementioned audio object, The number of currently active audio objects in the aforementioned extended reality scene, The current load on the renderer that will be used to render the aforementioned audio object, or Current audio energy level of the aforementioned audio object The method according to claim 2, based on one or more of the above.

5. The method according to claim 1, wherein determining the first characteristic of the 2D plane or the 1D line based on the first point and the second point includes determining the first dimension of the 2D plane or the 1D line based on the first point and the second point.

6. In order to produce the converted metadata, the acquired first metadata is further converted, Determining a third point on the first 3D shape that represents a third edge of the first 3D shape with respect to the listening position of the listener, Determining a fourth point on the first 3D shape that represents the fourth edge of the first 3D shape with respect to the listening position, The method according to claim 5, comprising determining a second dimension of the 2D plane or the 1D line based on the third point and the fourth point.

7. The method according to claim 1, wherein determining the first characteristic of the 2D plane or the 1D line based on the first point and the second point includes determining the horizontal angle of the 2D plane or the 1D line based on the first point and the second point.

8. To produce the converted metadata, the acquired first metadata is further converted, Determining a third point on the first 3D shape that represents a third edge of the first 3D shape with respect to the listening position of the listener, Determining a fourth point on the first 3D shape that represents the fourth edge of the first 3D shape with respect to the listening position, The method according to claim 7, comprising determining the perpendicular angle of the 2D plane or the 1D line based on the third point and the fourth point.

9. The method according to claim 1, wherein the position of the first point is smoothed using a technique that limits the magnitude of the change in position between two time instances to at most the magnitude of the change in the relative distance between the audio object and the listener between the same two time instances.

10. The method according to claim 9, wherein the change in rotation of the range of the audio object is taken into account by calculating the expected change in position of the anchor point, and the change in position of the first point is limited to at most the sum of the magnitude of the expected change in position of the anchor point and the magnitude of the change in the relative position between the audio object and the listener.

11. The method according to claim 1, wherein the audio object is represented by a multi-channel audio signal, and the method further comprises rendering the multi-channel audio signal using the converted metadata.

12. The method according to claim 11, wherein the multi-channel audio signal is rendered using a virtual source representing the edges of the 2D plane or the 1D line.

13. The method according to claim 11, wherein the multichannel audio signal is rendered using a virtual sound source having associated head-related transfer functions (HRTFs) representing sub-areas of the 2D plane or the 1D line.

14. A computer program that, when executed by a processing circuit, includes an instruction to cause the processing circuit to carry out the method described in any one of claims 1 to 13.

15. A device for representing audio objects in an extended reality scene, wherein the device is Memory unit and A processing circuit coupled to the memory unit and An apparatus comprising, wherein the apparatus is configured to carry out the method described in any one of claims 1 to 13.

Images