Spatial rendering of audio elements with ranges

Abstract

A method for rendering an audio element (e.g., a spatially heterogeneous audio element), where the audio element has a range and is represented using a set of virtual speakers including intermediate virtual speakers. The method includes selecting a position of an intermediate virtual speaker and / or calculating an attenuation factor of the intermediate virtual speaker based on a position of the listener.

Description

Description of the case

[0001] This application is a divisional application of the invention patent application with application date of October 11, 2022, application number 202280067969.3, and invention name “Spatial rendering of audio elements with range”. Technical Field

[0002] Embodiments related to rendering of audio elements are disclosed. Background Art

[0003] Spatial audio rendering is a process for presenting audio within an extended reality (XR) scene (e.g., a virtual reality (VR), augmented reality (AR), or mixed reality (MR) scene) so as to give the listener the impression that the sound is coming from a physical source located at a specific location and of a specific size and shape (i.e., range) within the scene. The rendering can be done through headphone speakers or other speakers. If the rendering is done via headphone speakers, the process used is called binaural rendering and uses spatial cues from human spatial hearing that make it possible to determine from which direction the sound is coming. These cues involve interaural time delay (ITD), interaural level difference (ILD), and / or spectral differences.

[0004] The most common form of spatial audio rendering is based on the concept of point sources, where each sound source is defined as emitting sound from a specific point. Since each sound source is defined as emitting sound from a specific point, the sound sources do not have any size or shape. Different methods have been developed to render sound sources with range (size and shape).

[0005] One such known approach is to create multiple copies of a mono audio element at locations around the audio element. This arrangement creates the perception of a spatially homogeneous object of a certain size. For example, this concept is used in the "object spread" and "object spread" features of the MPEG-H 3D Audio standard (see references [1] and [2]), and in the "object spread" feature of the EBU Audio Definition Model (ADM) standard (see reference [4]). This idea of ​​using a mono audio source has been further developed as described in reference [7], where the area-volume geometry of the sound object is projected onto a sphere around the listener and the sound is rendered to the listener using a pair of head-related (HR) filters evaluated as the integral of all HR filters covering the geometric projection of the object on the sphere. For spherical volume sources, this integral has an analytical solution. However, for arbitrary area-volume source geometries, the integral is evaluated by sampling the projected source surface on the sphere using so-called Monte Carlo ray sampling.

[0006] Another rendering approach renders a spatially diffuse component in addition to the mono audio signal, which creates the perception of some kind of diffuse object that has no apparent precise location compared to the original mono audio element. This concept is used, for example, in the "object diffuseness" feature of the MPEG-H 3D Audio standard (see reference [3]) and in the "object diffuseness" feature of the EBU ADM (see reference [5]).

[0007] Combinations of the two approaches are also known. For example, the "object range" feature of the EBU ADM combines the creation of multiple copies of a mono audio element with the addition of a diffuse component (see Ref. [6]).

[0008] In many cases, the actual shape of an audio element can be well described by basic shapes (e.g., a sphere or a box). But sometimes the actual shape is more complex and needs to be described in a more detailed form (e.g., a mesh structure or a parameterized description format).

[0009] However, these methods do not allow rendering audio elements with significant spatially heterogeneous properties, i.e. audio elements with a certain amount of spatial source variation within their spatial extent. Typically, these sources consist of the sum of multiple sources (e.g. the sound of a forest or the sounds of a cheering crowd). Most of these known solutions are only able to create objects with spatially homogeneous (i.e. no spatial variation within the element) or spatially diffuse properties, which is too limited to render some of the examples given above in a convincing way.

[0010] In the case of heterogeneous audio elements, as described in reference [8], the audio element includes at least two audio channels (i.e., audio signals) to describe the spatial variations within its range. Techniques exist for rendering these heterogeneous audio elements, where the audio element is represented by a multi-channel audio recording, and the rendering uses a number of such virtual speakers to represent the audio element and the spatial variations within it. By placing the virtual speakers at positions corresponding to the range of the audio element, the illusion of audio emanating from the audio element can be conveyed.

[0011] The number of virtual speakers required to achieve a reasonable spatial rendering of a spatially heterogeneous audio element depends on the range of the audio element. For spatially heterogeneous audio elements that are small or at a distance from the listener, a two-speaker setup may be sufficient. Figure 1As shown, for audio elements that are larger and / or close to the listener, a two-speaker setup may be too sparse and result in a psychoacoustic hole between the left speaker (SP-L) and the right speaker (SP-R) due to the far distance between the speakers placed. Adding a third center speaker will help to compensate for this effect, which is why most standardized multi-channel speaker setups have a center speaker. The most straightforward way to render spatially heterogeneous audio elements is to represent each of its audio channels as a virtual speaker, but the number of speakers can also be lower or higher than the number of audio channels. If the number of virtual speakers is lower than the number of audio channels, a downmixing step is required to derive the signal for each virtual speaker. If the number of virtual speakers is higher than the number of audio channels, an upmixing step is required to derive the signal for each virtual speaker. One implementation is to simply use two virtual speakers at fixed positions. Summary of the invention

[0012] There are certain challenges at present. For example, rendering spatially heterogeneous audio elements usually requires the use of multiple virtual speakers. In order to have an evenly distributed audio representation of the range, it may be beneficial to use a large number of speakers, but when the source signal has a limited number of channels (for example, a stereo signal), upsampling a large number of speakers may create problems where more speakers do not improve the spatial quality. In addition, using a large number of virtual speakers can lead to undesirably high complexity. On the other hand, using too few virtual speakers may significantly impair the spatial characteristics of the audio element, so that the rendering no longer represents the corresponding audio element well. Therefore, choosing multiple virtual speakers to render spatially heterogeneous audio elements is a trade-off between complexity and quality.

[0013] The problem of a psychoacoustic void between two speakers described previously is well known and is particularly problematic if the listener is not located exactly in the sweet spot of the speakers. For example, a typical multi-speaker setup designed for home theater use is built around the assumption that the listener is located somewhere around the sweet spot. In these systems, there is usually a center speaker placed halfway between the left and right front speakers, however, in the case of audio rendering for XR, when the listener can move freely in six degrees of freedom, a static speaker setup will not be ideal. In particular, if the listener moves close to the range of the audio source, the problem of psychoacoustic voids may be exacerbated.

[0014] Therefore, designing a static loudspeaker setup that provides a good spatial representation with a limited number of loudspeakers for any listening position is problematic.

[0015] Therefore, in one aspect, a method for rendering an audio element (e.g., a spatially heterogeneous audio element) is provided, wherein the audio element has a range and is represented using a set of virtual speakers including an intermediate virtual speaker. The method includes: based on a listener's position, selecting a position of the intermediate virtual speaker and / or calculating an attenuation factor of the intermediate virtual speaker.

[0016] In another aspect, a computer program comprising instructions is provided, which, when executed by a processing circuit of an audio renderer, causes the audio renderer to perform the above method. In one embodiment, a carrier containing the computer program is provided, wherein the carrier is one of an electrical signal, an optical signal, a radio signal, and a computer-readable storage medium. In another aspect, a rendering device configured to perform the above method is provided. The rendering device may include a memory and a processing circuit coupled to the memory.

[0017] An advantage of the embodiments disclosed herein is that they provide an adaptive method for placing virtual speakers to render spatially heterogeneous audio elements having a range. The embodiments are able to adapt the position of a virtual speaker representing the middle of the range to the current listener position so that the spatial distribution within the range is preserved using a small number of virtual speakers. Compared to the simpler method of evenly distributing speakers within the range of the audio element, the embodiments provide a more efficient solution that will work for all listening positions without using a large number of virtual speakers. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which are incorporated herein and form a part of the specification, illustrate various embodiments.

[0019] Figure 1 Psychoacoustic voids are shown.

[0020] Figure 2A An example virtual speaker setup is shown.

[0021] Figure 2B An example virtual speaker setup that can produce a psychoacoustic void is shown.

[0022] Figure 3 An example virtual speaker setup is shown.

[0023] Figure 4 An example virtual speaker setup is shown where a center speaker is placed close to an edge speaker.

[0024] Figure 5 One embodiment of how a preferred position of a central speaker may be determined is shown.

[0025] Figure 6 Another embodiment of how the preferred position of the middle speaker may be determined is shown.

[0026] Figure 7 One embodiment is shown of how the preferred position of the center speaker may be determined when the audio element range is rectangular in shape.

[0027] Figure 8 is a flow chart illustrating a process according to some embodiments.

[0028] Fig. 9 is a flow chart illustrating a process according to some embodiments.

[0029] Fig.10 is a flow chart illustrating a process according to some embodiments.

[0030] Fig.11A and Fig. 11B A system according to some embodiments is shown.

[0031] Fig.12 A system according to some embodiments is shown.

[0032] Fig.13 A signal modifier according to an embodiment is shown.

[0033] Fig.14 is a block diagram of an apparatus according to some embodiments. DETAILED DESCRIPTION

[0034] The present disclosure proposes, among other things, different ways of adapting the positions of virtual speakers (or simply referred to as "speakers") used to represent audio elements having a range (e.g., spatially heterogeneous audio elements having a particular size and shape). By using multiple (e.g., two or more) speakers to represent the outer edges of the audio element and using at least one speaker (hereinafter referred to as a "middle speaker") adaptively placed between the edges of the audio element or the edges of a simplified shape representing the audio element, an optimized rendering can be achieved with a small number of speakers with perceptually evenly distributed audio energy within the range of the audio element. In addition, the potential problem of undesirable excess energy due to an additional middle speaker overlapping one of the side speakers is solved.

[0035] Overview

[0036] The goal is to render audio elements with a range such that the sound perceived by the listener for any listening position is evenly distributed within the range. Embodiments use as few loudspeakers as possible and avoid or reduce the problem of psychoacoustic holes.

[0037] In some embodiments, the audio element is rendered using a set of speakers that represent edges of the audio element and a middle speaker that is adaptively placed so that it represents the middle of the audio element, wherein the placement of the middle speakers and / or an attenuation factor (also called a gain factor) of the middle speakers takes into account the listening position (e.g., the position of the listener in virtual space relative to the audio element).

[0038] Figure 2A An example of such a rendering setup is shown in Figure 2A An extent 200 representing an audio element is shown. The extent 200 representing the audio element may be the extent of the audio element (i.e., the extent 200 has the same size and shape of the actual extent of the audio element), or it may be a simplified extent (e.g., a line or rectangle) derived from the extent of the audio element. International Patent Application No. WO2021180820 describes different ways to generate such a simplified extent.

[0039] Figure 2A Also shown is that the left speaker 202 is positioned at the left edge point 212 of the range 200, the right speaker 204 is positioned at the right edge point 214 of the range, and the center speaker 203 is positioned somewhere between the left and right edge points.

[0040] Advantageously, in one embodiment, the positioning of the middle speaker 203 is controlled so that when the listener is at least a certain distance (D) from the audio element, the middle speaker 203 is placed at or near the midpoint (MP) 220 between the first edge point and the second edge point of the range, as this will provide the most uniform spatial distribution within the range. The distance D will generally depend on the size of the range 200.

[0041] like Figure 2B As shown, keeping the middle speaker 203 at or near the midpoint 220 can lead to the problem of psychoacoustic void 240 when the listener moves closer to the audio element. Figure 3 As shown, the center speaker 203 will be moved to a new position so as to maintain an even spatial distribution.

[0042] Adaptive placement of center speakers

[0043] Some embodiments herein adaptively position the middle speaker 203 based on the position of the listener. In many cases, the goal is to position the middle speaker at a selected "anchor point" of an audio element that can move with the listener. For example, in one embodiment, the anchor point is the point of range 200 that is closest to the listener. But in other embodiments, the anchor point can be defined differently. International Patent Application No. WO2021180820 describes different ways to select an anchor point for an audio element. However, there are situations where it is not advantageous to position the middle speaker at an anchor point.

[0044] For example, Figure 4 As shown, when the listener is close to one of the edges of the audio element, if the middle speaker is placed on the anchor point, the middle speaker and the corresponding side speaker will overlap and cause an undesirable increase in energy from the corresponding side. In this case, it would be advantageous to position the middle speaker closer to the midpoint between the left edge point and the right edge point (e.g., the positions of the left speaker and the right speaker) in order to have a more evenly distributed audio energy.

[0045] As another example, when the distance between the listener and the audio element is greater than a threshold (which may depend on the range of the audio element), experiments have shown that placing the center speaker closer to the midpoint 220 gives a more perceptually relevant output.

[0046] In summary, when the listener is close to the range but not close to one of the edges of the range, it is generally preferred to position the center speaker at the anchor point; at other listening positions, the preferred center speaker position is at or near the midpoint 220.

[0047] Hence, the proposed embodiment provides an adaptive placement of the middle loudspeaker which optimizes the position depending on the current listening position.

[0048] More specifically, in one embodiment, when the anchor position is close to one of the edge speakers, avoid placing the middle speaker at the anchor point. Therefore, in addition to considering the anchor point (A) 590 (see Figure 5 ), the positioning of the middle speaker 203 may also depend on the midpoint (MP) 220.

[0049] In one embodiment, equation (1) is used to calculate the preferred position of the middle speaker 203 (denoted as "M" 591) on a line 599 from one edge point of the range 200 (e.g., left edge point 212) to another edge point of the range 200 (e.g., right edge point 214):

[0050] M=α*A+(1-α)*MP (Equation 1).

[0051] Where A is the location of the anchor point on line 599, MP is the location of the midpoint on line 599, and is a factor that controls the weighting of the anchor point and the midpoint when positioning the center speaker on line 599.

[0052] The value of α is such that when the listener is close to the range but not close to the edge of the range, M is near or on the anchor point ( ), and when the listener is near one of the edges of range 200 or away from the range, M is near or at the midpoint ( ).therefore, The movement of the listener in the x-direction and the z-direction is considered at the same time. Once the M point 591 is determined, the middle speaker 203 can be "placed" at the M point. That is, the position of the middle speaker 203 is set to the M point.

[0053] In one embodiment, α is a function of the following two variables: xm_w and zm_w. That is:

[0054] α=f(xm_w,zm_w) (Equation 2).

[0055] The first variable xm_w is a weight reflecting the x-direction movement of the listener, and Is the weight that reflects the listener's z-direction movement. Use Figure 5 The parameters β and λ shown are used to set the values ​​of xm_w and zm_w.

[0056] In one embodiment, xm_w is a function of λ and β—that is:

[0057] (Equation 3).

[0058] From equation 3 and Figure 5 It can be seen that as the listener moves toward either edge of range 200, then: λ→0 or β→0, and thus xm_w→0. Similarly, as the listener moves toward the middle, then λ and β approach each other, and xm_w approaches 1.

[0059] Z weight is also a function of λ and β—that is:

[0060] zm_w=f2(λ,β)=sin((λ+β) / 2) (Equation 4).

[0061] From Equation 4 and Figure 5 It can be seen that the closer the listener is to the range, the larger λ and β become, where at a very close distance to the range, λ+β approaches 180 degrees, so zm_w approaches 1, and the farther the listener is from the range, the smaller λ and β become, so zm_w approaches 0.

[0062] In one embodiment, α is defined as:

[0063] α=f3(xm_w,zm_w)=min(1,d*xm_w*zm_w) (Equation 5)

[0064] in, Yes Control The above derivation of α is just one way of using information about the relative position of the listener to this range. Using λ and Any other way of deriving α (e.g., cosine or tangent of λ and β) or any other parameter that can reflect the relative position of the listener to the range (e.g., coordinates of the listener and the midpoint, anchor point, left edge, and right edge of the range) can be used to derive α.

[0065] Another method for dealing with excess energy when the listener is at the edge of an audio element may be to attenuate the energy of the signal played back through the center speaker, rather than changing its position. That is, the center speaker 203 is positioned at the anchor point, but as the listener approaches the edge of the audio element, the audio signal of the center speaker 203 is attenuated. To this end, a Figure 5 The angle described in , so that:

[0066]

[0067] Where X is the original time domain audio signal of the center speaker 203, and X' is the time domain signal played back by the center speaker 203. This method alleviates the energy excess problem, but may not improve the spatial perception of the audio elements.

[0068] Adaptive placement of middle speakers based on equal angles

[0069] In another embodiment, the placement of the center speaker is controlled only by the angles from the left and right edge points. Figure 6 These angles are = λ+θ and - = β-θ. In one embodiment, point M 591 is selected so that = If the distance (dA) between the listener and the anchor point is known, the distance (dM) from point M 591 to point 212 can be determined by calculating dM = dA*(tan(λ) + tan(θ)), so the x coordinate of point M is equal to the x coordinate of left edge point 212 + dM.

[0070] If dA is unknown, but v (the distance between the listener and the left edge point 212) and w (the distance between the listener and the right edge point 214) are known, the position of point M can be determined by calculating M = (v*Re+w*Le) / (v+w), where Re is the x-coordinate of the right edge point 214 and Le is the x-coordinate of the left edge point 212.

[0071] Adaptive placement of two-dimensional middle speakers

[0072] The examples provided above illustrate a one-dimensional range of audio elements. The above techniques also apply to two-dimensional ranges of audio elements, such as Figure 7 700 is shown. In this example, range 700 has a rectangular shape and four edge points are defined: top edge point 701, right edge point 702, bottom edge point 703, and left edge point 704. In this example, for each edge point, the edge point is located at the midpoint of the edge where the edge point is located. Therefore, left edge point 704 is a position that is exactly centered between the upper left corner and the lower left corner of range 700; right edge point 702 is exactly centered between the upper right corner and the lower right corner of range 700; top edge point 701 is exactly centered between the upper left corner and the upper right corner of range 700; and bottom edge point 703 is exactly centered between the lower left corner and the lower right corner of range 700.

[0073] In one embodiment, for each defined edge point 701 to 704, a speaker may be positioned at that point. Thus, in one embodiment, four speakers are used to represent the top edge, bottom edge, left edge, and right edge of the two-dimensional plane 700. In another embodiment, a speaker is positioned at each corner point of the plane 700.

[0074] Additionally, a middle speaker may also be employed and may be positioned using the same principles as already described. That is, the coordinates (Mx, My) of the middle speaker may be determined by calculating:

[0075] Mx = αx*A1x+(1-αx)*MPx; and

[0076] My=αy*A2y+(1-αy)*MPy, where:

[0077] αx=f3(xm_w,zm1_w), αy=f3(ym_w,zm2_w), A1x is the x-coordinate of anchor point A1, A2y is the y-coordinate of anchor point A2, MPx is the x-coordinate of the midpoint, MPy is the y-coordinate of the midpoint, xm_w=f1(λx,βx), ym_w=f1(λy,βy), zm1_w=f2(λx,βx), and zm2_w=f2(λy,βy).

[0078] Figure 8 s800 is a flowchart illustrating a process 800 for rendering an audio element (e.g., a spatially heterogeneous audio element) according to an embodiment, wherein the audio element has a range and is represented using a set of virtual speakers including an intermediate virtual speaker. The process 800 may begin at step s802. Step s802 includes: selecting a position of the intermediate virtual speaker and / or calculating an attenuation factor of the intermediate virtual speaker based on a position of a listener.

[0079] Fig. 9 is a flow chart illustrating a process 900 for rendering an audio element (eg, a spatially heterogeneous audio element) having a range and represented using a set of virtual speakers including a middle virtual speaker according to an embodiment. The process 900 may start at step s902.

[0080] Step s902 includes: selecting an anchor point for the audio element. Typically, the anchor point is on a straight line passing through a first edge point of a range associated with the audio element (e.g., the actual range of the audio element or a simplified range of the audio element) and a second edge point of the range, and the anchor point depends on the position of the listener. In some embodiments where the audio element has a complex range, selecting the anchor point is part of a process for creating a simplified range for the audio element based on the listener position and the range of the audio element.

[0081] Step s904 includes: placing a first speaker (e.g., a right speaker), and placing a second speaker (e.g., a left speaker). That is, determining the positions of the first speaker and the second speaker. In one embodiment, the speakers are positioned on opposite edges of the range. That is, the left speaker is positioned on a left edge point of the range, and the right speaker is positioned on a right edge point of the range. In an embodiment where the range associated with the audio element is a rectangle, the speakers are placed in each corner of the rectangle.

[0082] Step s906 includes determining a midpoint between the two speakers.

[0083] Step s908 includes determining a first angle (λ) between a straight line from the listener to the anchor point and a straight line from the listener to the left speaker, and a second angle (β) between a straight line from the listener to the anchor point and a straight line from the listener to the right speaker ( Figure 5 Examples of λ and β are shown in ).

[0084] Step s910 includes: determining an x-weight value (xm_w) and a z-weight value (zm_w) (eg, calculating xm_w=f1(λ,β) and zm_w=f2(λ,β)).

[0085] Step s912 includes: determining a factor (α) based on xm_w and zm_w. That is, α is a function of xm_w and zm_w (eg, α=f3(xm_w,zm_w)).

[0086] Step s914 comprises calculating M=α*A+(1-α)*MP, where M is the x-coordinate of the preferred position of the center speaker, A is the x-coordinate of the anchor point, and MP is the x-coordinate of the midpoint between the left and right speakers.

[0087] Fig.10 is a flow chart illustrating a process 1000 for rendering an audio element (eg, a spatially heterogeneous audio element) having a range and represented using a set of virtual speakers including a middle virtual speaker according to an embodiment. The process 1000 may start at step s1002.

[0088] Step s1002 includes: selecting an anchor point of the audio element (see the above step s902 ).

[0089] Step s1004 includes: placing a first speaker (eg, a right speaker) and placing a second speaker (eg, a left speaker), that is, determining the positions of the first speaker and the second speaker (see step s904 above).

[0090] Step s1006 includes determining a first angle (λ) between a straight line from the listener to the anchor point and a straight line from the listener to the left speaker, and a second angle (β) between a straight line from the listener to the anchor point and a straight line from the listener to the right speaker ( Figure 5 Examples of λ and β are shown in ).

[0091] Step s1008 includes: using λ and β to calculate a gain factor (g), for example, g=f4(λ,β).

[0092] Step s1010 comprises: processing the signal (X) of the intermediate loudspeaker using a gain factor (g) to produce a modified signal X'. For example, X'=g*X.

[0093] Example Use Case

[0094] Fig.11A An XR system 1100 is shown in which embodiments disclosed herein may be applied. The XR system 1100 includes speakers 1104 and 1105 (which may be speakers of headphones worn by a listener) and an XR device 1110, which may include a display for displaying images to a user and, in some embodiments, is configured to be worn by the listener. In the illustrated XR system 1100, the XR device 1110 has a display and is designed to be worn on the user's head and is generally referred to as a head-mounted display (HMD).

[0095] like Fig. 11B As shown, the XR device 1110 may include an orientation sensing unit 1101, a position sensing unit 1102, and a processing unit 1103, which is coupled (directly or indirectly) to an audio renderer 1151 for generating output audio signals (e.g., a left audio signal 1181 for a left speaker and a right audio signal 1182 for a right speaker as shown in the figure).

[0096] The orientation sensing unit 1101 is configured to detect changes in the orientation of the listener and provide information about the detected changes to the processing unit 1103. In some embodiments, given a detected change in orientation detected by the orientation sensing unit 1101, the processing unit 1103 determines an absolute orientation (relative to some coordinate system). There may also be different systems for determining orientation and position, for example, systems using lighthouse trackers (lidar). In one embodiment, given a detected change in orientation, the orientation sensing unit 1101 may determine an absolute orientation (relative to some coordinate system). In this case, the processing unit 1103 may simply multiplex the absolute orientation data from the orientation sensing unit 1101 and the position data from the position sensing unit 1102. In some embodiments, the orientation sensing unit 1101 may include one or more accelerometers and / or one or more gyroscopes.

[0097] The audio renderer 1151 generates an audio output signal based on an input audio signal 1161, metadata 1162 about the XR scene being experienced by the listener, and information 1163 about the position and orientation of the listener. The metadata 1162 of the XR scene may include metadata for each object and audio element included in the XR scene, and the metadata of the object may include information about the size of the object. The metadata 1162 may also include control information, such as a reverberation time value, a reverberation level value, and / or an absorption parameter. The audio renderer 1151 may be a component of the XR device 1110, or it may be remote from the XR device 1110 (e.g., the audio renderer 1151 or its components may be implemented in a so-called "cloud").

[0098] Fig.12An example implementation of an audio renderer 1151 for generating sounds of an XR scene is shown. The audio renderer 1151 includes a controller 1201 and a signal modifier 1202, which is used to modify an audio signal 1161 (e.g., an audio signal of a multi-channel audio element) based on control information 1210 from the controller 1201. The controller 1201 can be configured to receive one or more parameters and trigger the modifier 1202 to perform a modification (e.g., increase or decrease the volume level) on the audio signal 1161 based on the received parameters. The received parameters include information 1163 about the position and / or orientation of the listener (e.g., the direction and distance to the audio element), and metadata 1162 (e.g., range 200) about the audio element in the XR scene (in some embodiments, the controller 1201 itself generates the metadata 1162). Using the metadata and position / orientation information, the controller 1201 can calculate more than one gain factor (g) (also called attenuation factor) of the audio element in the XR scene as described herein.

[0099] Fig.13 An example implementation of the signal modifier 1202 is shown according to one embodiment. The signal modifier 1202 includes a directional mixer 1304, a gain adjuster 1306, and a speaker signal generator 1308.

[0100] The directional mixer receives an audio input 1161, in this example, the audio input 1161 includes a pair of audio signals 1301 and 1302 associated with an audio element (e.g., an audio element associated with range 200 or 700), and generates a set of k virtual speaker signals (VS1, VS2, ..., VSk) based on the audio input and control information 1391. In one embodiment, the signal for each virtual speaker can be derived by, for example, appropriate mixing of the signal including the audio input 1161. For example: VS1 = α × L + β × R, where L is the input audio signal 1301, R is the input audio signal 1302, and α and β are factors that depend on, for example, the position of the listener relative to the audio element and the position of the virtual speaker corresponding to VS1.

[0101] The gain adjuster 1306 can adjust the gain of any one or more virtual speaker signals based on the control information 1392, which can include the above-mentioned gain factors as calculated by the controller 1301. That is, for example, when the middle speaker 203 is placed close to another speaker (e.g., Figure 4 When the left speaker 202 is selected as shown, the controller 1301 may control the gain adjuster 1306 to adjust the gain of the virtual speaker signal for the middle speaker 203 by providing the gain adjuster 1306 with the gain factor calculated as described above.

[0102] Using the virtual speaker signals VS1, VS2, ..., VSk, the speaker signal generator 1308 generates output signals (e.g., output signals 1181 and 1182) for driving speakers (e.g., headphone speakers or other speakers). In one embodiment where the speakers are headphone speakers, the speaker signal generator 1308 can perform conventional binaural rendering to generate the output signals. In an embodiment where the speakers are not headphone speakers, the speaker signal generator can perform conventional speech panning to generate the output signals.

[0103] Fig.14 1 is a block diagram of an audio rendering device 1400 for performing the methods disclosed herein according to some embodiments (eg, the audio rendering device 1400 may be used to implement the audio renderer 1151). Fig.14As shown, the audio rendering device 1400 may include: a processing circuit (PC) 1402, which may include one or more processors (P) 1455 (e.g., a general-purpose microprocessor and / or one or more other processors, such as an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), etc.), which may be co-located in a single housing or a single data center, or may be geographically distributed (i.e., the device 1400 may be a distributed computing device); at least one network interface 1448, the at least one network interface 1448 including a transmitter (Tx) 1445 and a receiver (Tx). A machine (Rx) 1447 for enabling the apparatus 1400 to send data to and receive data from other nodes connected to a network 110 (e.g., an Internet Protocol (IP) network), to which a network interface 1448 is connected (directly or indirectly) (e.g., the network interface 1448 may be wirelessly connected to the network 110, in which case the network interface 1448 is connected to an antenna arrangement); and a storage unit (also referred to as a "data storage system") 1408, which may include one or more non-volatile storage devices and / or one or more volatile storage devices. In embodiments where the PC 1402 includes a programmable processor, a computer readable medium (CRM) 1442 may be provided. The CRM 1442 stores a computer program (CP) 1443 including computer readable instructions (CRI) 1444. The CRM 1442 may be a non-transitory computer readable medium, such as a magnetic medium (e.g., a hard disk), an optical medium, a memory device (e.g., a random access memory, a flash memory), or the like. In some embodiments, the CRI 1444 of the computer program 1443 is configured such that when executed by the PC 1402, the CRI causes the audio rendering device 1400 to perform the steps described herein (e.g., the steps described herein with reference to the flowcharts). In other embodiments, the audio rendering device 1400 may be configured to perform the steps described herein without requiring code. That is, for example, the PC 1402 may consist solely of one or more ASICs. Thus, the features of the embodiments described herein may be implemented in hardware and / or software.

[0104] Overview of Various Embodiments

[0105] A1. A method for rendering an audio element (e.g., a spatially heterogeneous audio element), wherein the audio element has a range and is represented using a set of virtual speakers including an intermediate virtual speaker, the method comprising: based on a listener's position, selecting a position of the intermediate virtual speaker and / or calculating an attenuation factor of the intermediate virtual speaker.

[0106] A2. A method according to embodiment A1, wherein the method includes selecting the position of the middle virtual speaker based on the position of the listener, and selecting the position of the middle virtual speaker based on the position of the listener includes: determining a first angle based on the position of the listener and i) the position of a first edge point of the audio element or a first edge point of a range determined based on the range of the audio element, or ii) the position of a first virtual speaker; determining a second angle based on the position of the listener and i) the position of a second edge point of the audio element or a second edge point of the range, or ii) the position of a second virtual speaker; and calculating a first coordinate Mx of the middle virtual speaker using the first angle and the second angle, wherein the selected position of the middle virtual speaker is at least partially specified by the calculated first coordinate.

[0107] A3. A method according to embodiment A2, wherein selecting the position of the middle virtual speaker based on the position of the listener further includes: determining a third angle based on the position of the listener and i) the position of a third edge point of the audio element or a third edge point of the range or ii) the position of a third virtual speaker; determining a fourth angle based on the position of the listener and i) the position of a fourth edge point of the audio element or a fourth edge point of the range or ii) the position of a fourth virtual speaker; and calculating the second coordinate My of the middle virtual speaker using the third angle and the fourth angle, wherein the selected position is at least partially specified by the calculated first coordinates and second coordinates.

[0108] A4. A method according to embodiment A2 or A3, wherein using the first angle and the second angle to calculate the first coordinate Mx of the intermediate virtual speaker includes: using the first angle and the second angle to determine a first factor α1; and calculating: Mx = (α1*Ax)+((1-α1)*MPx), wherein Ax is a coordinate for specifying the position of a determined anchor point on a straight line extending between the first edge point and the second edge point (or between the first virtual speaker and the second virtual speaker), and MPx is a coordinate for specifying the midpoint of the straight line.

[0109] A5. The method of embodiment A4, wherein determining α1 using the first angle and the second angle comprises: calculating a first weight xm_w using the first angle and the second angle, calculating a second weight zm_w using the first angle and the second angle, and determining α1 based on xm_w and zm_w.

[0110] A6. A method according to embodiment A5, wherein determining α1 based on xm_w and zm_w includes: determining whether d*xm_w*zm_w is less than 1, wherein d is a predetermined factor; and if d*xm_w*zm_w is not less than 1, setting α1 equal to 1, otherwise setting α1 equal to d*xm_w*zm_w.

[0111] A7. A method according to embodiment A5 or A6, wherein calculating xm_w using the first angle and the second angle includes calculating: xm_w=sin(λ) / sin(β) or xm_w=sin(β) / sin(λ), wherein λ is the first angle and β is the second angle.

[0112] A8. A method according to embodiments A5, A6 or A7, wherein calculating zm_w using the first angle and the second angle includes calculating: zm_w=sin((λ+β) / 2).

[0113] A9. A method according to embodiment A3, wherein using the third angle and the fourth angle to calculate the second coordinate My of the intermediate virtual speaker includes: using the third angle and the fourth angle to determine the second factor α2; and calculating: My=(α2*Ay)+((1-α2)*MPy), wherein Ay is a coordinate for specifying the position of a determined anchor point on a straight line extending between the third edge point and the fourth edge point (or between the third virtual speaker and the fourth virtual speaker), and MPy is a coordinate for specifying the midpoint of the straight line.

[0114] A10. A method according to embodiment A1, wherein the method includes selecting the position of the intermediate virtual speaker based on the position of the listener, and selecting the position of the intermediate virtual speaker based on the position of the listener includes: selecting 1) a position point between a first point (first edge point) of the audio element or a range determined based on the range of the audio element and a second point (second edge point) of the audio element or the range, or 2) a position point on a first straight line between a first virtual speaker and a second virtual speaker, so that: i) an angle between a second straight line extending from the position of the listener to the first point (or the first virtual speaker) and ii) a third straight line extending from the position of the listener to a selected position point on the first straight line is equal to an angle between i) a fourth straight line extending from the position of the listener to the second point (or the second virtual speaker) and ii) the third straight line.

[0115] A11. A method according to embodiment A10, wherein selecting the position point includes calculating the coordinates M of the position point by calculating the following formula: M=(v*Re+w*Le) / (v+w), wherein v is the length of the second straight line, w is the length of the third straight line, Re is the coordinates of the first point or the first virtual speaker, and Le is the coordinates of the second point or the second virtual speaker.

[0116] A12. The method according to embodiment A10 or A11 further includes: positioning the intermediate virtual speaker at the selected position point.

[0117] A13. A method according to any one of embodiments A1 to A12, wherein the method includes calculating the attenuation factor of the intermediate virtual speaker based on the position of the listener, and calculating the attenuation factor of the intermediate virtual speaker based on the position of the listener includes: determining a first angle based on the position of the listener and i) the first edge point of the audio element or the position of the first edge point of a range determined based on the range of the audio element, or ii) the position of the first virtual speaker; determining a second angle based on the position of the listener and i) the second edge point of the audio element or the position of the second edge point of the range, or ii) the position of the second virtual speaker; and calculating ε=sin(λ) / sin(β) or ε=sin(β) / sin(λ), wherein λ is the first angle, β is the second angle, and ε is the attenuation factor.

[0118] A14. The method according to embodiment A13 also includes: modifying the signal X for the intermediate virtual speaker to produce a modified intermediate virtual speaker signal X', so that X'=ε*X, and using the modified intermediate virtual speaker signal to render the audio element (for example, using the intermediate virtual speaker signal to generate an output signal).

[0119] A15. A method according to any one of embodiments A2 to A14, wherein the virtual speaker set also includes: a first virtual speaker positioned at the first edge point and a second virtual speaker positioned at the second edge point, and the method also includes rendering the audio element using information identifying the positions of the virtual speakers.

[0120] A16. A method according to embodiments A3 or A9, wherein the virtual speaker set also includes: a first virtual speaker positioned at the first edge point (e.g., a first corner point, or a first point that is a midpoint between a first pair of diagonal points), a second virtual speaker positioned at the second edge point (e.g., a second corner point, or a second point that is a midpoint between another pair of diagonal points), a third virtual speaker positioned at the third edge point (e.g., a third corner point, or a third point that is a midpoint between another pair of diagonal points), and a fourth virtual speaker positioned at the fourth edge point (e.g., a fourth corner point, or a fourth point that is a midpoint between another pair of diagonal points), and the method also includes rendering the audio element using information identifying the positions of the virtual speakers.

[0121] A17. A method according to embodiment A1, wherein the method includes selecting the position of the middle virtual speaker based on the position of the listener, and selecting the position of the middle virtual speaker based on the position of the listener includes: determining the distance from the position of the listener to the position of the audio element; determining whether the determined distance is greater than a threshold; and as a result of determining that the determined distance is greater than the threshold, selecting a position at the midpoint of the audio element or the midpoint of a range determined based on the range of the audio element.

[0122] A18. A method according to embodiment A1, wherein the method includes selecting the position of the middle virtual speaker based on the position of the listener, and selecting the position of the middle virtual speaker based on the position of the listener includes: i) obtaining listener information indicating the coordinates of the listener (e.g., x-coordinate); ii) obtaining midpoint information indicating the coordinates of the midpoint between a first point associated with the audio element (e.g., a first edge point of a range associated with the audio element) and a second point associated with the audio element (e.g., a second edge point of the range); and iii) selecting the position of the middle virtual speaker based on the midpoint information and the listener information.

[0123] A19. A method according to embodiment A18, wherein selecting the position of the middle virtual speaker based on the midpoint information and the listener information includes: i) determining the coordinates of an anchor point; and ii) selecting the position of the middle virtual speaker based on the midpoint information and anchor information, the anchor information indicating the determined coordinates of the anchor point.

[0124] A20. A method according to embodiment A19, wherein: i) the midpoint information includes a midpoint value MP specifying the coordinates of the midpoint, ii) the anchor information includes an anchor value A specifying the coordinates of the anchor point, and iii) selecting the position of the middle virtual speaker based on the midpoint information and the anchor information includes using MP and A to calculate the coordinate value M of the middle speaker.

[0125] A21. The method of embodiment A20, wherein the anchor value A depends on the indicated coordinates of the listener.

[0126] A22. The method of embodiment A21, wherein A=L, wherein L is the coordinates of the indicated listener (eg, Figure 5 and Figure 6 As shown, a coordinate system can be defined relative to the range so that the range extends along the x-axis).

[0127] A23. A method according to embodiments A20, A21 or A22, wherein calculating M using MP and A includes: calculating M=α*A+(1-α)*MP, wherein α is a factor that depends on the indicated coordinates of the listener.

[0128] A24. A method according to any one of embodiments A18 to A23, wherein the listener information further indicates a second coordinate of the listener (e.g., a y coordinate), the midpoint information further indicates a second coordinate of the midpoint (e.g., a y coordinate), and selecting the position of the middle virtual speaker based on the midpoint information and the listener information further includes: determining the second coordinate of the middle virtual speaker based on the second coordinate of the listener and the second coordinate of the midpoint.

[0129] A25. A method according to any one of embodiments A10 or A18 to A24, wherein a first virtual speaker is positioned at the first point and a second virtual speaker is positioned at the second point; and the method further comprises: rendering the audio element using information identifying the position of the virtual speaker.

[0130] A26. The method according to any one of embodiments A1 to A25 further includes: generating an intermediate virtual speaker signal for the intermediate virtual speaker based on the position of the intermediate virtual speaker; and using the intermediate virtual speaker signal to render the audio element (for example, using the intermediate virtual speaker signal to generate an output signal).

[0131] B1. A computer program, comprising instructions, which, when executed by a processing circuit of an audio renderer, cause the audio renderer to perform the method according to any one of the above embodiments.

[0132] B2. A carrier, containing a computer program, wherein the carrier is one of an electric signal, an optical signal, a radio signal and a computer-readable storage medium.

[0133] C1. An audio rendering device, configured to execute the method according to any one of the above embodiments.

[0134] C2. The audio rendering device of embodiment C1, wherein the audio rendering device comprises a memory and a processing circuit coupled to the memory.

[0135] Although various embodiments are described herein, it should be understood that they are presented by way of example only and not limitation. The breadth and scope of the present invention should not be limited by any of the above exemplary embodiments. In addition, any combination of the above objects with all possible variations thereof is included in the present disclosure, unless otherwise indicated or otherwise clearly conflicting with the context.

[0136] Additionally, although the processes described above and shown in the accompanying drawings are shown as a series of steps, this is for illustrative purposes only. Therefore, it is contemplated that some steps may be added, some steps may be omitted, the order of steps may be rearranged, and some steps may be performed in parallel.

[0137] References

[0138] [1] MPEG-H 3D Audio, Clause 8.4.4.7: “Spreading”

[0139] [2] MPEG-H 3D Audio, Clause 18.1: “Element Metadata Preprocessing”

[0140] [3] MPEG-H 3D Audio, Clause 18.11: “Diffuseness Rendering”

[0141] [4] EBU ADM Renderer Tech 3388, Clause 7.3.6: “Divergence”

[0142] [5] EBU ADM Renderer Tech 3388, Clause 7.4: “Decorrelation Filters”

[0143] [6] EBU ADM Renderer Tech 3388, Clause 7.3.7: “Extent Panner”

[0144] [7] Efficient HRTF-based Spatial Audio for Area and VolumetricSources“, IEEE Transactions on Visualization and Computer Graphics 22(4):1-1· January 2016

[0145] [8] Patent Publication WO2020144062, “Efficient spatially-heterogeneous audio elements for Virtual Reality.”

[0146] [9] Patent Publication WO2021180820, “Rendering of Audio Objects witha Complex Shape.”。

Claims

1. A method (800) for rendering an audio element, wherein: The audio element has a range (200) and is represented using a set of virtual speakers (202, 203, 204) including an intermediate virtual speaker (203), the method comprising: Based on the position of the listener, selecting (s802) the position of the intermediate virtual speaker, wherein selecting the position of the intermediate virtual speaker based on the position of the listener comprises: Selecting a position point 1) between a first point of the audio element or a range determined based on the range of the audio element and a second point of the audio element or the range or 2) on a first straight line between a first virtual speaker and a second virtual speaker, such that: The angle between i) a second straight line extending from the listener's position to the first point or the first virtual speaker and ii) a third straight line extending from the listener's position to a selected position point on the first straight line is equal to the angle between i) a fourth straight line extending from the listener's position to the second point or the second virtual speaker and ii) the third straight line.

2. The method according to claim 1, wherein: Selecting the location point includes calculating the coordinates M of the location point by calculating the following formula: M=(v*Re+w*Le) / (v+w), where: v is the length of the second straight line, w is the length of the third straight line, Re is the coordinate of the first point or the first virtual loudspeaker, and Le is the coordinate of the second point or the second virtual speaker.

3. The method according to claim 1 or 2, further comprising: The intermediate virtual loudspeaker is positioned at the selected position point.

4. The method according to any one of claims 1 to 3, wherein: The method further includes calculating an attenuation factor of the intermediate virtual speaker based on the position of the listener, wherein calculating the attenuation factor of the intermediate virtual speaker based on the position of the listener includes: determining a first angle based on the position of the listener and i) a position of a first edge point of the audio element or a first edge point of a range determined based on the range of the audio element or ii) a position of a first virtual speaker; determining a second angle based on the position of the listener and i) a position of a second edge point of the audio element or a second edge point of the range or ii) a position of a second virtual speaker; and Calculate ε=sin(λ) / sin(β) or ε=sin(β) / sin(λ), where λ is the first angle, β is the second angle, and ε is the attenuation factor.

5. The method according to claim 4, further comprising: The signal X for the intermediate virtual speaker is modified to produce a modified intermediate virtual speaker signal X' such that X'=ε*X, and the audio element is rendered using the modified intermediate virtual speaker signal.

6. The method according to any one of claims 1 to 5, further comprising: Based on the position of the intermediate virtual speaker, generating an intermediate virtual speaker signal for the intermediate virtual speaker; as well as The audio element is rendered using the intermediate virtual speaker signal.

7. A computer program (1443) comprising instructions (1444) which, when executed by a processing circuit (1402) of an audio renderer (1400), cause the audio renderer to perform the method according to any one of claims 1 to 6.

8. A carrier comprising a computer program according to claim 7, wherein: The carrier is one of an electric signal, an optical signal, a radio signal and a computer-readable storage medium (1442).

9. An audio rendering device (1400) for rendering an audio element, wherein: The audio element has a range (200) and is represented using a set of virtual speakers (202, 203, 204) including an intermediate virtual speaker (203), the audio rendering device being configured to: Based on the position of the listener, a position of the intermediate virtual speaker is selected (s802), wherein the audio rendering apparatus is configured to select the position of the intermediate virtual speaker based on the position of the listener in the following manner: Selecting a position point 1) between a first point of the audio element or a range determined based on the range of the audio element and a second point of the audio element or the range or 2) on a first straight line between a first virtual speaker and a second virtual speaker, such that: The angle between i) a second straight line extending from the listener's position to the first point or the first virtual speaker and ii) a third straight line extending from the listener's position to a selected position point on the first straight line is equal to the angle between i) a fourth straight line extending from the listener's position to the second point or the second virtual speaker and ii) the third straight line.

10. The audio rendering device according to claim 9, wherein: Selecting the location point includes calculating the coordinates M of the location point by calculating the following formula: M=(v*Re+w*Le) / (v+w), where: v is the length of the second straight line, w is the length of the third straight line, Re is the coordinate of the first point or the first virtual loudspeaker, and Le is the coordinate of the second point or the second virtual speaker.

11. The audio rendering device according to claim 9 or 10, wherein: The audio rendering device is further configured to position the intermediate virtual speaker at the selected position point.

12. The audio rendering device according to any one of claims 9 to 11, wherein: The audio rendering device is configured to calculate an attenuation factor of the intermediate virtual speaker based on the position of the listener in the following manner: determining a first angle based on the position of the listener and i) a position of a first edge point of the audio element or a first edge point of a range determined based on the range of the audio element or ii) a position of a first virtual speaker; determining a second angle based on the position of the listener and i) a position of a second edge point of the audio element or a second edge point of the range or ii) a position of a second virtual speaker; as well as Calculate ε=sin(λ) / sin(β) or ε=sin(β) / sin(λ), where λ is the first angle, β is the second angle, and ε is the attenuation factor.

13. The audio rendering device according to any one of claims 9 to 12, further configured to: generating an intermediate virtual speaker signal for the intermediate virtual speaker based on the position of the intermediate virtual speaker; and The audio element is rendered using the intermediate virtual speaker signal.

Images