Rendering of stereoscopic audio elements

Abstract

A method for rendering an audio element (e.g., a stereoscopic audio element comprising two or more audio signals). In one embodiment, the method includes obtaining distance gain model rendering parameters associated with stereoscopic audio elements. The method further includes selecting a distance gain model from a set of two or more candidate distance gain models based on the value of the distance gain model rendering parameter. The method further includes rendering the stereoscopic audio element using the selected distance gain model.

Description

Technical Field

[0001] Embodiments related to rendering of volumetric audio elements are disclosed. Background Art

[0002] Spatial audio rendering is a process used to present audio within an extended reality (XR) scene (e.g., a virtual reality (VR), augmented reality (AR), or mixed reality (MR) scene) so that the listener is given the impression that the sound is coming from a physical source at a specific location within the scene and having a specific non-zero size and shape (i.e., having a "spatial extent" or simply "extent"). Rendering can be done through headphone speakers or other speakers. If rendering is done through headphone speakers, the process used is called binaural rendering, and uses spatial cues from human spatial hearing that can determine which direction a sound is coming from. The cues involve interaural time delay (ITD), interaural level difference (ILD), and / or spectral differences.

[0003] In real life, we often perceive a sound that is actually the sum of sound waves generated by many sound sources located on a specific surface or in a specific volume or area. Conceptually, such a surface, volume or area can be considered as a single audio element with spatially heterogeneous characteristics (i.e., an audio element with a specific amount of spatial source variation in its spatial extent). A subclass of such audio elements is spatially heterogeneous audio elements, for which the perceived heterogeneous spatial characteristics do not vary very much along certain paths in 3D space. Here are a few examples: 1) Crowd sounds (the sum of the sounds from many individuals standing close to each other within a volume defined in space that arrive at the listener's two ears); 2) River sounds (the sum of all water splashing sound waves emitted from the river surface that arrive at the listener's two ears); 3) Beach sounds (the sum of all sound waves generated by waves hitting the shoreline of the beach that arrive at the listener's two ears; 4) Fountain sounds (the sum of all sound waves generated by water hitting the fountain's water surface that arrive at the listener's two ears); and 5) Busy highway sounds (the sum of the sounds from many cars that arrive at the listener's two ears).

[0004] In the case of a river sound, the perceived characteristics of the sound will not change significantly for a listener walking by the river. The same is true for a listener walking along a beach or around a crowd.

[0005] Existing methods of representing these types of sounds include functionality for modifying the perceived size of monophonic audio objects, which is typically controlled by additional metadata associated with the object (e.g., "size," "extension," or "diffusivity" parameters).

[0006] One such known approach is to create multiple copies of a mono audio element at locations around the audio element. This arrangement creates the perception of a spatially homogeneous object of a certain size. This concept is used, for example, in the "object extension" and "object emanation" features of the MPEG-H 3D Audio standard (see 8.4.4.7 and 18.1 of reference [1]) and the "object emanation" feature of the EBU Audio Definition Model (ADM) standard (see 7.3.6 of reference [2]).

[0007] This idea of ​​using a monophonic audio source has been further developed as described in [3], where the area-volume geometry of the sound object is projected onto a sphere around the listener and the sound is rendered to the listener using a pair of head-related (HR) filters evaluated as the integral of all HR filters covering the geometric projection of the object on the sphere. For spherical volume sources, this integral has an analytical solution. However, for arbitrary area-volume source geometries, the integral is evaluated by sampling the projected source surface on the sphere using so-called Monte Carlo ray sampling.

[0008] Another rendering approach renders a spatially diffuse component in addition to the mono audio signal, which creates the perception of a slightly diffuse object without a distinct pin-point location compared to the original mono audio element. This concept is used, for example, in the "object diffuseness" feature of the MPEG-H 3D Audio standard (see reference [3]) and in the "object diffuseness" feature of the EBU ADM (see 7.4 of reference [2]).

[0009] Combinations of the two approaches are also known. For example, the "object range" feature of EBU ADM combines the creation of multiple copies of a mono audio element with the addition of a diffuse component (see 7.3.7 of reference [2]). Summary of the invention

[0010] There are certain challenges. For example, in the case where a stereo audio element (i.e., an audio element with a range) represents so many individual sources that the individual sources essentially behave as one large composite source, the rendering of the audio element can be based on a stereo behavior, where, for example, the distance gain (i.e., the relative level of the rendered audio element as a function of the listening distance) is calculated based on the size of its range; but if the stereo audio element represents sound sources that, due to their spatial distribution over the audio element, would be expected to behave as individual sources with their own specific positions within the range of the audio element, then the expected behavior is that of a collection of point sources, where the distance gain function follows the inverse distance law 1 / r. Therefore, this choice of gain function depends on the specific audio element and the type of sound source it represents.

[0011] In some cases, the content creator can decide what desired rendering behavior should be applied by explicitly setting rendering parameters of the metadata of the audio element that controls the desired rendering behavior. In other cases, it is preferred that the renderer automatically selects the most appropriate rendering parameters for the audio element, or a pre-processing step can set the parameters so that the renderer uses a suitable gain function. However, there is currently no such automatic rendering method available.

[0012] Thus, in one aspect, a method for rendering an audio element, such as a stereo audio element comprising two or more audio signals, is provided. In one embodiment, the method includes obtaining a distance gain model rendering parameter associated with the stereo audio element. The method further includes selecting a distance gain model from a set of two or more candidate distance gain models based on a value of the distance gain model rendering parameter. The method further includes rendering the stereo audio element using the selected distance gain model.

[0013] In another embodiment, the method includes obtaining a spatial audio value S of an audio element (e.g., a stereo audio element), wherein S indicates a spatial audio density of the stereo audio element. The method further includes selecting one or more rendering options for the stereo audio element based on the obtained spatial audio value. The method further includes rendering the stereo audio element using the selected rendering options.

[0014] In another aspect, a method performed by an encoder is provided. The method includes obtaining a spatial audio value S for a stereo audio element, wherein S indicates a spatial audio density of the audio element. The method further includes at least one of the following two steps: (1) selecting one or more rendering options for the stereo audio element based on the obtained spatial audio value or (2) processing metadata for the stereo audio element (e.g., storing metadata, sending metadata, etc.), wherein the metadata includes at least one of the following: i) information identifying the selected rendering option and / or ii) the spatial audio value.

[0015] In another aspect, a computer program is provided, the computer program comprising instructions, when the instructions are executed by a processing circuit of a device, the instructions cause the device to perform any of the methods described herein. In one embodiment, a carrier containing the computer program is provided, wherein the carrier is one of an electronic signal, an optical signal, a radio signal, and a computer-readable storage medium. In another aspect, an apparatus is provided, the apparatus being configured to perform any of the methods described herein. The apparatus may include a memory and a processing circuit coupled to the memory.

[0016] An advantage of the embodiments disclosed herein is that they enable automatic selection of a suitable rendering behavior (e.g., a gain function) for an audio element based on the spatial characteristics of an audio signal representing the audio element. The analysis step of the method can be performed at runtime within a renderer or as a pre-processing step prior to runtime rendering. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings, which are incorporated herein and constitute a part of the specification, illustrate various embodiments.

[0018] Figure 1 A system according to an embodiment is shown.

[0019] Figure 2 is a flow chart illustrating a process according to some embodiments.

[0020] Figure 3 is a flow chart illustrating a process according to some embodiments.

[0021] Figure 4A and 4B A system according to some embodiments is shown.

[0022] Figure 5 A system according to some embodiments is shown.

[0023] Figure 6 A signal modifier according to an embodiment is shown.

[0024] Figure 7 is a block diagram of an apparatus according to some embodiments.

[0025] Figure 8 is a flow chart illustrating a process according to some embodiments. DETAILED DESCRIPTION

[0026] Figure 1A system 100 according to one embodiment is shown. The system 100 includes an encoder 102 and an audio renderer 104 (or simply renderer 104), wherein the encoder 102 can communicate with the renderer 104 via a network 110 (e.g., the Internet or other network) (the encoder and renderer can also be located in the same computing device or implemented in the same computing device). In one embodiment, the encoder 102 generates a bitstream including an audio signal and metadata, and provides the bitstream to the renderer 104 (e.g., sending the bitstream via the network 110). In some embodiments, instead of providing the bitstream directly to the renderer 104, the bitstream is stored in a data storage unit 106, and the renderer retrieves the bitstream from the data storage unit 106. The renderer 104 can be part of a device 103 (e.g., a smartphone, a television, a computer, an XR device) having an audio output device 105 (e.g., one or more speakers).

[0027] Among other things, the present disclosure proposes selecting a rendering option (e.g., selecting a particular distance gain model) for use in rendering an audio signal of a stereo audio element (i.e., an audio element having a spatial extent) by determining a spatial audio value (S) of the audio element and selecting an appropriate rendering option for the audio element based on S. For example, an encoder may select a particular distance gain model to use in rendering the stereo audio element by setting a "distanceGainModel" rendering parameter for the stereo audio element to a particular value identifying a particular distance gain model based on the value of S (e.g., if S>threshold, then distanceGainModel=1, otherwise distanceGainModel=0). In one embodiment, S is determined by analyzing the audio signal of the audio element and measuring a level of spatial sparsity (or spatial density) of the audio signal.

[0028] introduce

[0029] Certain aspects of the rendering of stereo audio elements are based on certain assumptions about the spatial characteristics of the audio elements. An example of this is the modeling of distance gain, which can be based on one of three main ideal models:

[0030] 1) Point source, where the sound propagation from the source is modeled as a spherical propagation from a point in all directions. An example could be the sound of a bird;

[0031] 2) Line sources, where sound propagation is modeled as cylindrical propagation from a linear object (e.g., the sound of a train); and

[0032] 3) Surface sources, where sound propagation is modeled as surface waves (e.g., a wide and tall waterfall).

[0033] In practice, stereo audio elements rarely, if ever, have the exact distance gain behavior of any of these three "prototype" sources. International patent application WO2021 / 121698 describes a more realistic distance gain model for stereo audio elements that derives the physically correct distance gain at any given distance from a stereo audio element of a given size based on the size of the audio element. This model will further be referred to as the "stereo distance gain model".

[0034] Line, surface, and stereo source models are based on the assumption that the sound source can be considered as an infinite collection of uniformly distributed point sources, i.e., the source is continuous. However, this assumption does not hold if the sound signal of an audio element represents a single point source or a recording of several point sources sparsely distributed on a line or surface. In this case, the point source sound propagation model should be used to render the audio element even if the audio element has a large range.

[0035] Another aspect of rendering of stereo sources that may be affected to some extent by the validity of the "continuous source" assumption is the "effective spatial range" aspect described in International Patent Application No. WO2022 / 017594, which describes a method of determining the acoustically relevant portion of a stereo source for a listener at a given listening distance relative to the source based on the size of the source using the continuous source assumption, and modifying the effective size of the source used in the rendering accordingly. Here, if the audio element consists of a sparse source distribution rather than a continuous source distribution, then applying a modification of the effective range size may be inappropriate.

[0036] WO2021 / 121698 describes the use of metadata that indicates whether a renderer should render stereo audio elements using a stereo distance gain model based on the audio element size, or whether the sources should instead be rendered using a point source distance gain.

[0037] As described herein, the above-mentioned metadata described in WO2021 / 121698 for controlling the distance gain behavior of a stereo audio element can be selected based on a spatial audio value (e.g., a spatial sparsity value or a spatial density value) for the stereo audio element, which can be determined as described in more detail below.

[0038] WO2022 / 017594 describes metadata indicating whether a renderer uses the described effective spatial range model for a stereo source, depending on, for example, whether the stereo source radiates sound from its entire extent or is just a conceptual volume containing a limited number of individual sound sources. The above metadata described in WO2022 / 017594 for controlling the effective spatial range of a stereo audio element can be selected based on a spatial audio value for the stereo audio element.

[0039] Definition of spatial density / sparseness

[0040] The spatial density of an audio signal (or its inverse, spatial sparsity) describes the degree to which the sound energy of the audio signal is evenly distributed, both spatially and temporally. A recorded audio signal representing a single point source has a low spatial density; similarly, a recorded audio signal representing multiple point sources where only one is active at a time also has a low spatial density. Conversely, a recorded audio signal representing multiple sound sources where individual sources are difficult to perceive has a high spatial density.

[0041] As an example, a recording of one person clapping their hands represents a signal with low spatial density. A recording of a group of people clapping their hands once also represents a signal with low spatial density. However, a recording of a group of people clapping their hands continuously represents a signal with high spatial density.

[0042] Spatial density is not a binary measurement, but a continuum between spatial sparseness and spatial density. Many sounds will lie in the middle, somewhere between the extreme points.

[0043] In one embodiment, deciding which rendering options to use for rendering a given stereo audio element is based on a binary decision, where a threshold is used to decide between which audio elements should be considered spatially dense and which audio elements should be considered spatially sparse. In another embodiment, there is no binary decision, but spatial audio values ​​(e.g., spatial sparsity / density measures) are used as weighting factors to enable a smooth transition from one rendering behavior representing a spatially sparse signal to another rendering behavior representing a spatially dense signal.

[0044] Preprocessing implementation

[0045] In one embodiment, the encoder 102 receives a stereo audio element (more specifically, the encoder receives metadata describing the stereo audio element plus a multi-channel audio signal representing the audio element). The stereo audio element may be an MPEG-I immersive audio element of type ObjectSource, which has a "signal" attribute identifying the multi-channel signal corresponding to the audio element, an "extent" attribute identifying a geometry data element representing the spatial extent of the audio element, and other attributes indicating, for example, the position, direction, and gain of the audio element.

[0046] The encoder 102 then derives a spatial audio value S (e.g., a spatial sparse value) for the audio element. Based on the spatial audio value, the encoder 102 selects one or more rendering options (e.g., a distance gain model). For example, the encoder 102 may select a distance gain model from a set of two or more candidate distance gain models by setting a distance gain model rendering parameter (denoted as "distanceGainModel") indicating that a particular one of a plurality of distance gain models should be used for the audio element. The distance gain model rendering parameter may be a Boolean parameter (flag) indicating whether a particular distance gain model (e.g., a stereo distance gain model) should be applied when rendering the audio element, or whether another particular distance gain model (e.g., a point source distance gain model) should be applied. Alternatively, the distance gain model rendering parameter may have a value that identifies one of a plurality of distance gain models.

[0047] The encoder then inserts the rendering parameters set based on S into the metadata of the audio element and / or the bitstream carrying the audio element (e.g., an MPEG-I immersive audio bitstream). Table 1 below illustrates the metadata for each audio element, where the distance gain model rendering parameter named "distanceGainModel" is a Boolean flag (i.e., 1 bit). Table 1

[0048] The renderer then receives the audio element and the distance gain model rendering parameters (e.g., it receives a bitstream containing the audio signal of the audio element and accompanying metadata containing the distance gain model rendering parameters and range information), and renders the audio element using the distance gain model indicated by the distance gain model rendering parameters.

[0049] In another embodiment, instead of selecting a rendering option based on the spatial audio value, the encoder 102 simply includes the spatial audio value S in the metadata of the audio element, and the renderer selects one or more rendering options (e.g., a distance gain model) based on the spatial audio value after receiving the spatial audio value.

[0050] As described above, the encoder 102 selects one or more rendering options based on the spatial audio value. Thus, in addition to (or instead of) selecting a distance gain model, the encoder 102 can use the spatial audio value S to select other rendering options (e.g., to configure or control other rendering aspects of a stereo source).

[0051] For example, the spatial audio value can be used to decide whether the renderer should apply a model for modifying the effective spatial range when rendering an audio element. As in the example of the distance gain model above, the encoder can derive a Boolean rendering parameter that indicates whether the renderer applies the effective spatial range model, or it can send the spatial audio value directly, in which case the renderer makes the decision based on the spatial audio value.

[0052] In some embodiments, instead of the encoder 102 determining the spatial audio value, the renderer 104 may determine the spatial audio value and then select one or more rendering options based on the spatial audio value. In addition, any device external to the encoder 102 and the renderer 104 may be configured to calculate the spatial audio value (or assist in calculating the spatial audio value).

[0053] Calculate spatial audio value (S)

[0054] In one embodiment, the first step of calculating S of an audio element is to filter the audio signal (X[nC,nT]) of the audio element using a filter (e.g., a high-pass filter), where nC is the number of audio channels included in the audio signal and nT is the number of samples in each audio channel to generate a filtered audio signal (X'[c,t]).

[0055] In one embodiment, where X′[c,t] is the multi-channel audio signal filtered by h(n) at channel c and time t, L is the filter length, and h[n] is the filter coefficient.

[0056] The high-pass filter (h[n] in equation (1) above) enables directionality and sparsity analysis to be better in the domain representing subjective loudness than in the unfiltered domain.

[0057] It was experimentally found that a filter with a -3dB point at 1478Hz and a -6dB point at 1000Hz is suitable for use as a pre-processing high-pass filter (h[n]), however other high-pass filter designs can also be used as a subjectively driven pre-processing high-pass filter: for example, using a 3dB design point in the range = [400Hz-2kHz].

[0058] Table 2 below shows an example frequency response of h(n):

[0059] This particular filter is in Matlab TMThe following was used for design: Fd=1000 design; N=40; Fd_norm=Fd / (48000 / 2); beta=5.0; % FIR filter design using Kaiser window h[n]=fir1(N,Fd_norm,'high',kaiser(N+1,beta)).

[0060] The filtered audio signal is then divided into frames according to Formula 2: x i [c, t] = X′[c, i*F+t]; t∈[0,…,F-1] (Formula 2) Where F is the frame length, and i is the frame number.

[0061] An example frame length F is 100 milliseconds, corresponding to F = 4800 samples at a sampling frequency of 48 kHz.

[0062] The average energy of each frame i across all channels c is calculated for the final normalization of the sparsity measure, and the average energy is calculated as follows:

[0063] Spatial Analysis and Directivity Ratios:

[0064] For each frame i, a singular value decomposition (SVD) analysis is performed. SVD is a technique that decomposes the signal into a matrix U, a diagonal matrix Σ with diagonal D, and a matrix V T That is, SVD is a technique for decomposing a matrix into a collection of component matrices, and can be used to reveal interesting properties of the original matrix. X=UΣV T , (Formula 4) (The x above here i is the matrix X for frame i). D j =[Σ 1,1 ,Σ 2,2 ,…Σ Nc,Nc ,](Formula 5)

[0065] Computing the SVD involves finding XX T and X T The eigenvalues ​​and eigenvectors of X. T The eigenvectors of X form the columns of V, XX T The eigenvectors of form the columns of U. In addition, the singular values ​​in Σ are XX T or X T The square roots of the eigenvalues ​​of X. The singular values ​​are the diagonal entries of the Σ matrix and are usually arranged in descending order. The singular values ​​are always real numbers. If the input matrix X is real, then U and V are also real.

[0066] In Python (v 3.8.2, module numpy.linalg), SVD analysis can be done as follows: D = numpy.linalg.svd(X, full_matrices = False, compute_uv = False). TM In version R2018b, singular value decomposition analysis can be performed as follows: D = svd(X).

[0067] For each frame i, the relationship between the extreme diagonal elements is calculated as follows: dirDB i =10*log10(dir i )(Formula 7)

[0068] dir i is a measure of the directional dynamics of a multichannel signal X, dir i The range of dir is usually [1.0…12.0]. i It can be thought of as the strength relationship between the principal axis and the most diffuse (and weakest) axis.

[0069] The Mid-Side relationship of each frame i is calculated as follows:

[0070] Finally, the average energy relationship between multichannel-mid and multichannel-side is MSenRel i is calculated as follows:

[0071] MSenRel i The range is usually [.0 1…22.0]. Now, the ratio MSenRel i is the ratio of low-complexity mid to average-side, which complements the dynamic range ratio dir in identifying frame-by-frame changes between input channels in the spatial domain. i .

[0072] Then, the score value for each frame i is calculated as: score i =dir i *MSenRel i (Formula 14)

[0073] For frame i, a score of 1.0 (0dB) iIndicates a very dense signal, and a score in the range of 120.0 (about 21dB) i Indicates a very sparse signal.

[0074] If the ratio dir of frame i i *MSenRel i In the logarithmic domain, the fractional calculation will be an addition process, as shown in Equation 15: scoreDB i =dirDB i +en_relDB i (Formula 15)

[0075] The next step is to calculate the total energy as:

[0076] The total energy is used to normalize the frame-by-frame measurements across the entire set of analysis frames.

[0077] Next, the relative energy is calculated as shown in Equation 17:

[0078] Frame parameter relativeEnergy i The individual weight of frame i is now normalized to all frames in the analyzed multi-channel audio signal.

[0079] For each frame i, the original score for a given frame is i and relative energy relativeEnergy i Weighted to generate energy normalization score normScore i , as shown in Formula 18: normScore i =score i *relativeEnergy i (Formula 18)

[0080] Finally, the spatial audio value S is calculated according to formula 19:

[0081] Once S is determined, S can be used to select one or more rendering options. In one embodiment, the rendering option is selected by comparing S to a threshold (denoted as "SparseLim" or "SparseLimDB"). In one embodiment, SparseLim == 20 and SparseLimDB=10*log10(SparseLim)==13.01dB. For example, if S≥SparseLim, the distanceGainModel parameter is set to 1, otherwise it is set to 0. Therefore, in this example, S is a spatial sparsity value, where values ​​of S greater than SparseLim correspond to spatially sparse audio signals, and values ​​of S less than SparseLim correspond to spatially dense audio signals. That is, the sum of the relative energy-weighted scores is thresholded to obtain a sparse versus dense decision for a given audio file.

[0082] Multiple discrete rendering methods

[0083] For cases where more than two discrete rendering options are available, the spatial audio value S can be split / quantized into a linear range. For example, for a case with three available distance gain models (dgm1, dgm2, dgm3), the following logic can be used to select one of the three distance gain models: if S≤10, select dgm1; otherwise, if S≤30, select dgm2; otherwise, select dgm3.

[0084] That is, in general, based on a trained linear quantizer, the spatial audio value S can be quantized (essentially segmented) into a specific number of target rendering methods.

[0085] Rendering solution based on weighting factors

[0086] To enable a smooth transition from one rendering behavior to another, you can smoothly transition between distance gain based rendering methods by using S to select a weighting factor value s_wfactor in the range of 0.0 to 1.0, as shown below: s_wfactor=min(1.0,max(0.0,S / 40))(Formula 20)

[0087] For example, the final distance gain g DistFinal can be calculated as g DistFinal =s_wfactor*g Dist1 +(1-s_wfactor)*g Dist2 , Among them, the final distance gain (g DistFinal ) is calculated as g Dist1 and gDist2 , which are the distance gains calculated using different distance gain models.

[0088] Compute spatial sparsity / density analysis in real time:

[0089] In cases where the input audio signal varies over time and therefore may contain varying degrees of sparsity, it is beneficial to adaptively change the available rendering options used over time. There are two basic approaches that are proposed to apply: (A) sliding window analysis, with decisions updated every Nupd frames corresponding to a window width Tupd (seconds), and (B) continuous analysis, updated every frame.

[0090] (A) Sliding window analysis with a predetermined block / window length.

[0091] Using this real-time solution, the audio input can be analyzed over time Tana (producing Nana time samples for each channel), possibly with an overlap of Toverlap (producing Noverlap time samples for each channel). Typically, Toverlap = Tana / 2 is used to achieve smooth updates of the analyzed sparsity.

[0092] Basic example: Tana and Toverlap are for example 6 seconds and 3 seconds respectively. The main advantage of this approach is that the sparsity evaluation can be run as a lower priority background process and then applied when ready. The disadvantage is that there will be a delay due to the input signal windowing, which will cause the rendering to be suboptimal for a short period of time (application of the delay in the sound scene). In addition, the input signal features analyzed for the block-by-block window will need to be buffered.

[0093] That is, for the sliding window analysis, the same formula as described by the pre-processing will be used, but run every Tupdate = Tana - Toverlap and applied when the analysis process is ready. Table 3 Example timing for method (A)

[0094] (B) Continuously updating (e.g., updating the sparsity analysis at the same rate as the graphics engine generates new images (e.g., in the 30-90 Hz range) or at the rate at which the sound environment update driver runs (e.g., 200 Hz, 100 Hz, 50 Hz, or 10 Hz)).

[0095] In order to save the time used for (dir i and energy i), preferably applying an autoregressive process to estimate the long-term feature total energy (totalEnergy in the preprocessing solution) and the past total accumulated scores i .

[0096] The storage and memory consumption implementation defined previously uses the following summation:

[0097] It can be expressed as follows via a dual memory efficient autoregressive 1st order (AR(1)) process:

[0098] First, a running AR(1) estimate of the long-term frame average energy is updated every frame: MeanEn AR (i) = α * MeanEn AR (i-1)+(1-α)*energy i (Formula 22) Starting Condition: MeanEn AR (-1) = set to energy 0 .

[0099] Experimentally, it was found that for a frame size of 100 milliseconds: Where i is the frame number starting from zero, N AR-init = 60 is the initialization frame limit corresponding to a 6 second state initialization phase. This AR(1) setting yields a good estimate of the average frame energy: Among them, N AR-integration is the total frame energy integrated over the frame duration, e.g. N AR-integration can be set to 80 to correspond to an 8 second energy integration for SVD and a 100 millisecond Mid-Side analysis frame size. That is, the product N AR-integration *MeanEn AR (i) corresponds to the totalEnergy parameter in the pre-processing solution above, where the total energy is chosen to roughly correspond to the last 8 seconds of the analyzed audio. normScore AR (i) = β * normScore AR (i-1)+(1-β)*score i *relativeEnergy AR (i) (Formula 25) Starting condition normScore AR (-1) = set to score 0, the AR(1) filter coefficient β is set according to the following formula:

[0100] Then the obtained score normScore AR (i) Thresholding is performed to obtain the following binary discrete decisions for each frame used in the spatial audio rendering method: S=N AR-integration *normScore AR (i) (Formula 27) Which then uses spatiallyDense AR (i) to control the rendering method used for the current frame.

[0101] Figure 2 is a flow chart illustrating a process 200 for rendering a stereo audio element according to an embodiment. The process 200 may begin at step s202. Step s202 includes obtaining a spatial audio value S for a stereo audio element, wherein S indicates a spatial audio density of the audio element. Step s204 includes selecting one or more rendering options for the stereo audio element based on S. Step s206 includes rendering the stereo audio element using the selected rendering options. In some embodiments, obtaining S includes receiving metadata including S.

[0102] Figure 3 is a flow chart illustrating a process 300 performed by the encoder 102 according to an embodiment. The process 300 may start at step s302. Step s302 includes obtaining a spatial audio value S for a stereo audio element, wherein S indicates a spatial audio density of the stereo audio element. After step s302, step s304 and / or step s306 are performed. Step s304 includes selecting one or more rendering options for the stereo audio element based on S. Step s306 includes processing metadata for the stereo audio element (e.g., storing metadata, sending metadata, etc.), wherein the metadata includes at least one of: i) information identifying the selected rendering option and / or ii) the spatial audio value.

[0103] In some embodiments, obtaining S comprises calculating S using an audio signal of the stereo audio element or a portion of the audio signal.

[0104] In some embodiments, the process comprises a processing step (step s306), the stereo audio element has a spatial extent, and the metadata further comprises information indicating the spatial extent.

[0105] In some embodiments, the process includes a selection step (step s204 or s304), and one of the one or more selected rendering options is a distance gain model selected from a set of candidate distance gain models. In some embodiments, selecting the distance gain model includes or consists of: setting a distance gain model rendering parameter to a specific value identifying the distance gain model. In some embodiments, selecting the distance gain model includes: determining whether S satisfies a condition; and as a result of determining that S satisfies the condition, setting the distance gain model rendering parameter to a first value identifying the distance gain model. In some embodiments, determining whether S satisfies the condition includes: comparing S to a threshold.

[0106] In some embodiments, the process comprises a selecting step, and one of the one or more selected rendering options is a selected weighting factor value s_wfactor.

[0107] In some embodiments, the audio signal of the stereo audio element includes at least a first set of audio frames, and obtaining S includes: i) for each audio frame included in the first set of audio frames, calculating a normalized score for the audio frame, and ii) adding the normalized scores to produce a sum, wherein S is equal to the sum. In some embodiments, calculating the normalized score for the audio frame includes calculating a score for the audio frame and calculating a relative energy value for the audio frame, wherein the normalized score is equal to the product of the score and the relative energy value.

[0108] Figure 8 8 is a flow chart illustrating a process 800 for rendering an audio element (e.g., a stereo audio element) comprising two or more audio signals according to an embodiment. The process 800 may begin at step s802. Step s802 includes obtaining a distance gain model rendering parameter associated with the stereo audio element, the obtained distance gain model rendering parameter having a value. Step s804 includes selecting a distance gain model from a set of two or more candidate distance gain models based on the value of the distance gain model rendering parameter. Step s806 includes rendering the stereo audio element using the selected distance gain model.

[0109] In some embodiments, the value of the distance gain model rendering parameter is set based on an analysis of two or more audio signals associated with the stereo audio element.

[0110] In some embodiments, process 800 further includes generating a spatial audio value S based on an analysis of two or more audio signals; determining whether S satisfies a condition; and as a result of determining that S satisfies the condition, setting the distance gain model rendering parameter to the value. In some embodiments, S indicates spatial sparsity of the stereo audio element.

[0111] In some embodiments, obtaining the distance gain model rendering parameter includes obtaining metadata for a stereo audio element, and the metadata includes i) a stereo audio element identifier identifying the stereo audio element and ii) the distance gain model rendering parameter. In some embodiments, the metadata further includes: a first diffuseness parameter indicating whether the metadata further includes a second diffuseness parameter. In some embodiments, the metadata further includes a diffuseness parameter indicating a diffuseness of the stereo audio element.

[0112] Example Use Case

[0113] Figure 4A An XR system 400 is shown in which embodiments disclosed herein may be applied. The XR system 400 includes speakers 404 and 405 (which may be speakers of headphones worn by a listener) and an XR device 410, which may include a display for displaying images to a user, and in some embodiments, is configured to be worn by the listener. In the illustrated XR system 400, the XR device 410 has a display and is designed to be worn on the user's head and is generally referred to as a head-mounted display (HMD).

[0114] like Figure 4B As shown in the figure, the XR device 410 may include a direction sensing unit 401, a position sensing unit 402 and a processing unit 403, wherein the processing unit 403 is (directly or indirectly) coupled to the audio renderer 104 for generating output audio signals (for example, as shown in the figure, a left audio signal 481 for a left speaker and a right audio signal 482 for a right speaker).

[0115] The direction sensing unit 401 is configured to detect changes in the direction of the listener and provide information about the detected changes to the processing unit 403. In some embodiments, the processing unit 403 determines an absolute direction (relative to a certain coordinate system) based on the changes in the direction detected by the direction sensing unit 401. There may also be different systems for determining direction and position, such as systems using lighthouse trackers (lidar). In one embodiment, the direction sensing unit 401 may determine an absolute direction (relative to a certain coordinate system) based on the changes in the detected direction. In this case, the processing unit 403 may simply multiplex the absolute direction data from the direction sensing unit 401 and the position data from the position sensing unit 402. In some embodiments, the direction sensing unit 401 may include one or more accelerometers and / or one or more gyroscopes.

[0116] The audio renderer 104 generates an audio output signal based on an input audio signal 461, metadata 462 about the XR scene that the listener is experiencing, and information 463 about the position and direction of the listener. The input audio signal 461 and the metadata 462 may come from a source 452, which may be remote from the audio renderer 104 or may be co-located with the audio renderer. The metadata 462 for the XR scene may include metadata for each object and audio element contained in the XR scene, and the metadata for the stereo audio element may include information about the range of the audio element and the rendering parameters discussed above (e.g., one or more parameters for controlling the distance gain model for rendering the audio element). The metadata 462 for objects in the XR scene may also include control information, such as reverberation time values, reverberation level values, and / or absorption parameters. The audio renderer 104 may be a component of the XR device 410, or it may be remote from the XR device 410 (e.g., the audio renderer 104 or its components may be implemented in a so-called "cloud").

[0117] Figure 5 An example implementation of an audio renderer 104 for generating sounds for an XR scene is shown. The audio renderer 104 includes a controller 501 and a signal modifier 502, the signal modifier 502 being used to modify an audio signal 461 (e.g., an audio signal of a multi-channel stereo audio element) based on control information 510 from the controller 501. The controller 501 may be configured to receive one or more parameters, and to trigger the modifier 502 to perform a modification (e.g., increase or decrease the volume) on the audio signal 461 based on the received parameters. The received parameters include information 463 about the position and / or direction of the listener (e.g., the direction and distance to the audio element) and metadata 462 about the audio element in the XR scene (in some embodiments, the controller 501 itself generates the metadata 462 or a portion thereof). Using the metadata (e.g., one or more distance gain model control parameters) and the position / direction information, the controller 501 may calculate one or more gain factors (g) (also known as attenuation factors) for the audio elements in the XR scene.

[0118] Figure 6An example implementation of a signal modifier 502 according to one embodiment is shown. The signal modifier 502 includes a directional mixer 604, a gain adjuster 606, and a speaker signal generator 608. The directional mixer receives an audio input 461, which in this example includes a pair of audio signals 601 and 602 associated with an audio element, and generates a set of k virtual speaker signals (VS1, VS2, ..., VSk) based on the audio input and control information 691. In one embodiment, a signal for each virtual speaker can be obtained by, for example, appropriately mixing the signal including the audio input 461. For example: VS1 = α × L + β × R, where L is the input audio signal 601, R is the input audio signal 602, and α and β are factors depending on, for example, the position of the listener relative to the audio element and the position of the virtual speaker corresponding to VS1. The gain adjuster 606 can adjust the gain of any one or more of the virtual speaker signals based on the control information 692.

[0119] Using the virtual speaker signals VS1, VS2, ..., VSk, the speaker signal generator 608 generates output signals (e.g., output signals 481 and 482) for driving speakers (e.g., headphone speakers or other speakers). In an embodiment where the speakers are headphone speakers, the speaker signal generator 608 can perform conventional binaural rendering to generate the output signals. In an embodiment where the speakers are not headphone speakers, the speaker signal generator can perform conventional speaking panning to generate the output signals.

[0120] Figure 7 700 is a block diagram of an apparatus 700 for performing any of the methods disclosed herein according to some embodiments. When the apparatus 700 is configured to perform an encoding method (e.g., process 300), the apparatus 700 may be referred to as an encoding apparatus; similarly, when the apparatus 700 is configured to perform an audio rendering method (e.g., process 200), the apparatus 700 may be referred to as an audio rendering apparatus. Figure 7As shown in FIG. 7 , the device 700 may include: a processing circuit (PC) 702, which may include one or more processors (P) 755 (e.g., one or more general-purpose microprocessors and / or one or more other processors, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), etc.), which may be co-located in a single housing or in a single data center, or may be geographically distributed (i.e., the device 700 may be a distributed computing device); at least one network interface 748 (e.g., a physical interface or an air interface), which includes a transmitter (Tx) 745; and a receiver (Rx) 747 for enabling the apparatus 700 to send and receive data to and from other nodes connected to a network 110 (e.g., an Internet Protocol (IP) network), the network interface 748 being connected (physically or wirelessly) to the network 110 (e.g., the network interface 748 may be coupled to an antenna device including one or more antennas for enabling the apparatus 700 to wirelessly send / receive data); and a storage unit (also known as a "data storage system") 708, which may include one or more non-volatile storage devices and / or one or more volatile storage devices. In an embodiment where the PC 702 includes a programmable processor, a computer readable storage medium (CRSM) 742 may be provided. The CRSM 742 may store a computer program (CP) 743 including computer readable instructions (CRI) 744. The CRSM 742 may be a non-transitory computer readable medium, such as a magnetic medium (e.g., a hard disk), an optical medium, a storage device (e.g., a random access memory, a flash memory), etc. In some embodiments, the CRI 744 of the computer program 743 is configured so that when executed by the PC 702, the CRI causes the device 700 to perform the steps described herein (e.g., the steps described herein with reference to the flowcharts). In other embodiments, the device 700 may be configured to perform the steps described herein without code. That is, for example, the PC 702 may consist solely of one or more ASICs. Thus, the features of the embodiments described herein may be implemented in hardware and / or software.

[0121] Overview of Various Embodiments

[0122] A1. A method 200 for rendering a stereo audio element, comprising: obtaining a spatial audio value S for a stereo audio element, wherein S indicates a spatial audio density of the audio element; selecting one or more rendering options for the stereo audio element based on the obtained spatial audio value; and rendering the stereo audio element using the selected rendering options.

[0123] A2. The method of embodiment A1, wherein obtaining S comprises: calculating S using an audio signal of a stereo audio element or a portion of the audio signal.

[0124] A3. The method of embodiment A1, wherein obtaining S includes receiving metadata including S.

[0125] A4. The method of any of embodiments A1-A3, wherein one of the one or more selected rendering options is a distance gain model selected from a set of candidate distance gain models.

[0126] A5. The method of embodiment A4, wherein selecting the distance gain model includes or consists of setting a distance gain model rendering parameter to a specific value that identifies the distance gain model.

[0127] A6. The method of embodiment A4, wherein selecting the distance gain model comprises: determining whether S satisfies a condition; and as a result of determining that S satisfies the condition, setting the distance gain model rendering parameter to a first value identifying the distance gain model.

[0128] A7. The method of embodiment A6, wherein determining whether S satisfies a condition comprises: comparing S with a threshold.

[0129] A8. The method of any one of embodiments A1-A7, wherein one of the one or more selected rendering options is a selected weighting factor value s_wfactor.

[0130] A9. A method as described in any of embodiments A1-A8, wherein the audio signal of the stereo audio element includes at least a first set of audio frames, and obtaining S includes: i) for each audio frame included in the first set of audio frames, calculating a normalized score for the audio frame, and ii) adding the normalized scores to produce a sum, wherein S is equal to the sum.

[0131] A10. The method of embodiment A9, wherein calculating a normalized score for an audio frame comprises calculating a score for the audio frame and calculating a relative energy value for the audio frame, wherein the normalized score is equal to the product of the score and the relative energy value.

[0132] B1. A method 300 performed by an encoder, comprising: obtaining a spatial audio value S for a stereo audio element, wherein S indicates a spatial audio density of the audio element; and performing at least one of the following: selecting one or more rendering options for the stereo audio element based on the obtained spatial audio value; or processing metadata for the stereo audio element (e.g., storing metadata, sending metadata, etc.), wherein the metadata includes at least one of the following: i) information identifying the selected rendering option and / or ii) the spatial audio value.

[0133] B2. The method of embodiment B1, wherein obtaining S comprises: calculating S using an audio signal of a stereo audio element or a portion of the audio signal.

[0134] B3. The method of any one of embodiments B1-B2, wherein the method includes a metadata processing step, the stereo audio element has a spatial range, and the metadata further includes information indicating the spatial range.

[0135] B4. The method of any one of embodiments B1-B3, wherein the method includes a selection step, and one of the one or more selected rendering options is a distance gain model selected from a set of candidate distance gain models.

[0136] B5. The method of embodiment B4, wherein selecting the distance gain model includes or consists of setting a distance gain model rendering parameter to a specific value that identifies the distance gain model.

[0137] B6. The method of embodiment B4, wherein selecting the distance gain model includes: determining whether S satisfies a condition; and as a result of determining that S satisfies the condition, setting the distance gain model rendering parameter to a first value that identifies the distance gain model.

[0138] B7. The method of embodiment B6, wherein determining whether S satisfies a condition comprises: comparing S with a threshold.

[0139] B8. The method of any one of embodiments B1-B7, wherein the method includes a selection step, and one of the one or more selected rendering options is a selected weighting factor value s_wfactor.

[0140] B9. A method as described in any of embodiments B1-B8, wherein the audio signal of the stereo audio element includes at least a first set of audio frames, and obtaining S includes: i) for each audio frame included in the first set of audio frames, calculating a normalized score for the audio frame, and ii) adding the normalized scores to produce a sum, wherein S is equal to the sum.

[0141] B10. The method of embodiment B9, wherein calculating a normalized score for an audio frame comprises calculating a score for the audio frame and calculating a relative energy value for the audio frame, wherein the normalized score is equal to the product of the score and the relative energy value.

[0142] C1. A computer program 743, comprising instructions 744. When the instructions 744 are executed by the processing circuit 702 of the device 700, the instructions 744 cause the device to perform the method of any of the above embodiments.

[0143] C2. A carrier containing the computer program of embodiment C1, wherein the carrier is one of an electronic signal, an optical signal, a radio signal and a computer-readable storage medium 742.

[0144] D1. A device 700, configured to execute the method of any of the above embodiments.

[0145] D2. The apparatus of embodiment D1, wherein the apparatus comprises a memory 742 and a processing circuit 702 coupled to the memory.

[0146] Although various embodiments are described herein, it should be understood that they are presented by way of example only, not limitation. Therefore, the breadth and scope of the present disclosure should not be limited by any of the above-described exemplary embodiments. In addition, unless otherwise specified herein or the context is clearly contradictory, otherwise, the present disclosure encompasses any combination of the above-described objects in all possible variations thereof.

[0147] In addition, although the processes described above and shown in the drawings are shown as a series of steps, this is only for illustrative reasons. Therefore, it is conceivable to add some steps, omit some steps, rearrange the order of the steps, and perform some steps in parallel.

[0148] References

[0149] [1]MPEG-H 3D Audio,ISO / IEC 23008-3:201x(E)

[0150] [2]EBU Tech 3388, "ADM Renderer for use in Next Generation AudioBroadcasting," BTF Renderer Group, Specification Version 1.0, Geneva, March 2018.

[0151] [3]Efficient HRTF-based Spatial Audio for Area and Volumetric Sources", IEEE Transactions on Visualization and Computer Graphics 22(4):1-1, January 2016.

[0152] [4]Patent Publication WO2020144062, “Efficient spatially-heterogeneousaudio elements for Virtual Reality.”

[0153] [5]Patent Publication WO2021180820,“Rendering of Audio Objects with aComplex Shape.”

[0154] [6]Patent Publication WO2022017594,“SPATIAL EXTENT MODELING FORVOLUMETRIC AUDIO SOURCES.”

[0155] [7]Patent Publication WO2021121698,“AUDIO RENDERING OF AUDIOSOURCES.”

Claims

1. A method (800) for rendering a stereo audio element comprising two or more audio signals, the method comprising: obtaining (s802) a distance gain model rendering parameter associated with the stereo audio element, the obtained distance gain model rendering parameter having a value; selecting (s804) a distance gain model from a set of two or more candidate distance gain models based on said value of said distance gain model rendering parameter; and The stereo audio element is rendered (s806) using the selected distance gain model.

2. The method according to claim 1, wherein: The value of the distance gain model rendering parameter is set based on an analysis of the two or more audio signals associated with the stereo audio element.

3. The method according to claim 2, further comprising: generating a spatial audio value S based on said analysis of said two or more audio signals; Determine whether S satisfies the condition; as well as As a result of determining that S satisfies the condition, the distance gain model rendering parameter is set to the value.

4. The method according to claim 3, wherein: S indicates the spatial sparsity of the stereo audio element.

5. The method according to any one of claims 1 to 4, wherein: Obtaining the distance gain model rendering parameters comprises obtaining metadata for the stereo audio element, and The metadata includes: i) an audio element identifier identifying the audio element, and ii) the distance gain model rendering parameters.

6. The method according to claim 5, wherein: The metadata further includes a first diffusion parameter indicating whether the metadata further includes a second diffusion parameter.

7. The method according to claim 5, wherein: The metadata further includes a diffuseness parameter indicating diffuseness of the stereo audio element.

8. A method (200) for rendering a stereo audio element, comprising: Obtaining (s202) a spatial audio value S for the stereo audio element, wherein S indicates a spatial audio density of the stereo audio element; selecting (s204) one or more rendering options for the stereo audio element based on the obtained spatial audio value; and The stereo audio element is rendered (s206) using the selected rendering option.

9. The method according to claim 8, wherein: Obtaining S comprises: calculating S using an audio signal of the stereo audio element or a portion of the audio signal.

10. The method according to claim 8, wherein: Obtaining S includes: receiving metadata including S.

11. The method according to any one of claims 8 to 10, wherein: A rendering option of the one or more selected rendering options is a range gain model selected from a set of candidate range gain models.

12. The method according to claim 11, wherein: Selecting a distance gain model includes or consists of setting a distance gain model rendering parameter to a specific value identifying the distance gain model.

13. The method according to claim 11, wherein: Selecting the distance gain model includes: Determine whether S satisfies the condition; and As a result of determining that S satisfies the condition, a distance gain model rendering parameter is set to a first value identifying the distance gain model.

14. The method according to claim 13, wherein: Determining whether S satisfies the condition includes comparing S with a threshold.

15. The method according to any one of claims 8 to 14, wherein: One of the one or more selected rendering options is a selected weighting factor value s_wfactor.

16. The method according to any one of claims 8 to 15, wherein: The audio signal of the stereo audio element comprises at least a first set of audio frames, and Obtaining S includes: i) for each audio frame included in the first set of audio frames, calculating a normalized score for the audio frame, and ii) adding the normalized scores to produce a sum, wherein S is equal to the sum.

17. The method according to claim 16, wherein: Calculating a normalized score for the audio frame includes: calculating a score for the audio frame and calculating a relative energy value for the audio frame, wherein the normalized score is equal to a product of the score and the relative energy value.

18. A method (300) performed by an encoder (102), comprising: obtaining (s302) a spatial audio value S for a stereo audio element, wherein S indicates a spatial audio density of the audio element; and Do at least one of the following: selecting (s304) one or more rendering options for the stereo audio element based on the obtained spatial audio value; or Metadata for the stereo audio element is processed (s306), wherein the metadata comprises at least one of: i) information identifying the selected rendering option and / or ii) the spatial audio value.

19. The method according to claim 18, wherein: Obtaining S comprises: calculating S using an audio signal of the stereo audio element or a portion of the audio signal.

20. The method according to any one of claims 18-19, wherein: The method comprises the metadata processing step, The stereo audio element has a spatial extent, and The metadata further includes information indicating the spatial extent.

21. The method according to any one of claims 18 to 20, wherein: The method comprises the selecting (s304) step, and A rendering option of the one or more selected rendering options is a range gain model selected from a set of candidate range gain models.

22. The method according to claim 21, wherein: Selecting a distance gain model includes or consists of setting a distance gain model rendering parameter to a specific value identifying the distance gain model.

23. The method according to claim 21, wherein: Selecting the distance gain model includes: Determine whether S satisfies the condition; and As a result of determining that S satisfies the condition, a distance gain model rendering parameter is set to a first value identifying the distance gain model.

24. The method according to claim 23, wherein: Determining whether S satisfies the condition includes comparing S with a threshold.

25. The method according to any one of claims 18 to 24, wherein: The method comprises the selecting (s304) step, and One of the one or more selected rendering options is a selected weighting factor value s_wfactor.

26. The method according to any one of claims 18 to 25, wherein: The audio signal of the stereo audio element comprises at least a first set of audio frames, and Obtaining S includes: i) for each audio frame included in the first set of audio frames, calculating a normalized score for the audio frame, and ii) adding the normalized scores to produce a sum, wherein S is equal to the sum.

27. The method according to claim 26, wherein: Calculating the normalized score for the audio frame includes: calculating a score for the audio frame and calculating a relative energy value for the audio frame, wherein the normalized score is equal to a product of the score and the relative energy value.

28. A computer program (743) comprising instructions (744) which, when executed by processing circuitry (702) of an apparatus (700), cause the apparatus to perform the method of any preceding claim.

29. A carrier comprising the computer program of claim 28, wherein: The carrier is one of an electronic signal, an optical signal, a radio signal, and a computer-readable storage medium (742).

30. An apparatus (700) for rendering an audio element comprising two or more audio signals, the apparatus being configured to perform a method comprising: obtaining (s802) a distance gain model rendering parameter associated with the audio element, the obtained distance gain model rendering parameter having a value; selecting (s804) a distance gain model from a set of two or more candidate distance gain models based on said value of said distance gain model rendering parameter; and The audio element is rendered (s806) using the selected distance gain model.

31. The device according to claim 30, wherein The apparatus is further configured to perform the method of any one of claims 2-7.

32. An apparatus (700) for rendering a stereo audio element, the apparatus being configured to perform a method comprising: Obtain (s202) a spatial audio value S for the stereo audio element, wherein: S indicates the spatial audio density of the stereo audio element; selecting (s204) one or more rendering options for the stereo audio element based on the obtained spatial audio value; as well as The stereo audio element is rendered (s206) using the selected rendering option.

33. The device according to claim 32, wherein: The apparatus is further configured to perform the method of any one of claims 9-17.

34. An encoding device (700), the encoding device being configured to perform a method comprising the following operations: Obtain (s302) a spatial audio value S for a stereo audio element, where: S indicates the spatial audio density of the audio element; as well as Do at least one of the following: selecting (s304) one or more rendering options for the stereo audio element based on the obtained spatial audio value; or Metadata for the stereo audio element is processed (s306), wherein the metadata comprises at least one of: i) information identifying the selected rendering option and / or ii) the spatial audio value.

35. The device according to claim 34, wherein The apparatus is further configured to perform the method of any one of claims 19-27.

Images