Efficient Head-Related Filter Generation

Abstract

A method for generating a head-related (HR) filter for audio rendering is provided. The method includes generating HR filter model data indicative of an HR filter model, and (i) sampling one or more basis functions based on the generated HR filter model data, and (ii) generating first basis function shape data and shape metadata. The method further includes providing the generated first basis function shape data and shape metadata for storage in one or more storage media.

Description

[Technical field]

[0001] SUMMARY OF THE DISCLOSURE Embodiments are disclosed that relate to methods and systems for efficient head-related filter generation. [Background technology]

[0002] The human auditory system is equipped with two ears that capture sound (audio) waves propagating toward a listener. In this disclosure, the words "sound" and "audio" are used interchangeably. FIG. 1 shows sound waves propagating toward a listener from a direction of arrival (DOA) specified by a pair of elevation and azimuth angles in a spherical coordinate system. On its propagation path toward the listener, each sound wave interacts with the listener's upper torso, head, outer ear, and the listener's surrounding material before reaching the listener's left and right eardrums. This interaction produces temporal and spectral changes in the sound waveforms that reach the left and right eardrums, some of which are DOA dependent. The human auditory system has learned to interpret these changes to infer various spatial properties of the sound waves themselves, as well as the acoustic environment in which the listener is located. This ability is called spatial hearing, which relates to how a listener evaluates spatial cues embedded in the binaural signal, i.e., the sound signal in the right and left ear canal, to infer the location of an auditory event evoked by the sound event (physical sound source) and the acoustic characteristics caused by the physical environment the listener is in (e.g., a small room, a tiled bathroom, an auditorium, a windowless cave). This human ability, i.e., spatial hearing, can be exploited to create spatial audio scenes by reintroducing spatial cues into the binaural signal, which would result in a spatial perception of sound.

[0003] The main spatial cues include (1) angular cues: binaural cues, i.e., interaural level difference (ILD) and interaural time difference (ITD), as well as monaural (or spectral) cues, and (2) distance cues: intensity and direction-to-reverberation (D / R) energy ratio. The mathematical representation of the short-time (e.g., 1-5 ms) DOA-dependent or angular temporal and spectral changes of a waveform are so-called head-related (HR) filters. The frequency-domain (FD) representation of the HR filters is the so-called head-related transfer function (HRTF), and the time-domain (TD) representation of the HR filters is the so-called head-related impulse response (HRIR). Figure 2 shows the sound waves propagating towards the listener and the difference in the sound paths to the two ears, which gives rise to the ITD. Figure 14 shows an example of the spectral cues (HR filters) of the sound waves shown in Figure 2. The two plots shown in Figure 14 show the magnitude response of a pair of HR filters taken at 0 degrees elevation (θ) and 40 degrees azimuth (φ). The data is from the Center for Image Processing and Integrated Computing (CIPIC) database: subject ID 28. This database is publicly available and can be accessed from the link https: / / www.ece.ucdavis.edu / cipic / spatial-sound / hrtf-data / .

[0004] HR filter-based binaural rendering techniques are gradually being established, where spatial audio scenes are generated by directly filtering audio source signals with pairs of HR filters at desired locations. This approach is particularly attractive for many emerging applications such as virtual reality (VR), augmented reality (AR) or mixed reality (MR) (sometimes collectively referred to as extended reality (XR)), and mobile communication systems where headsets are typically used.

[0005] HR filters are often estimated from measurements as impulse responses of a linear dynamic system that transforms original sound signals (i.e., input signals) into left and right ear signals (i.e., output signals), which may be measured in the ear channels of a listening subject (e.g., an artificial head, a mannequin, or a human subject) at a predefined set of elevation and azimuth angles on a sphere of constant radius from the listening subject. The estimated HR filters are often provided as finite impulse response (FIR) filters and may be used directly in that format. To achieve efficient binaural rendering, pairs of HRTFs may be converted to interaural transfer functions (ITFs) or modified ITFs to prevent sharp spectral peaks. Alternatively, HRTFs may be described by parametric representations. Such parameterized HRTFs may be easily integrated with parametric multichannel audio coders (e.g., MPEG Surround and Spatial Audio Object Coding (SAOC)).

[0006] To describe the quality of different spatial audio rendering techniques, the concept of Minimum Audible Angle (MAA) can be useful. MAA characterizes the sensitivity of the human auditory system to the angular displacement of a sound event. With regard to localization in azimuth, studies have reported that MAA is smallest (about 1 degree) at the front and back for broadband noise bursts and much larger (about 10 degrees) for lateral sound sources. MAA in the median plane increases with elevation angle. MAA as small as 4 degrees on average in elevation angle has been reported for broadband noise bursts.

[0007] Spatial rendering of audio, which leads to a convincing spatial perception of sounds at arbitrary locations in space, requires a pair of HR filters that represent locations within the MAA of the corresponding locations. If the mismatch in angles for the HR filters is below a limit (i.e., if the angle for the HR filter is within the MAA), the mismatch is not noticeable by the listener. However, if the mismatch is greater than this limit (i.e., if the angle for the HR filter is outside the MAA), such a larger location mismatch may lead to a correspondingly more noticeable inaccuracy in the position perceived by the listener. Summary of the Invention

[0008] Although HR filter measurements are taken at finite measurement locations, audio rendering may require determining HR filters for any possible location on a sphere around the listener (e.g., 150 in FIG. 1). A method of mapping is therefore required to convert from the discrete measurements made at finite measurement locations to a continuous spherical angular domain. Several methods exist for such mapping. The methods include directly using the closest available measurements, using interpolation methods, and / or using modeling techniques.

[0009] 1. Direct use of the nearest neighboring measurement points

[0010] The simplest technique for mapping is to use the HR filter at the closest (i.e., nearest) point among a set of measurement points. Some computational work may be required to determine the nearest neighboring measurement point, and such work may be nontrivial for an irregularly sampled set of measurement points on a sphere around the listener. For general object location, there may be some angular error between the desired filter location (corresponding to the object location) and the nearest available HR filter measurement point. For a sparsely sampled set of HR filter measurements, this may lead to a noticeable error in the object location. The error may be reduced or virtually eliminated when a more densely sampled set of measurement points is used. For moving objects, the HR filter changes in a gradual manner that does not correspond to the intended smooth movement.

[0011] In general, densely sampled measurements of HR filters are difficult to take for human subjects because they require the subject to sit motionless during data collection, and small accidental movements of the subject limit the angular resolution that can be achieved. Also, the measurement process is time-consuming for both the subject and the technician. Instead of taking such densely sampled measurements, inferring spatial relationship information about the missing HR filters may be more efficient given a sparsely sampled HR filter data set (described below). Although densely sampled HR filter measurements are easy to capture for a dummy head, the resulting HR filter set may not always be suitable for all listeners, leading to inaccurate or ambiguous object location perception.

[0012] 2. Interpolation between nearby measurement points

[0013] If the sample measurement points are not spaced closely enough, interpolation between neighboring measurement points can be used to generate an approximate filter for the required DOA. The interpolation filter varies in a continuous manner between the individual sample measurement points, avoiding the abrupt changes that can occur when the above method (i.e., method 1) is used. This interpolation method incurs additional complexity in generating the interpolated HR filter values, and the resulting HR filter has a DOA that is perceived as spread out (as fewer points) by mixing filters from different locations. Also, measures must be taken to prevent phase matching problems that arise from directly mixing filters, which can add complexity.

[0014] 3. Modeling-Based Filter Generation

[0015] More sophisticated techniques can be used to build a model for the underlying system that leads to the HR filter and how it varies with angle. Given a set of HR filter measurements, the model parameters are tuned to reproduce the measurements with minimal error, thereby creating a mechanism for generating the HR filter more generally as a continuous function of angle space, not just at the measurement locations.

[0016] Other methods exist for generating HR filters as a continuous function of DOA that do not require an input set of measurements but instead use high-resolution 3D scans of the listener's head and ears to model wave propagation around the listener's head to predict the behavior of the HR filter.

[0017] A category of HR filter models that utilize weighted basis functions and vectors to represent HR filters is presented below.

[0018] 3.1. HR Filter Model Using Weighted Basis Vectors - Mathematical Framework

[0019] Consider a model for an HR filter having the following form: TIFF0007656688000001.tif18170

[0020] Where: TIFF0007656688000002.tif9170 is the estimated HR filter, a vector of length K for a particular (θ,φ) angle, α n,k is a set of scalar weighting values ​​independent of the angles (θ,φ), F k,n (θ,φ) is a set of scalar-valued functions that depend on the angles (θ,φ), e k teeth, TIFF0007656688000003.tif10170 is a set of orthogonal basis vectors spanning the K-dimensional space of filters.

[0021] Model function F k,n The (θ,φ) are determined as part of the model design and are typically chosen to capture well the variation of the HR filter set across the elevation and azimuth dimensions. For a given model function, the model parameters α n,k can be estimated using data fitting methods such as minimized least squares.

[0022] It is not uncommon to use the same modeling function for all of the HR filter coefficients, which gives rise to a specific subset of this type of model, where the model function F k,n (θ,φ) does not depend on the position k in the filter. F k,n (θ,φ)=F n (θ,φ),∀k (2)

[0023] Therefore, the model can be expressed as follows: TIFF0007656688000004.tif19170

[0024] In one embodiment, e kThe basis vectors are the natural basis vectors e1 = [1,0,0,...0], e2 = [0,1,0,...0], ... aligned with the coordinate system being used. For compactness, when the natural basis vectors are used, the vectors can be rewritten as TIFF0007656688000005.tif19170

[0025] Here, α n is a vector of length K. This leads to the equivalent formula for the model: TIFF0007656688000006.tif18170

[0026] That is, the parameter α n,k When it is estimated, TIFF0007656688000007.tif9170 is a fixed basis vector α n where the angular variation of the HR filter is expressed as a linear combination of the weighting value F n It is captured at (θ,φ).

[0027] Thus, the individual filter coefficients k are obtained as follows: TIFF0007656688000008.tif20170

[0028] This equivalent formula is a compact formula when the unit basis vectors are natural basis vectors. However, the following method can be applied (without this convenient notation) to models that use any choice of basis vectors (including non-orthogonal basis vectors as well as orthogonal basis vectors) in any domain. Other embodiments of the same underlying modeling technique would be different choices of basis vectors in the time domain (e.g., Hermite polynomials, sinusoids, etc.), or in domains other than the time domain, such as the frequency domain (e.g., via Fourier transform), or in any other domain where it is natural to represent HR filters.

[0029] TIFF0007656688000009.tif8170 is the result of the model evaluation specified in equation (5) and should be similar to the measurement of h at the same location. test ,φ test ), h(θ test ,φ test )and TIFF0007656688000010.tif9170 can be compared to assess the quality of the model. If the model is deemed accurate, the model estimates h for some general point, which is not necessarily one of the points where h is measured. This can be used to generate TIFF0007656688000011.tif9170.

[0030] An equivalent matrix formulation of equation (5) is: TIFF0007656688000012.tif11170

[0031] where f(θ,φ) = a row vector of weighting values ​​for one ear, which has length N, i.e., f(θ,φ) = [F1(θ,φ),F2(θ,φ),...,F N (θ,φ)], where α=basis function for one ear, which is organized as rows in a matrix K rows by N columns, i.e. TIFF0007656688000013.tif26170

[0032] As explained in WO2021 / 074294 (herein incorporated by reference), B-spline functions are suitable basis functions for HR filter modeling for the elevation angle θ and the azimuth angle φ. This is because the function F n Show that (θ,φ) can be determined as follows. F N (θ,φ)=Θ p (θ)Φ p,q (φ) (8)

[0033] For p=1,...,,P and q=1,...,Qp, n=(p-1)Q p +q, where P is the number of elevation basis functions, and Q p is the number of azimuth basis functions that can vary for different elevation angles p. For the elevation angles, standard B-spline functions can be used, and for the azimuth angles, periodic B-spline functions can be used.

[0034] As explained above, the three types of methods for inferring HR filters over a continuous range of angles have varying levels of computational complexity and varying levels of perceived location accuracy. The direct use of nearest neighbor measurement points is the simplest, but requires densely sampled measurements of the HR filters, which are not easy to obtain and usually result in large amounts of data. In contrast, methods that use models for HR filters have the advantage that they can generate HR filters with point-like localization properties that vary smoothly as the DOA changes. These methods also express a set of HR filters in a more compact form and therefore may require less resources for transmission and / or storage (including storage in program memory when they are in use). These advantages come at the cost of numerical complexity (the model must be evaluated before the filters can be used to generate HR filters). Such complexity is problematic for rendering systems with limited computational capacity, since such limited capacity limits the number of audio objects that can be rendered, for example, in a real-time audio scene.

[0035] It is desirable for a spatial audio renderer to be able to evaluate HR filters for any elevation-azimuth angle in real time from a model evaluation equation such as equation (5). Therefore, the HR filter evaluation specified in equation (5) needs to be performed very efficiently.

[0036] The repeated evaluation of the HR filter model has the drawback of complexity not only in evaluating the model output, but also in evaluating the basis functions of the model. Moreover, the contribution of some basis functions may be small (e.g., 0) for the evaluation of some HR filter directions. This means that the filter evaluation becomes unnecessarily complex. On the other hand, it is crucial that the memory consumption required for the HR filter evaluation is not significantly increased, especially for application in mobile devices where both memory and computational complexity possibilities are limited.

[0037] From the B-spline basis functions (described, for example, in WO2021 / 074294), the filter evaluation described in equation (5) is n It can be seen that this involves determining (θ,φ), P·Q for each elevation angle p in the evaluation of TIFF0007656688000014.tif9170 p Multiplication and further P·Q for each factor n p These operations are then performed for every filter coefficient k, which in total results in the HR filter The evaluation of TIFF0007656688000015.tif9170 results in a significant number of operations.

[0038] 3(a) and 3(b) show periodic B-spline basis functions.

[0039] Figure 3(a) shows an example of four periodic B-spline basis functions for the [0,360] degree modeling range, with knot points at 0 (=360), 90, 180 and 270 degrees. In this example, all basis functions within each segment between the knot points are non-zero.

[0040] Figure 3(b) shows an example of eight periodic B-spline basis functions for a [0,360] degree modeling range, with knot points at 0 (=360), 45, ..., 315 degrees. In this case, the non-zero part of each basis function covers only half the modeling range, i.e., 180 degrees.

[0041] As shown in Figures 3(a) and 3(b), for some B-spline configurations, only some B-spline functions are non-zero for some directions (θ,φ). For example, the B-spline function starting at 0 degrees in Figure 3(b) can be zero for any angle between 180 and 360 degrees. This means that the HR filter evaluation in equation (5) can involve a significant number of multiplications and additions with zero components. The result is a complexity-inefficient model-based HR filter evaluation.

[0042] According to some embodiments of the present disclosure, the problem of inefficient HR filter evaluation may be solved by a memory-efficient structured representation for complexity-efficient HR filter evaluation and / or the avoidance of multiplications and additions with zero-valued components.

[0043] Thus, in one aspect, a method for generating a head-related (HR) filter for audio rendering is provided. The method includes generating HR filter model data indicative of an HR filter model. Generating the HR filter model data includes selecting at least one set of one or more basis functions. The method also includes (i) sampling the one or more basis functions and (ii) generating first basis function shape data and shape metadata based on the generated HR filter model data. The first basis function shape data identifies one or more compact representations of the one or more basis functions, and the shape metadata includes information regarding a structure of the one or more compact representations for the one or more basis functions. The method further includes providing the first generated basis function shape data and the shape metadata for storage in one or more storage media.

[0044] In some embodiments, the method may further include detecting an occurrence of a triggering event. Such a triggering event may indicate that a head-related (HR) filter should be generated for audio rendering, which may be evoked from the audio renderer, for example, when a head-related (HR) filter is required to render a frame of audio or to prepare the rendering by generation of a head-related (HR) filter that is stored in memory for later use. In some embodiments, the triggering event is merely a decision to retrieve basis function shape data and / or shape metadata from one or more storage media. The method may further include outputting second basis function shape data and shape metadata for audio rendering as a result of detecting the occurrence of the triggering event.

[0045] In another aspect, a method for generating a head-related (HR) filter for audio rendering is provided. The method includes acquiring shape metadata indicating whether to acquire converted versions of one or more compact representations of one or more basis functions. The method further includes acquiring basis function shape data that identifies (i) the one or more compact representations of the one or more basis functions or (ii) the converted versions of the one or more compact representations of the one or more basis functions. The method further includes generating an HR filter by using (i) the one or more compact representations of the one or more basis functions or (ii) the converted versions of the one or more compact representations of the one or more basis functions based on the acquired shape metadata and the acquired basis function shape data.

[0046] In another aspect, an apparatus is provided for generating a head-related (HR) filter for audio rendering. The apparatus is adapted to generate HR filter model data indicative of an HR filter model. Generating the HR filter model data includes selecting at least one set of one or more basis functions. The apparatus is further adapted to (i) sample the one or more basis functions and (ii) generate first basis function shape data and shape metadata based on the generated HR filter model data. The first basis function shape data identifies one or more compact representations of the one or more basis functions, and the shape metadata includes information regarding a structure of the one or more compact representations for the one or more basis functions. The apparatus is further adapted to provide the generated first basis function shape data and shape metadata for storage in one or more storage media.

[0047] The apparatus is further adapted to detect occurrence of a triggering event and output second basis function shape data and shape metadata for audio rendering as a result of detecting the occurrence of the triggering event. Such a triggering event may indicate that a head-related (HR) filter should be generated for audio rendering, which may be evoked from the audio renderer, for example, when a head-related (HR) filter is required to render a frame of audio or to prepare rendering by generation of a head-related (HR) filter that is stored in memory for later use. In some embodiments, the triggering event is merely a decision to retrieve basis function shape data and / or shape metadata from one or more storage media. In one embodiment, the apparatus comprises a processing circuit and a storage unit that stores instructions for configuring the apparatus to perform any of the processes disclosed herein.

[0048] In another aspect, an apparatus is provided for generating a head-related (HR) filter for audio rendering. The apparatus is adapted to obtain shape metadata indicating whether to obtain a converted version of one or more compact representations of one or more basis functions. The apparatus is further adapted to obtain basis function shape data identifying (i) the one or more compact representations of the one or more basis functions or (ii) the converted version of the one or more compact representations of the one or more basis functions. The apparatus is further adapted to generate an HR filter by using (i) the one or more compact representations of the one or more basis functions or (ii) the converted version of the one or more compact representations of the one or more basis functions based on the obtained shape metadata and the obtained basis function shape data.

[0049] In another aspect, a computer program is provided comprising instructions which, when executed by a processing circuit, cause the processing circuit to perform the method described above. In one embodiment, a carrier is provided containing the computer program, the carrier being one of an electronic signal, an optical signal, a radio signal, and a computer readable storage medium.

[0050] Embodiments of the present disclosure enable perceptually transparent (inaudible) optimization for spatial audio renderers that utilize modeling-based HR filters, for example, to render a mono source at position (r, θ, φ) relative to a listener, where r is the radius and (θ, φ) are the elevation and azimuth angles, respectively.

[0051] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate various embodiments. [Brief description of the drawings]

[0052] [Figure 1] FIG. 1 illustrates the propagation of sound waves from a source located at angles θ, φ towards a listener. [Diagram 2] FIG. 1 illustrates sound waves propagating towards a listener interacting with the head and ears and the resulting ITD. [Diagram 3] 3(a)-3(b) are diagrams illustrating exemplary periodic B-spline basis functions. [Figure 4] 4(a)-4(c) are diagrams illustrating exemplary compact representations of the basis functions shown in FIGS. 3(a)-3(b). [Diagram 5] FIG. 2 illustrates an example standard B-spline basis function. [Figure 6] 6(a)-6(d) are diagrams illustrating exemplary compact representations of the basis functions shown in FIG. [Figure 7] FIG. 1 is a diagram of a system, according to some embodiments. [Figure 8] FIG. 2 is a diagram of a process for generating an HR filter, according to some embodiments. [Figure 9] FIG. 1 is a diagram of a system, according to some embodiments. [Figure 10A] FIG. 1 illustrates an apparatus, according to some embodiments. [Figure 10B] FIG. 1 illustrates an apparatus, according to some embodiments. [Figure 11] FIG. 2 is a diagram of a process, according to some embodiments. [Figure 12] FIG. 2 is a diagram of a process, according to some embodiments. [Figure 13] 1 is a diagram of an apparatus, according to some embodiments. [Figure 14] FIG. 3 shows the ITD and HR filters of the acoustic wave shown in FIG. 2. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0053] Some embodiments of the present disclosure are directed to a binaural audio renderer. The renderer may operate standalone or together with an audio codec. Potentially compressed audio signals and their associated metadata (e.g., data specifying the position of the rendered audio source) may be provided to the audio renderer. The renderer may also be provided with head tracking data obtained from a head tracking device (e.g., inside-out inertial-based tracking device(s) such as an accelerometer, gyroscope, compass, or outside-in based tracking device(s) such as a LIDAR). Such head tracking data may affect the metadata (i.e., rendering metadata) used for rendering (e.g., so that the audio object (source) is perceived at a fixed position in space independent of the listener's head rotation). The renderer also obtains the HR filters to be used for binauralization. An embodiment of the present disclosure provides an efficient representation and method for HR filter generation based on weighted basis vectors according to WO2021 / 074294 or equation (1).

[0054] A scalar-valued function F n (θ,φ) is the set of P elevation basis functions Θ p (θ), p=0,...,p-1 and a set of Q azimuthal basis functions Φ q It is assumed that the set of azimuth or elevation basis functions is a function g(·) of p(φ). As explained in WO2021 / 074294, the set of azimuth or elevation basis functions may also vary for different p or q (e.g., the set of azimuth basis functions Φ p,q (θ), which is the azimuthal basis function Q p In one embodiment, F n (θ,φ) is Θ p (θ) and Φ p,q (φ). In other words, F n(θ,φ)=g(Θ p (θ),Φ p,q (φ))=Θ p (θ)Φ p,q (φ) (9) It is.

[0055] Some embodiments of the present disclosure are based on an efficient structure of the HR filter model(s) and perceptually p (θ) and the azimuthal basis function Φ q Based on spatial sampling of (φ).

[0056] 1. HR filter model design

[0057] First, the HR filter model (corresponding to equation (1)) is defined as the HR filter length K, the number of elevation basis functions P, and the number of azimuth basis functions Q. p and a set of basis functions Θ p (θ) and Φ p,q (φ), each basis function may be smooth and may place more weight on some segments (angles) of the elevation and azimuth modeling ranges (e.g., some parts of [-90,...,90] and [0,...,360], respectively). Thus, some basis functions may be 0 for some segments of the modeling ranges.

[0058] In some embodiments, the elevation and azimuth basis functions are designed / selected with certain properties to be used efficiently for HR filter modeling and efficient structured HR filter generation. The basis functions can be defined over a periodic modeling range (e.g., continuous at 0 / 360 degree azimuth boundaries as shown in Figures 3(a) and 3(b), or over a non-periodic range, e.g., [-90,90] degree elevation as shown in Figure 5).

[0059] Thus, according to some embodiments,

[0060] [Property 1] At least one of the basis functions has a first segment that is non-zero-valued and another segment that is zero-valued; and / or

[0061] [Property 2] The at least one non-zero portion of the basis functions is a. is equal to the nonzero part of another basis function, or b. has a nonzero length that is a unit fraction of the length of the nonzero part of another basis function with the same shape, i.e. TIFF0007656688000016.tif10170, where L1 and L2 are the respective lengths and x=1,2,3,..., and / or c. It is symmetrical, or d. It is the mirror (inverse) of the non-zero part of another basis function.

[0062] More basis functions with the same properties may result in a more efficient implementation. However, there may be other factors, such as modeling efficiency and performance, that may also influence the choice of basis functions. For example, depending on the sampling grid of the measured HR filter data, a different number of basis functions should be selected to avoid obtaining an underdetermined system. The basis functions may generally be described analytically (e.g., as splines with polynomials).

[0063] In some embodiments, cubic B-spline functions (i.e., fourth order or degree 3) are used as basis functions Φ for azimuth and elevation angles, respectively. p,q (φ) and Θ p (θ) is used.

[0064] Figures 3(a) and 3(b) show periodic B-spline basis functions for azimuth angles, and Figure 5 shows the corresponding standard B-spline basis functions for elevation angles. The points are marked with different symbols for better discrimination in the figures, but the functions are continuous and can be evaluated at any angle.

[0065] 2. HR filter modeling

[0066] Model design parameters that define the model (e.g., K, P, Q p , Θ p (θ) and Φ p,q (φ)) can be used later for HR filter modeling, where the model parameters α n,k can be estimated using a data fitting method such as minimized least squares (e.g., as described in WO2021 / 074294).

[0067] 3. Basis function sampling

[0068] One aspect of the present disclosure is the basis function Φ p,q (θ) and Θ p The azimuth sampling interval ΔΦ and elevation sampling interval ΔΘ may be selected based on this observation. Studies suggest ΔΦ=1° and ΔΘ=4° for transmission quality (i.e., non-audible losses), but larger sampling intervals may be selected as a compromise between spatial accuracy requirements and memory and (computational) complexity requirements for HR filter estimation.

[0069] If the selected sample spacing values ​​ΔΦ, ΔΘ are larger than the MAA, then interpolation may be used to generate smoothly varying curves and avoid step-like changes that may occur with a very coarsely spaced set of sample points (this approach further reduces memory usage but increases numerical complexity). Basis function sampling may generally be performed in a pre-processing stage, where sampled basis functions to be used for HR filter estimation are generated and stored in memory.

[0070] 3.1. Efficient Representation of Periodic B-Spline Basis Functions

[0071] Figures 3(a) and 3(b) show two examples of periodic B-spline functions for azimuth angles, each showing a set of basis functions covering 360 degrees. As shown in the figures, in both examples, all equal symmetric non-zero parts of the basis functions (coherent properties 2a and 2c described above) are obtained, and this always happens as long as there is a constant spacing between the knot points.

[0072] This means that each of the periodic B-Spline basis functions can be efficiently represented by 1 / 2 of its non-zero shape (due to its symmetric properties). Although the B-Spline basis functions can be computed during run-time, it is more efficient in terms of computational complexity to store pre-computed shapes (i.e., numerical sampling) of the B-Spline basis functions in memory. On the other hand, it is generally desirable to minimize memory requirements (i.e., the amount of memory required to store the pre-computed shapes). The structure of the B-Spline basis function(s) according to embodiments of the present disclosure provides a good compromise between computational complexity and memory requirements.

[0073] Since the number of HR filter measurement points is typically highest at 0° elevation angle and decreases towards ±90°, fewer basis functions can be utilized towards the polar areas of the sampling sphere.

[0074] Using a varying number of azimuth B-spline basis functions for each elevation angle, different knot point intervals I K A compact expression for the set of periodic B-spline functions with (p) can be obtained.

[0075] For knot point spacings of integer decimation factor M TIFF0007656688000017.tif11170, then the non-zero portion of the basis functions will be coherent with property 2b described in section 1 of this disclosure above, and no separate shape needs to be stored, but only the decimation factor M is needed to recover the shape. In this case, the maximum knot point spacing I K Every Mth point of the shape with (p1) has knot-point spacing I K (p2)=I K This corresponds to a sample with a shape of / M, as shown in Figure 4(a) to Figure 4(c).

[0076] FIG. 4(a)-FIG. 4(c) show compact representations of the B-spline basis functions of FIG. 3(a)-FIG. 3(b). Since the non-zero portion of the periodic basis functions is symmetric, only 1 / 2 of the shape is needed to represent the complete shape. Furthermore, the B-spline basis functions of FIG. 3(b) sample points (circle) are obtained by subsampling of FIG. 3(a) sample points (plus). In FIG. 4(a), the plus represents 1 / 2 of the sample points of the basis function in FIG. 3(a). In FIG. 4(b), the plus represents 1 / 2 of the sample points of the basis function in FIG. 3(b). FIG. 4(c) shows the overlapped shape functions of (a) and (b). Shape function (b) can be obtained by subsampling of shape function (a), while the plus represents the range of [0,...,180] degrees and the plus represents the range of [0,...,90] degrees.

[0077] As explained above, in Figures 4(a) to 4(c), the sample points (○) of the shape in Figure 3(b) can be taken as every other sample point (+) for the shape in Figure 3(a).

[0078] 3.2 Efficient Representation of Standard B-Spline Basis Functions

[0079] For periodic B-spline basis functions, a compact representation can be obtained by sampling the standard B-spline basis functions.

[0080] FIG. 5 shows standard elevation B-spline basis functions for P=9. Although some of the basis functions shown in FIG. 5 are not symmetric as in the case of periodic B-spline basis functions (e.g., the basis functions shown in FIG. 3(a) and FIG. 3(b)), it can be seen that the first and last spline functions (from the left side) have mirrored shapes of each other for the non-zero portion (coherent with property 2d described in section 1 of this disclosure above). Similarly, the second and penultimate non-zero spline functions have mirrored shapes of each other, and the third and penultimate non-zero spline functions have mirrored shapes of each other. These properties of having mirror shapes allow for memory-efficient storage of the basis functions. Thus, in some embodiments, a fixed interval for the knot points may be preferred and used. For model evaluation, the stored shapes may be read forward or backward depending on the segment being evaluated. The fourth to penultimate (fourth, fifth and sixth) B-spline basis functions shown in FIG. 5 retain the same properties as the azimuthal B-spline basis functions, i.e., they are symmetric and equal with respect to the non-zero portion.

[0081] 6(a)-6(b) show a compact representation of the standard B-spline basis functions shown in FIG.

[0082] Figure 6(a) shows a compact representation of the first and last basis functions of Figure 5. This corresponds to the mirror shape of the non-zero part of the last basis function.

[0083] Figure 6(b) shows a compact representation of the second and penultimate basis functions of Figure 5. This corresponds to the mirror shape of the non-zero portion of the penultimate basis function.

[0084] Figure 6(c) shows a compact representation of the third and penultimate basis functions of Figure 5. This corresponds to the mirror shape of the non-zero portion of the penultimate basis function.

[0085] Figure 6(d) shows a compact representation of the fourth, fifth and sixth basis functions of Figure 5, which corresponds to half of the symmetric non-zero portion of the basis functions.

[0086] Independent of the total number of B-spline basis functions covering the modeling range (between -90° and 90° in this case), only four independent non-zero B-spline basis function shapes are needed. Furthermore, one of these non-zero B-spline function shapes (e.g., the function shown in FIG. 6(d)) is symmetric with respect to the periodic spline function, and therefore only ½ of the non-zero part needs to be stored.

[0087] 3.3 Storage in memory

[0088] As a result of the basis function sampling, a compact representation of the basis functions (i.e., the basis function shapes) is stored in memory along with shape metadata. The shape metadata may comprise information representing any one or combination of the following: 1. The number of basis functions (the number of azimuth basis functions can be different for different elevation angles); 2. The starting point of each basis function (within the modeling interval), 3. A shape index for each basis function (identifying which of the stored shapes should be used for that basis function); 4. The shape resampling factor M for each basis function, 5. An inversion indicator for each basis function (indicating whether the stored shape should be inverted for that particular basis function); 6. Basis function structures such as B-splines, and 7. The width of the non-zero part of each basis function.

[0089] In some embodiments, if the flip indicator indicates that the stored shape needs to be flipped, the shape stored in the storage medium may be read backwards from the storage medium such that the flipped shape is provided to the renderer.

[0090] Some parameters (e.g., inversion indicator and basis function structure) may not need to be stored and transmitted to the renderer in some embodiments (especially when the model structure is already known to the renderer). For example, if standard cubic B-splines are utilized as in FIG. 5, there is no need to signal that the last three basis functions need to be inverted if it is known that both the basis function sampling and the structured HR filter generation assume that the first four shapes (the first three shapes and 1 / 2 of the fourth shape) were stored in that order. It may further be known that all basis functions between the first and last three basis functions may be composed by the fourth stored shape. In the case of B-splines, the shape metadata may instead include information about the knot points. It may also be known that a periodic B-spline function is used for the azimuth basis functions and a standard B-spline function is used for the elevation. This is an example where shape metadata parameters may be stored in different storage media.

[0091] In addition, the HR filter model parameter α n,k is stored in memory along with the basis function shapes and corresponding shape metadata. In other embodiments, the HR filter model parameters, basis function shapes, and / or shape metadata may be stored in different storage media.

[0092] 4. HR filter generation

[0093] Based on the stored shapes and parameters, a structured HR filter generation can be performed by reading the basis function shapes from memory and applying them correctly for each basis function based on the shape metadata and avoiding unnecessary computational complexity (e.g., unnecessary multiplications and additions), thereby reducing the HR filter model parameters α n,k This results in a very efficient estimation of the HR filter using

[0094] While sampling of B-spline basis functions may reduce the computational complexity (involved in audio rendering) through structured tabulation of the sampled basis functions, the HR filter generation (or model evaluation) may also be optimized to further reduce computational complexity.

[0095] For every direction (θ,φ), assuming the construction of the azimuth and elevation basis functions (i.e., cubic B-spline basis functions) according to FIG. 3 and FIG. 5, there are at most four non-zero B-spline basis functions for every azimuth and elevation angle to be evaluated. Thus, F in equation (8) n For the evaluation of (θ,φ), there will be at most 4·4 = 16 non-zero components. Thus, the filter evaluation in equation (5) can be reduced to the following equation: TIFF0007656688000018.tif22170, where TIFF0007656688000019.tif9170 is F n denotes all non-zero components of (θ,φ).

[0096] Compared to a complete evaluation of N=P·Q (where we use a constant azimuthal basis function, i.e., Q for all p) p =Q), HR filter generation based on equation (9) offers significant savings in complexity, which becomes greater the more basis functions are used to model the HR filter data.

[0097] At most points there are four non-zero basis functions, but at knot points, fewer than four basis functions contribute to the non-zero component.

[0098] The following describes a method for providing optimized model estimation for the generation of HR filters.

[0099] 4.1 Basis Estimation for Periodic B-Spline Basis Functions (Azimuthal Case)

[0100] (1) Knot Segment Index I n Determine (θ,φ). TIFF0007656688000020.tif17170 where φ is the azimuth angle to be evaluated and I m (0) is the azimuth angle at the first knot point, and I K (p) is the knot point spacing for the azimuth B-spline function at elevation index p.

[0101] (2) Determine the closest segment sample point. TIFF0007656688000021.tif18170Here, round() is the rounding function, N s (p) is the number of samples per segment (e.g., TIFF0007656688000022.tif12170), M(p) is the decimation factor for the elevation angle of index p. An example of a suitable rounding function is: TIFF0007656688000023.tif18170, where TIFF0007656688000024.tif8170 shows a floor function that outputs the largest integer less than or equal to its input.

[0102] (3) Number of non-zero basis functions for azimuth angle Determine TIFF0007656688000025.tif9170. TIFF0007656688000026.tif57170

[0103] (4) Calculate the B-spline sample values ​​and shape indices. TIFF0007656688000027.tif66170, where S p is the 1 / 2 sampled shape function at elevation angle p, subsampled by a factor M(p) (described in section 3.1 above). Index of TIFF0007656688000028.tif10170 TIFF0007656688000029.tif10170 is also stored. Q p is the total number of azimuth B-spline basis functions for elevation index p. mod(·) is the modulo function used to determine whether the evaluated azimuth angle φ lies on a knot point.

[0104] 4.2 Basis Estimation for Standard B-Spline Functions (Elevation Case)

[0105] (1) Knot Segment Index I n Determine (θ, p). TIFF0007656688000030.tif18170 where θ is the elevation angle to be evaluated and I m (0) is the elevation angle at the first knot point, and I K is the knot point spacing for the elevation B-spline function.

[0106] (2) Determine the closest segment sample point. TIFF0007656688000031.tif18170Here, round() is the rounding function, N s is the number of samples per segment (e.g., TIFF0007656688000032.tif12170). The rounding function can be the same as that used for the periodic B-spline basis functions.

[0107] (3) Number of non-zero basis functions Determine TIFF0007656688000033.tif10170 TIFF0007656688000034.tif47170

[0108] At the first and last knot points, TIFF0007656688000035.tif10170 can also be used.

[0109] Calculate B-spline sample values ​​and shape indices TIFF0007656688000036.tif131170 Here, I S is the associated sampled shape function at elevation angle p This is an index representing TIFF0007656688000037.tif8170.

[0110] P is the total number of elevation B-spline basis functions. The basis function index (i+I n ) is greater than P-4, the shape is read backwards. Otherwise, if the shape index is greater than the length of the stored shape, which can happen in the case of symmetric shapes, the shape is also read backwards. Stored shape value Index of TIFF0007656688000038.tif10170 TIFF0007656688000039.tif10170 is also stored. len(·) determines the length of the input vector, and min(·,·), max(·,·) determine the minimum and maximum values ​​of the input arguments, respectively.

[0111] 4.3 HR filter evaluation

[0112] Once the azimuth and elevation B-spline basis functions have been evaluated, F n (θ,φ) can be determined by: TIFF0007656688000040.tif41170

[0113] Then, each HR filter coefficient TIFF0007656688000041.tif9170 can be determined as follows: TIFF0007656688000042.tif23170 where HR filter tap index k=0,...,K-1.

[0114] 5. Binaural Rendering

[0115] In some embodiments, the method described above may be used for the zero time delay portion of the HR filters, i.e. excluding the onset time delay of each filter, or the delay difference between the left and right HR filters due to the interaural time difference. The method described above may be utilized in an equivalent manner to evaluate the interaural time difference, which is modeled in a similar manner by B-spline basis functions (e.g., as described in WO2021 / 074294). In such a case, a single ITD is determined, i.e., K=1, as opposed to an HR filter in which the number of filter taps is K≫1. The obtained interaural time difference is then calculated using the generated HR filter ( This can be taken into account either by modifying the image (image: TIFF0007656688000043.tif10170) or by taking the time difference into account by applying an offset during the filtering step.

[0116] Separate weight matrix TIFF0007656688000044.tif8170 but with the same basis functions, i.e. TIFF0007656688000045.tif9170 using HR filters for the left and right sides, respectively TIFF0007656688000046.tif8170 is generated. Therefore, TIFF0007656688000047.tif9170 is evaluated only once for each updated direction (θ,φ).

[0117] A binaural audio signal for the mono source u(n) can then be obtained by filtering the audio source signal with a left HR filter and a right HR filter, respectively (e.g., by using well-known techniques). The filtering can be done using regular convolution techniques in the time domain, or in a more optimized manner, e.g., when the filters are long, using overlap-add techniques in the discrete Fourier transform (DFT) domain. K=96 taps corresponds to a 2 ms filter for a 48 kHz sample rate.

[0118] The embodiments of the present disclosure are based on two main categories of optimization: pre-computed sampled basis functions and structured HR filter evaluation. In some embodiments, the sampled basis functions are calculated and stored in memory in a pre-processing stage. Also, the structured HR filter evaluation can be performed at runtime in the renderer or can be pre-computed and stored as a set of sampled HR filters. Since the memory required to store a sampled HR filter set with high accuracy azimuth and elevation resolution is large, in some embodiments the HR filters are evaluated during runtime.

[0119] 7 illustrates an example system 700 according to some embodiments. The system 700 comprises a pre-processor 702 and an audio renderer 704. The pre-processor 702 and the audio renderer 704 may be included in the same entity or in different entities. Also, the different modules included in the pre-processor 702 (e.g., 710, 712, 714, and / or 716) may be included in the same entity or in different entities, and the different modules included in the audio renderer 704 (718 and / or 720) may be included in the same entity or in different entities.

[0120] In one example, the pre-processor 702 is included in one of an audio encoder, a network entity (e.g., in the cloud), and an audio decoder (i.e., the audio renderer 704). The audio renderer 704 may be included in any electronic device capable of generating an audio signal (e.g., a desktop, laptop computer, tablet, mobile phone, head-mounted display, XR simulation system, etc.).

[0121] The pre-processor 702 includes an HR filter model design module 710, an HR filter modeling module 712, a basis function sampling module 714, and a memory 716. The HR filter model design module 710 is configured to output design data 720 to the HR filter modeling module 712. The HR filter modeling module 712 may receive HR filter data 722 and obtain an HR filter model based on the received design data 720 and the received HR filter data 722. In some embodiments, the HR filter model is designed according to the properties (1) and (2)(a)-(2)(d) described above.

[0122] Obtaining the HR filter model may include selecting a basis function structure, i.e., selecting a set of basis functions for azimuth angles ("azimuth basis functions") and / or a set of basis functions for elevation angles ("elevation basis functions"). The azimuth basis functions may be selected to be periodic over a modeling range (e.g., between 0° and 360°). The modeling range may be a set of N seg The matrix may be divided into equal-sized segments. The basis functions may be selected such that at least one basis function is zero-valued in one or more segments. Also, the basis functions may be at most N b <{P,Q p} basis functions are non-zero in segment i (i.e., at most (less than P) TIFF0007656688000048.tif9170 elevation basis functions are non-zero and / or at most (Q p is less than TIFF0007656688000049.tif9170 azimuth basis functions are non-zero), where P is the total number of elevation basis functions and Q p is the total number of azimuth basis functions for elevation angle p. Furthermore, the basis functions (azimuth basis functions and / or elevation basis functions) may be selected such that the non-zero portions of some basis functions are symmetric, mirror, or subsampled versions of the non-zero portions of other basis functions to take advantage of the optimization techniques described in this disclosure.

[0123] After obtaining the HR filter model, the HR filter modeling module 712 outputs HR filter model data 724 to the basis function sampling module 714. The HR filter model data 724 may indicate the obtained HR filter model (i.e., the selected basis function structure). Based on the received HR filter model data 724, the basis function sampling module 714 may sample the basis functions in intervals ΔΦ (for azimuth basis functions) and ΔΘ (for elevation basis functions) to obtain a compact representation (of the non-zero part) of the azimuth basis function and / or the elevation basis function. A compact representation of the basis function may be obtained because not all parts of the basis function are required to represent the basis function. For example, in the case of a symmetric non-zero part of the basis function, only ½ of the shape of the basis function is required to represent the shape. In the case of a mirrored or inverted non-zero part of the basis function, only one of the mirrored parts is required to represent the shape of the basis function. For a subsampled non-zero portion of the basis functions, only the largest shape is needed to represent the shape of the basis function.

[0124] After obtaining the compact representation of the basis functions, the basis function sampling module 714 may store basis function shape data 728 and shape metadata 730 in the memory 716. The basis function shape data 728 may indicate the shape of the compact representation of the basis functions. The shape metadata 730 may include information about the structure of the compact representation in terms of the HR filter model basis functions. For example, the shape metadata 730 may include information about the shape, orientation (e.g., whether or not it is inverted), and subsampling factor M in terms of the model basis functions. More information about the shape metadata 730 was provided above in section 3.3 of this disclosure.

[0125] In addition to the basis function shape data 728 and the shape metadata 730, the memory 716 may also store additional HR filter model parameters 726 (eg, an α parameter).

[0126] The audio renderer 704 includes a structured HR filter generator 718 and a binaural renderer 720. The structured HR filter generator 718 reads basis function shape data 732, shape metadata 734, and additional HR filter model parameter(s) 736 from the memory 716 and receives rendering metadata 738. The basis function shape data 732 may be the same as or related to the basis function shape data 728. Similarly, the shape metadata 734 and the model parameter(s) 736 may be the same as or related to the shape metadata 730 and the model parameter(s) 726, respectively.

[0127] The structured HR filter generator 718 may generate HR filter information 740 indicative of the HR filter based on (i) basis function shape data 732, (ii) shape metadata 734, (iii) additional HR filter model parameter(s) 736, and (iv) rendering metadata 738. The rendering metadata 738 may specify the direction (θ, φ) to be evaluated.

[0128] 8 shows an example process 800 according to some embodiments. The process 800 may be implemented by a structured HR filter generator 718 included in the audio renderer 704.

[0129] Process 800 may begin at step s802, where the structured HR filter generator 718 identifies a segment in the modeled region based on the received rendering metadata 738. For example, the rendering metadata 738 specifies a particular direction (θ, φ) to be evaluated, and the generator 718 identifies the segment to which the specified direction belongs.

[0130] After performing step s802, in step s804, the structured HR filter generator 718 identifies sample points within the segment identified in step s802.

[0131] After performing step s804, in step s806, generator 718 identifies compact representations of the basis functions (ie, the azimuth basis functions and the elevation basis functions) based on the basis function shape data 732.

[0132] After performing step s806, in step s808, the generator 718 determines based on the shape metadata 734 whether the identified compact representation should be read normally, inverted, or subsampled according to a subsampling factor M, and performs inversion and / or subsampling if necessary.

[0133] After performing step s808, in step s810, the generator 718 generates at most N b Such evaluations are performed for at most N basis functions for the identified segment. bThe method includes obtaining sample values ​​within each of the compact representations of the non-zero basis functions. A detailed description of how the basis functions are evaluated is provided above in Sections 4.1 and 4.2.

[0134] After performing step s810, in step s812, the structured HR filter generator 718 generates an HR filter based on (i) the obtained azimuth basis function values, (ii) the obtained elevation basis function values, and (iii) the additional model parameter(s) 736 (e.g., parameter α). The HR filter may be generated as a sum of multiplied values ​​of the azimuth basis function values ​​and the elevation basis function values ​​weighted by the corresponding model weight parameter (α) for each filter tap k separately. A detailed description of how the HR filter is generated was provided above in section 4.3.

[0135] The HR filters (for the left and right sides) generated by the structured HR filter generator 718 are then provided to the binaural renderer 720.

[0136] Using the HR filters generated by the generator 718, the binaural renderer 720 binauralizes the audio signal 742, i.e. generates two audio output signals (for the left and right side).

[0137] FIG. 9 illustrates an exemplary system 900 for creating sound for an XR scene. The system 900 includes a controller 901, a signal modifier 902 for a first audio stream 951, a signal modifier 903 for a second audio stream 952, a speaker 904 for the first audio stream 951, and a speaker 905 for the second audio stream 952. Although two audio streams, two modifiers, and two speakers are illustrated in FIG. 9, this is merely for illustrative purposes and does not limit the embodiments of the present disclosure in any way. For example, in some embodiments, there may be N audio streams corresponding to N audio objects to be rendered, where the audio stream includes a single mono signal corresponding to a single audio object. Additionally, although FIG. 9 illustrates the system 900 receiving and modifying the first audio stream 951 and the second audio stream 952 separately, the system 900 may receive a single audio stream representing the multiple audio streams. The first audio stream 951 and the second audio stream 952 may be the same or different. If the first audio stream 951 and the second audio stream 952 are the same, the single audio stream may be split into two audio streams that are equivalent to a single audio stream, thereby generating the first audio stream 951 and the second audio stream 952.

[0138] The controller 901 may be configured to receive one or more parameters and trigger the modifiers 902 and 903 to implement modifications to the first audio stream 951 and the second audio stream 952 based on the received parameters (e.g., increase or decrease the volume levels according to a gain function). The received parameters are (1) information 953 about the listener's position (e.g., distance and direction to the audio source) and (2) metadata 954 about the audio source. The information 953 may include the same information as the rendering metadata 738 shown in FIG. 7. Similarly, the metadata 954 may include the same information as the shape metadata 734 shown in FIG. 7.

[0139] In some embodiments of the present disclosure, the information 953 may be provided from one or more sensors included in the XR system 1000 shown in FIG. 10A. As shown in FIG. 10A, the XR system 1000 is configured to be worn by a user. As shown in FIG. 10B, the XR system 1000 may include an orientation sensing unit 1001, a position sensing unit 1002, and a processing unit 1003 coupled to the controller 1001 of the system 1000. The orientation sensing unit 1001 is configured to detect a change in the orientation of the listener and provide information about the detected change to the processing unit 1003. In some embodiments, the processing unit 1003 determines an absolute orientation (with respect to some coordinate system) given the detected change in orientation detected by the orientation sensing unit 1001. There may also be different systems for determining orientation and position, for example, the HTC Vive system using a lighthouse tracker (lidar). In one embodiment, the orientation sensing unit 1001 may determine an absolute orientation (with respect to some coordinate system) given the detected change in orientation. In this case, the processing unit 1003 may simply multiplex absolute orientation data from the orientation sensing unit 1001 and absolute position data from the position sensing unit 1002. In some embodiments, the orientation sensing unit 1001 may include one or more accelerometers and / or one or more gyroscopes. The types of XR system 1000 and / or components of the XR system 1000 shown in Figures 10A and 10B are provided for illustrative purposes only and do not limit the embodiments of the present disclosure in any way. For example, although the XR system 1000 is shown including a head-mounted display over the user's eyes, the system may not be equipped with such a display, for example, in the case of an audio-only implementation.

[0140] 11 is a flow chart illustrating a process 1100 for generating an HR filter for audio rendering. The process 1100 may begin at step s1102.

[0141] Step s1102 includes generating HR filter model data indicative of the HR filter model. Generating the HR filter model data may include selecting at least one set of one or more basis functions.

[0142] Step s1104 includes sampling the one or more basis functions based on the generated HR filter model data (s1104).

[0143] Step s1106 includes generating first basis function shape data and shape metadata based on the generated HR filter model data, where the first basis function shape data identifies one or more compact representations of the one or more basis functions and the shape metadata includes information regarding a structure of the one or more compact representations for the one or more basis functions.

[0144] Step s1108 includes providing the generated first basis function shape data and the shape metadata for storage in one or more storage media.

[0145] Step s1110 includes detecting the occurrence of a triggering event.

[0146] Step s1112 includes outputting the second basis function shape data and the shape metadata for audio rendering as a result of detecting the occurrence of the triggering event.

[0147] Such a triggering event may indicate that a head-related (HR) filter should be generated for audio rendering, which may be evoked from the audio renderer, for example, when a head-related (HR) filter is required to render a frame of audio or to prepare a rendering by generation of an HR filter that is stored in memory for later use. In some embodiments, the triggering event is merely a decision to retrieve basis function shape data and / or shape metadata from one or more storage media.

[0148] In some embodiments, the at least one set of one or more basis functions satisfies the following condition: (i) the at least one set of one or more basis functions is periodic over the modeled range; (ii) at least one basis function included in said at least one set is zero-valued in one or more segments included in the modeled range; (iii) at most N basis functions included in the at least one set are non-zero in a segment included in the modeled range, N being a positive integer less than the total number of basis functions included in the at least one set; and (iv) at least one non-zero portion of the one or more basis functions is either one or a combination of: (1) symmetric or mirror with respect to another non-zero portion of the one or more basis functions; or (2) a subsampled version of another non-zero portion of the one or more basis functions. is selected so that any one or combination of the following is satisfied:

[0149] In some embodiments, the compact representation of the one or more basis functions indicates a shape of a non-zero portion of the one or more basis functions, where the shape of the non-zero portion of the one or more basis functions is symmetric or mirror with respect to the shape of another non-zero portion of the one or more basis functions.

[0150] In some embodiments, the shape metadata includes the following information: (i) The number of basis functions, (ii) the starting point for each basis function; and (iii) one or more shape indexes, each of which identifies a particular shape to be used for audio rendering; and (iv) a shape resampling factor for one or more basis functions; and (v) an inversion indicator for one or more basis functions, the inversion indicator indicating whether to obtain an inverted version of the one or more compact representations of the one or more basis functions stored in the one or more storage media; and (vi) a basis set structure; and (vii) The width of the non-zero part of each basis function and The present invention has any one or combination of the above.

[0151] In some embodiments, the method further comprises providing additional HR filter model parameters for storage in said one or more storage media.

[0152] In some embodiments, the method is performed by a pre-processor prior to the occurrence of an event that triggers audio rendering.

[0153] In some embodiments, the method is performed by a pre-processor that is separate and contained in a separate network entity from the audio renderer.

[0154] In some embodiments, the second basis function shape data and the shape metadata are used to generate an HR filter.

[0155] In some embodiments, the first basis function shape data and the second basis function shape data are the same.

[0156] In some embodiments, the second basis function shape data identifies a converted version of the one or more compact representations of the one or more basis functions, wherein the converted version of the one or more compact representations of the one or more basis functions is a symmetric or mirrored version and / or a subsampled version of the one or more compact representations of the one or more basis functions.

[0157] 12 is a flow chart illustrating a process 1200 for generating an HR filter for audio rendering. The process 1200 may begin at step s1202.

[0158] Step s1202 includes obtaining shape metadata indicating whether to obtain a converted version of one or more compact representations of one or more basis functions.

[0159] Step s1204 includes obtaining basis function shape data that identifies (i) the one or more compact representations of the one or more basis functions or (ii) a converted version of the one or more compact representations of the one or more basis functions.

[0160] Step s1206 includes generating an HR filter based on the obtained shape metadata and the obtained basis function shape data by using (i) the one or more compact representations of the one or more basis functions or (ii) a converted version of the one or more compact representations of the one or more basis functions.

[0161] In some embodiments, the method further includes obtaining data corresponding to the one or more compact representations of the one or more basis functions from a storage medium after obtaining the shape metadata indicating how to obtain a converted version of the one or more compact representations of the one or more basis functions, the data being obtained in a predefined manner such that a converted version of the one or more compact representations of the one or more basis functions is obtained.

[0162] In some embodiments, the method includes receiving data identifying the one or more compact representations of the one or more basis functions and providing the received data for storage in another storage medium. Obtaining basis function shape data identifying converted versions of the one or more compact representations of the one or more basis functions includes reading the stored received data in a predefined manner from the other storage medium.

[0163] In some embodiments, the converted version of the one or more compact representations of the one or more basis functions is a symmetric or mirrored version and / or a subsampled version of the one or more compact representations of the one or more basis functions.

[0164] In some embodiments, acquiring data in a predefined manner includes (i) acquiring data in a predefined sequence and / or (ii) acquiring data in parts.

[0165] In some embodiments, the converted version of the compact representation of the one or more basis functions is a symmetric or mirrored version and / or a subsampled version of the compact representation of the one or more basis functions.

[0166] In some embodiments, the method further includes obtaining rendering metadata indicating a particular direction or location to be evaluated, and identifying sample points related to the particular direction or location to be evaluated based on the obtained rendering metadata.

[0167] In some embodiments, the one or more compact representations of the one or more basis functions indicate a shape of a non-zero portion of the one or more basis functions, where the shape of the non-zero portion of the one or more basis functions is symmetric or mirror with respect to a shape of another non-zero portion of the one or more basis functions.

[0168] In some embodiments, the shape metadata comprises any one or combination of the following information: (i) a number of basis functions; (ii) a starting point for each basis function; (iii) one or more shape indices, each identifying a particular shape to be used for HR filter generation; (iv) a shape resampling factor for one or more basis functions; (v) an inversion indicator for one or more basis functions, the inversion indicator indicating whether to obtain an inverted version of the one or more compact representations of the one or more basis functions stored in a storage medium; (vi) a basis function structure; and (vii) a width of a non-zero portion of each basis function.

[0169] In some embodiments, the method further includes obtaining an audio signal and filtering the obtained audio signal to generate a left audio signal for the left side and a right audio signal for the right side using the generated HR filter, the left audio signal and the right audio signal being associated with a particular direction and / or location indicated by the rendering metadata.

[0170] Fig. 13 is a block diagram of an apparatus 1300 according to some embodiments for implementing the pre-processor 702 or the audio renderer 704 shown in Fig. 7. As shown in Fig. 13, the apparatus 1300 includes a processing circuit (PC) 1302 that may include one or more processors (P) 1355 (e.g., a general-purpose microprocessor and / or one or more other processors, such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), etc.), which may be co-sited in a single housing or in a single data center or may be geographically distributed (i.e., the apparatus 1300 may be a distributed computing device), and at least one network interface 1348, each of which may be a network interface that the apparatus 1300 may use to communicate with the network interface. The PC 1302 may comprise at least one network interface 1348, a transmitter (Tx) 1345 and a receiver (Rx) 1347 for enabling the network interface 1348 to transmit data to and receive data from other nodes connected to the network 110 (e.g., an Internet Protocol (IP) network) to which the network interface 1348 is connected (directly or indirectly) (e.g., the network interface 1348 may be wirelessly connected to the network 110, in which case the network interface 1348 is connected in an antenna configuration), and one or more storage units (a.k.a. "data storage system") 1308, which may include one or more non-volatile storage devices and / or one or more volatile storage devices. In an embodiment in which the PC 1302 includes a programmable processor, a computer program product (CPP) 1341 may be provided. The CPP 1341 includes a computer readable medium (CRM) 1342, which stores a computer program (CP) 1343 comprising computer readable instructions (CRI) 1344. The CRM 1342 may be a non-transitory computer-readable medium, such as a magnetic medium (eg, a hard disk), an optical medium, a memory device (eg, a random access memory, a flash memory), or the like.In some embodiments, the CRI 1344 of the computer program 1343, when executed by the PC 1302, is configured such that the CRI causes the device 1300 to perform the steps described herein (e.g., steps described herein with reference to flow charts). In other embodiments, the device 1300 may be configured to perform the steps described herein without the need for code; that is, for example, the PC 1302 may simply consist of one or more ASICs. Thus, features of the embodiments described herein may be implemented in hardware and / or software.

[0171] Although various embodiments have been described herein, it should be understood that the embodiments have been presented by way of example only, and not by way of limitation. Thus, the breadth and scope of the present disclosure should not be limited by any of the exemplary embodiments described above. Moreover, unless otherwise indicated herein or clearly contradicted by context, any combination of the above-described elements in all possible variations thereof is encompassed by the present disclosure.

[0172] Additionally, while the process and message flows described above and illustrated in the Figures have been depicted as a sequence of steps, this was done for purposes of illustration only, and it is contemplated that some steps may be added, some steps may be omitted, the order of the steps may be rearranged, and some steps may be performed in parallel.

[0173] 6. Abbreviations TIFF0007656688000050.tif132170

Claims

1. A method (1200) for generating a head-related (HR) filter for audio rendering, the method comprising: Obtaining (s1202) shape metadata indicating whether to obtain a converted version of one or more compact representations of one or more basis functions, the one or more basis functions being functions for HR filter modeling for an elevation angle θ or an azimuth angle φ, the one or more compact representations being representations required to represent the shape of the basis functions; Obtaining basis function shape data (s1204), which identifies (i) the one or more compact representations of the one or more basis functions or (ii) the converted version of the one or more compact representations of the one or more basis functions; generating the HR filter (s1206) based on the acquired shape metadata and the acquired basis function shape data by using (i) the one or more compact representations of the one or more basis functions or (ii) the converted version of the one or more compact representations of the one or more basis functions; The method (1200).

2. The method further comprising: obtaining data corresponding to the one or more compact representations of the one or more basis functions from a storage medium after obtaining the shape metadata indicating how to obtain the converted versions of the one or more compact representations of the one or more basis functions. Further comprising: the data corresponding to the one or more compact representations of the one or more basis functions are obtained in a predefined manner such that the converted versions of the one or more compact representations of the one or more basis functions are obtained. The method of claim 1.

3. The method further comprising: receiving data identifying the one or more compact representations of the one or more basis functions; providing the received data for storage on a storage medium; and Including, obtaining basis function shape data identifying the converted version of the one or more compact representations of the one or more basis functions comprises reading the stored data in a predefined manner from the storage medium. The method of claim 1.

4. the converted version of the one or more compact representations of the one or more basis functions is a symmetric or mirrored version and / or a subsampled version of the one or more compact representations of the one or more basis functions.

4. The method according to any one of claims 1 to 3.

5. 3. The method of claim 2, wherein acquiring the data in the predefined manner comprises: (i) acquiring the data in a predefined sequence; and / or (ii) acquiring the data in parts.

6. The method further comprising: Obtaining rendering metadata indicating a particular direction or location to be evaluated; identifying sample points related to the particular direction or location to be evaluated based on the obtained rendering metadata; The method of claim 1 , further comprising:

7. the one or more compact representations of the one or more basis functions indicate a shape of a non-zero portion of the one or more basis functions; the shape of the non-zero portion of the one or more basis functions is symmetric or mirror with respect to the shape of another non-zero portion of the one or more basis functions; 7. The method according to any one of claims 1 to 6.

8. The shape metadata includes the following information: (i) the number of basis functions; and (ii) the starting point of each basis function; and (iii) one or more shape indices, each of which identifies a particular shape to be used for HR filter generation; and (iv) a shape resampling factor for one or more basis functions; and (v) an inversion indicator for one or more basis functions, the inversion indicator indicating whether to obtain an inverted version of the one or more compact representations of the one or more basis functions; and (vi) a basis function structure; and (vii) the width of the non-zero part of each basis function; 8. The method of claim 1 , comprising any one or combination of:

9. The method further comprising: Obtaining an audio signal; filtering the captured audio signal using the generated HR filter to generate a left audio signal for a left side and a right audio signal for a right side; Further comprising: the left audio signal and the right audio signal are associated with the particular direction and / or location indicated by the rendering metadata; The method according to claim 6.

10. An apparatus (1300) for generating a head-related (HR) filter for audio rendering, the apparatus comprising: Obtaining (s1202) shape metadata indicating whether to obtain a converted version of one or more compact representations of one or more basis functions, the one or more basis functions being functions for HR filter modeling for an elevation angle θ or an azimuth angle φ, the one or more compact representations being representations required to represent the shape of the basis functions; Obtaining basis function shape data (s1204), which identifies (i) the one or more compact representations of the one or more basis functions or (ii) the converted version of the one or more compact representations of the one or more basis functions; generating the HR filter (s1206) based on the acquired shape metadata and the acquired basis function shape data by using (i) the one or more compact representations of the one or more basis functions or (ii) the converted version of the one or more compact representations of the one or more basis functions; The apparatus (1300) is configured to perform the above.

11. The apparatus, obtaining data corresponding to the one or more compact representations of the one or more basis functions from a storage medium after obtaining the shape metadata indicating how to obtain the converted versions of the one or more compact representations of the one or more basis functions. It is further configured to the data corresponding to the one or more compact representations of the one or more basis functions are obtained in a predefined manner such that the converted versions of the one or more compact representations of the one or more basis functions are obtained.

11. The apparatus of claim 10.

12. The apparatus, receiving data identifying the one or more compact representations of the one or more basis functions; providing the received data for storage on a storage medium; and It is further configured to obtaining basis function shape data identifying the converted version of the one or more compact representations of the one or more basis functions comprises reading the stored data in a predefined manner from the storage medium.

11. The apparatus of claim 10.

13. the converted version of the one or more compact representations of the one or more basis functions is a symmetric or mirrored version and / or a subsampled version of the one or more compact representations of the one or more basis functions.

13. Apparatus according to any one of claims 10 to 12.

14. The apparatus of claim 11 , wherein acquiring the data in the predefined manner comprises (i) acquiring the data in a predefined sequence and / or (ii) acquiring the data in parts.

15. The apparatus, Obtaining rendering metadata indicating a particular direction or location to be evaluated; identifying sample points related to the particular direction or location to be evaluated based on the obtained rendering metadata; 15. The apparatus of claim 10, further configured to:

16. the one or more compact representations of the one or more basis functions indicate a shape of a non-zero portion of the one or more basis functions; the shape of the non-zero portion of the one or more basis functions is symmetric or mirror with respect to the shape of another non-zero portion of the one or more basis functions; 16. Apparatus according to any one of claims 10 to 15.

17. The shape metadata includes the following information: (i) the number of basis functions; and (ii) the starting point of each basis function; and (iii) one or more shape indices, each of which identifies a particular shape to be used for HR filter generation; and (iv) a shape resampling factor for one or more basis functions; and (v) an inversion indicator for one or more basis functions, the inversion indicator indicating whether to obtain an inverted version of the one or more compact representations of the one or more basis functions; and (vi) a basis function structure; and (vii) the width of the non-zero part of each basis function; 17. The apparatus of claim 10, further comprising any one or combination of:

18. The apparatus, Obtaining an audio signal; filtering the captured audio signal using the generated HR filter to generate a left audio signal for a left side and a right audio signal for a right side; wherein the left audio signal and the right audio signal are associated with the particular direction and / or location indicated by the rendering metadata.

16. The apparatus of claim 15.

Images