Generating a head-related filter dataset corresponding to a full spatial range

Abstract

There is provided a method of generating a first set of head-related (HR) filters for producing spatial audio signals. The first set of HR filters corresponds to a first spatial range. The method comprises obtaining a second set of HR filters corresponding to a second spatial range, which is a portion of the first spatial range. The method further comprises generating a third set of HR filters by adding to the second set of HR filters a subset of HR filters corresponding to a third spatial range which does not overlap the second spatial range. The method further comprises, using a machine learning, ML, model, generating the first set of HR filters corresponding to the first spatial range based on the third set of HR filters. The first spatial range includes the second and third spatial ranges.

Description

GENERATING A HEAD-RELATED FILTER DATASET CORRESPONDING TO A FULL SPATIAL RANGE TECHNICAL FIELD

[0001] This disclosure relates to generating a head-related (HR) filter dataset corresponding to a full spatial range. BACKGROUND

[0002] The human auditory system is equipped with two ears that capture the sound waves propagating towards the listener. A sound wave propagating towards a listener from a direction of arrival (DoA) may be specified by a pair of an elevation angle and an azimuth angle in the spherical coordinate system. On the propagation path towards the listener, the sound wave interacts with the listener’s upper torso, head, outer portions of ears, and the surrounding matter before reaching the listener’s left and right ear drums. This interaction results in temporal and spectral changes of the sound waveforms reaching the left and right eardrums, some of which are DoA dependent. Our auditory system has learned to interpret these changes to infer various spatial characteristics of the sound wave as well as the acoustic environment in which the listener finds himself / herself. This capability is called spatial hearing.

[0003] The spatial hearing concerns inferring the location of an auditory event, e.g., sound emission by a physical audio source, and acoustic characteristics caused by a physical environment, e.g., small room, tiled bathroom, auditorium, cave, etc., based on spatial cues embedded in a binaural signal, i.e., the sound signals in the right and the left ear canals. This human capability can in turn be exploited to create a spatial audio scene by reintroducing the spatial cues in the binaural signal that would lead to a spatial perception of a sound.

[0004] The main spatial cues include 1) angular-related cues: binaural cues (i.e., the interaural level difference (ILD) and the interaural time difference (ITD), and monaural (or spectral) cues); 2) distance-related cues: intensity and direct-to-reverberant (D / R) energy ratio. A mathematical representation of the short time DoA dependent temporal and spectral changes (1-5 msec) of the waveform is called HR filters. The frequency domain (FD) representations of the HR filters are called head-related transfer functions (HRTFs), and the time domain (TD)representations are the head-related impulse responses (HRIRs).

[0005] FIGS.1A-1B and 1D-1E illustrate an example of ITD and spectral cues of a sound wave propagating towards a listener, and FIG.1C shows a direction of a sound wave propagating towards the listener. The four plots shown in FIGS.1A-1B and 1D-1E illustrate the time domain and the frequency domain responses of a pair of left and right HR filters obtained at an elevation of 0 degree and an azimuth of 40 degrees (Note that this data is from Center for Image Processing and Integrated Computing (CIPIC) database: subject-ID 28, which is provided in Reference [3]. The reference [3] is listed at the end of this disclosure. The database is publicly available and can be access from the link http: / / sofacoustics.org / data / database / cipic / ).

[0006] This human capability can in turn be exploited to create a spatial audio scene by reintroducing the spatial cues in the binaural signal that would lead to a spatial perception of a sound. An HR filter based binaural rendering approach has been gradually established. In the approach, an audio source signal is binauralized by directly filtering the audio source signal with a pair of HR filters of desired locations. This approach is particularly attractive for many emerging applications such as virtual reality (VR), augmented reality (AR), mixed reality (MR), or extended reality (XR), and mobile communication systems, where headsets are commonly used.

[0007] HR filters are often estimated from acoustic measurements as the impulse response of a linear dynamic system that transforms the original sound signal, i.e., the input signal, into the left and right ear signals, i.e., the output signals. These left and right ear signals can be measured inside the ear canals of a listening subject at a predefined set of elevation and azimuth angles on a spherical surface that is located at a constant radius from a listening subject, e.g., an artificial head, a manikin or human subjects.

[0008] The estimated, either by measurement or by numerical simulation, HR filters are often provided as Finite Impulse Response (FIR) filters. These FIR filters are usually several milliseconds long, where the temporal span of each filter can be divided into the three sequential temporal regions -- the pre-active region, the active region, and the post-active region. In the pre-active region and the post-active region, the filter taps are zero or, due to estimation noise, very close to zero and contribute very little to the binauralization. The active region contains the main parts of the impulse response of the filter that stand for actual binauralization. In thebeginning of the active region, there are typically strong oscillations around zero, and these oscillations taper off and decrease to close-to-zero values in the end of the region. In the plots of the time domain responses in FIGS.1A and 1B, the active region is roughly indicated by a dashed ellipse.

[0009] HR filter sets may be used directly in their original form by a binaural audio renderer where a spatial audio is rendered by filtering an audio source signal with a pair of HR filters of a desired location. Alternatively, HR filter sets may be used for predictive modeling where the resulting model is used to provide HR filters for a spatial audio renderer.

[0010] Rendering a spatial audio source that leads to a convincing spatial perception of a sound at an arbitrary location in space requires a pair of HR filters at the corresponding location, and thus, a set of HR filters at finely sampled locations on a full sphere is needed, where the elevation angles cover the full range from bottom to top, i.e., ^^ ∈ ^െగ గଶ ,ଶ^ and the azimuth angles ^ ^^, ^ ^ ^^^ at 0, 0 corresponds to a point directly ahead, ^0,గଶ^ corresponds to a point directly to the left,^0, ^^^corresponds to a point directly behind,గଶ^ corresponds to a point directly to the right, ^ଶ, 0^ଶto a point directly below.

[0011] One way of obtaining HR filters is using an acoustic measurement. The acoustic measurement generally requires an acoustically well-treated environment, e.g., an anechoic chamber, a set of dedicated audio playback and acquisition equipment, e.g., loudspeakers, in-ear microphones, an amplifier, a soundcard, etc., a mechanical system that determines the source / listener positions, and a dedicated software package for playing, acquiring, and processing audio data. FIG.2 shows a simplified setup for the acoustic measurement.

[0012] A basic mechanical setup needed for performing the acoustic measurement comprises, for example, a vertical arc mounted with fixed / movable loudspeaker(s) determining elevation angles and a turntable with a chair affixed upon determining azimuth angles. The mechanical setup should always be carefully designed to have a minimal effect on incident acoustic waves. For example, the sides of loudspeakers may be wrapped in acoustic absorbers, the supporting structure of the chair may be covered by acoustic absorbers, and so on. For humansubjects, a special chair is usually designed to have a structure with head rest and back rest that provides a reference position of the subject’s head relative to the loudspeaker(s) in order to minimize undesired head movements during measurements. SUMMARY

[0013] Certain challenges presently exist. For example, as discussed above, rendering a spatial audio source at an arbitrary location in space requires HR filters at finely sampled locations on the entire sphere. However, obtaining an HR filter set having HR filters with such high spatial density via the acoustic measurement is difficult because it requires the human subject to stay still for many hours. Moreover, it is often that HR filters at low elevation angles, e.g., ^^ ^ െగଷ, could not be measured due to certain mechanical restrictions, e.g., the installation of chair and turntable block the propagation path, the anechoic room is not high enough, and so on.

[0014] Instead of relying on the acoustic measurement, HR filters with a high spatial density may be obtained via a numerical simulation using Boundary Element Method (BEM) based on an optical capture of ear, head, and / or upper torso of a subject. This method obtains an HR filter set by simulating the sound-field scattered by human’s outer ears, head, and torso. The advantages of this method are that it is insensible to measurement noise, and it allows experiments that are out of reach in real life. However, the simulation approach requires a fine- grained mesh of outer ears of a human subject in order to calculate HR filters for the full audible frequency range. However, in order to obtain such fine grained meshes, advanced 3D capture devices and procedures are generally required, and safety measures need to be taken to protect subjects from radiation. Thus, like the acoustic measurement approach, the numerical simulation approach may not be practical. Therefore, there is a need for a different way to obtain HR filters with a high spatial density.

[0015] Accordingly, in one aspect of some embodiments of this disclosure, there is provided a method of generating a first set of head-related, HR, filters for producing spatial audio signals, wherein the first set of HR filters corresponds to a first spatial range. The method comprises obtaining a second set of HR filters corresponding to a second spatial range, wherein the second spatial range is a portion of the first spatial range. The method further comprises generating a third set of HR filters by adding to the second set of HR filters a subset of HR filterscorresponding to a third spatial range, wherein the third spatial range does not overlap the second spatial range; and using a machine learning, ML, model, generating the first set of HR filters corresponding to the first spatial range based on the third set of HR filters, wherein the first spatial range includes the second and third spatial ranges.

[0016] In another aspect, there is provided a computer program comprising instructions which when executed by processing circuitry cause the processing circuitry to perform the embodiments above.

[0017] In another aspect, there is provided a carrier containing the computer program of the above embodiment, wherein the carrier is one of an electronic signal, an optical signal, a radio signal, and a computer readable storage medium.

[0018] In another aspect, there is provided an apparatus for generating a first set of head- related, HR, filters for producing spatial audio signals, wherein the first set of HR filters corresponds to a first spatial range. The apparatus is configured to obtain a second set of HR filters corresponding to a second spatial range, wherein the second spatial range is a portion of the first spatial range. The apparatus is further configured to generate a third set of HR filters by adding to the second set of HR filters a subset of HR filters corresponding to a third spatial range, wherein the third spatial range does not overlap the second spatial range; and using a machine learning, ML, model, generate the first set of HR filters corresponding to the first spatial range based on the third set of HR filters, wherein the first spatial range includes the second and third spatial ranges.

[0019] In another aspect, there is provided an apparatus comprising a processing circuitry; and a memory, said memory containing instructions executable by said processing circuitry, whereby the apparatus is operative to perform the method of at least one of the embodiments described above.

[0020] Some embodiments of this disclosure provide any one or more of the following advantages.

[0021] They provide a smooth extrapolation solution for an incomplete HR filter dataset with sample points on a spherical dome.

[0022] They enable the use of existing source datasets, which improves the quality of the solution.

[0023] The interpolated and / or extrapolated HR filters obtained via the embodiments deliver accurate perceptual localization cues and provide timbral characteristics that are consistent with those HR filters provided in an incomplete HR filter dataset.

[0024] They generate HR filters at any arbitrary location in space at an efficiency that is enough for a real-time VR / AR / MR / XR system. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate various embodiments.

[0026] FIGS.1A, 1B, 1D, and 1E show time domain and frequency domain responses of a pair of HR filters.

[0027] FIG.1C shows a sound wave propagating towards a listener.

[0028] FIG.2 shows a simplified setup for an acoustic measurement.

[0029] FIG.3 shows a system according to some embodiments.

[0030] FIGS.4A, 4B, 5A, and 5B illustrate the concept of an HR filter.

[0031] FIG.6 shows a direction of arrival of an audio wave as observed from the listener, within a three-dimensional (3D) space.

[0032] FIG.7 shows sample locations on a sphere where a set of HR filters are measured.

[0033] FIGS.8B-8D shows examples of distribution of sample locations.

[0034] FIG.9A shows a process for performing spatial extrapolation.

[0035] FIG.9B shows a process of performing a pre-processing operation.

[0036] FIGS.10A-10D illustrates different ways of performing a slicing.

[0037] FIG. 11 shows a block diagram of a pre-processing operation for training a model used for performing a spatial extrapolation.

[0038] FIG.12 illustrates a simple example of a masking operation for training a model used for performing a spatial extrapolation.

[0039] FIG.13 shows a process according to some embodiments.

[0040] FIG.14 shows an apparatus according to some embodiments. DETAILED DESCRIPTION

[0041] FIG. 3 shows an exemplary system 300 according to some embodiments. System 300 comprises a headphone 306, an audio rendering unit 312, and a server 314. Server 314 is configured to transmit audio data 316 to audio rendering unit 312 via network 310. Network 310 may be a wired network or a wireless network. Network 310 may be a cloud via which audio data 316 is transmitted from server 314 to audio rendering unit 312. In this disclosure, audio data is defined as data used for, after rendering, e.g., processing with HR filter(s), providing a listener with an audio experience as if the listener is in a three-dimensional (3D) space where the audio source(s) are located. The audio data includes audio samples of source signals corresponding to audio source(s). In some embodiments, the audio data may additionally include HR filter information, e.g., an HR filter dataset, indicating HR filters.

[0042] After receiving audio data 316, audio rendering unit 312 may generate binaural audio signals and transmit the generated audio signals to headphone 306. Headphone 306 is configured to generate audio based on the audio signals, thereby providing user 302 with spatial audio experience. In some embodiments, instead of headphone 306, other audio generating devices such as an array of speakers may be used. The number of the speakers in the array can be any number larger than two.

[0043] In some embodiments, system 300 may additionally comprise an XR display headset 304, such as VR, MR, or AR displays. XR display headset 304 may be configured to generate different views of an XR environment based on the head orientation of user 302.

[0044] XR display headset 304 may be communicatively coupled to headphone 306. For example, XR display headset 304 may detect the head orientation of user 302, and based on the detected head orientation of user 302, display a different view of the XR environment. XR display headset 304 may also trigger audio rendering unit 312 to generate different audio signals corresponding to different views such that user 302 can hear different audio based on the head orientation of user 302.

[0045] FIGS.4A, 4B, 5A, and 5B illustrate the basic concept of HR filtering.

[0046] FIG. 4A shows an audio wave 402, a.k.a., sound wave, propagating in a firstdirection and reaching the right ear of user 302, and FIG. 4B shows an audio wave 412 propagating in a second direction that is different from the first direction, and reaching the right ear of user 302. As shown in FIGS.4A and 4B, depending on the DoA of the audio waves with respect to the center of user 302’s head, the audio waves are diffracted and / or reflected in different ways, e.g., see the paths formed by the dotted arrows in FIGS.4A and 4B. For simple explanation, only the reflections are shown in FIGS.4A and 4B.

[0047] As explained above, HR filters are used for generating audio effects in which these different diffractions and reflections caused by different DoAs are factored. In other words, depending on the DoA of an audio wave, the audio wave goes through different temporal and spectral changes before being sensed by user 302, and a mathematical representation of such temporal and spectral changes is called an HR filter. Note that the reflection paths shown in FIGS. 4A and 4B are provided for illustration purpose only and may be different from actual reflection paths in a real world environment.

[0048] FIG.5A shows an exemplary time domain response of a right HR filter for audio wave 402 and FIG.5B shows an exemplary time domain response of a right HR filter for audio wave 412. As shown in the figures, because of the different temporal and spectral changes the audio waves go through, the waveforms including the amplitude and the time of arrival (TOA) (or onset delay) are different for audio waves 402 and 412. Note that the responses shown in FIGS.5A and 5B are provided just to show few aspects of the impact of the HR filters, and thus may be different from the real responses.

[0049] As discussed above, the temporal and spectral changes of an audio wave may vary depending on the DoA of the audio wave. In FIG.6, a DoA vector 602 indicates the propagation direction of an audio wave within a 3D space defined by three axes 612, 614, and 616, where axis 612 is the front axis of the listener. DoA vector 602 may be defined using two angles -- azimuth angle ( ^^) and elevation angle ( ^^). The azimuth angle ( ^^) is an angle between an axis 612 (e.g., an x-axis) and a projection vector 604 corresponding to a projection of DoA vector 602 onto a plane formed by axis 612 and axis 614. The elevation angle ( ^^) is an angle between DoA vector 602 and projection vector 604.

[0050] Since the temporal and spectral changes of an audio wave may vary depending on the DoA which may be defined by the azimuth angle ( ^^) and the elevation angle ( ^^), in someembodiments, different HR filters, which represent such temporal and spectral changes, are provided for different combinations of the azimuth angle ( ^^) and the elevation angle ( ^^).

[0051] FIG. 7 shows exemplary sample locations, a.k.a., sample points, on a sphere 750 surrounding user 302 where a set of HR filters, a.k.a., an HR filter set are measured. As shown in FIG.6, each sample point, e.g., 690, may be defined by a pair of an azimuth angle ( ^^) and an elevation angle ( ^^), and as discussed above, the azimuth angle is an angle between axis 612 and a projection, e.g., 604, of a line, e.g., 602, formed by a sample point, e.g., 690, and a center, e.g., 694, of the sphere 750 onto the plane formed by axis 612 and axis 614. The elevation angle is an angle between the line, e.g., 602, and the projection, e.g., 604. In some embodiments, the center of the sphere 750 may correspond to the center of the head of user 302.

[0052] The HR filter set may be used for generating audio depending on the head orientation of user 302. For example, an HR filter at a sample point 712 shown in FIG.7 may be used for generating audio corresponding to a first combination ( ^^^, ^^^) of an azimuth angle and an elevation angle, which corresponds to the audio wave’s firstwhile an HR filter at a sample point 714 included in the HR filter set may be used for generating audio corresponding to a second combination ( ^^ଶ, ^^ଶ) of an azimuth angle and an elevation angle, which corresponds to the audio wave’s second DoA. By generating different audio using different HR filters corresponding to the different head orientations of user 302 and by providing such different audio to user 302, user 302 can perceive the spatial position of the audio source accurately.

[0053] As explained above, rendering a spatial audio source at an arbitrary location in a space generally requires having HR filters at finely sampled locations on the entire sphere, e.g., 750 shown in FIG. 7, surrounding the listener. But many HR filter datasets that are available today, a.k.a., “existing HR filter datasets” or “original HR filter datasets,” do not have HR filters at certain locations, e.g., at certain elevation angles and / or certain azimuth angles. FIGS.8B-8D illustrate examples of these missing HR filters.

[0054] Each of FIGS. 8A-8D shows an exemplary distribution of sample points corresponding to HR filters on the sphere 750 from a view point 720. FIG. 8A shows an ideal distribution of sample points. In FIG. 8A, the distribution of sample points is ideal because it covers the entire range of the elevation angle ( ^^) and the entire range of the azimuth angle ( ^^). Similar to the sample points shown in FIG. 8A, the sample points shown in FIGS. 8B and 8Ccover the entire range of the elevation angle ( ^^) and the entire range of the azimuth angle ( ^^). But, in FIGS.8B and 8C, because the density of the sample points is low, there are missing HR filters within the covered range of the elevation angle ( ^^) and azimuth angle ( ^^). On the contrary, in FIG.8D, the density of the HR filters is low, and the HR filters do not cover the entire range of the elevation angle and the entire range of the azimuth angle. For example, in FIG.8D, there is no HR filters measured at certain elevation angles and at certain azimuth angles.

[0055] Accordingly, in some embodiments of this disclosure, there is provided a method for generating those missing HR filters at locations that lie not only inside but also outside the boundary of a covered range of the HR filter datasets. Here, the covered range means a range of elevation and / or azimuth angles to which the HR filters included in the original HR filter dataset correspond.

[0056] 1. Spatial Interpolation

[0057] As explained above, in FIG. 8B, the HR filters of the original HR filter dataset cover the entire region (e.g., ^^ ∈ ^െగଶ ,గଶ^) of the elevation angle but because the density of thefilters within the region covered by the HR filters. Therefore, in some embodiments, spatial interpolation of the available HR filters in the original HR filter dataset is performed in order to generate the missing HR filters.

[0058] A commonly adopted approach for performing the spatial interpolation is the local neighborhood approach, where a missing HR filter is inferred by weighting the contributions of the available / existing HR filters at its nearest surrounding positions. The existing HR filters, e.g., measured HR filters, may be preprocessed before the interpolation. For example, the existing HR filters at two or more nearest locations may be first converted to minimum phase and then a linear interpolation may be applied.

[0059] Another approach of performing the spatial interpolation is the variational approach. The variational approach is a data-driven approach that linearly transforms the measured HR filters into another space defined by a set of basis functions, where one group of basis functions covers the spatial dimension, and another group covers the frequency dimension. The basis functions can be obtained by eigen-decomposition of the covariance matrix of the measured HR filters. Spherical harmonics (SH), which is complete and orthogonal on a sphere, have been widely used as basis functions to cover the elevation and azimuth angle dimensions.Basic-spline (B-spline) basis functions were introduced in the modeling of HR filters recently.

[0060] However, the above discussed approaches may not be sufficient to generate an accurate HR filter model given limited amount of information available in a spatially sparse sampled HR filter dataset, e.g., elevation and azimuth resolution larger than 10 degrees. To address this problem, an approach based on domain adaptation was developed.

[0061] In the domain adaptation approach, given the spatial sampling spareness of the input incomplete dataset ^^, a proper approach for spatial interpolation may be chosen. ^^ is then modelled accordingly and the resulting interpolation model is ^^୧୬^^୰୮൫ ^^൯.^^୧୬^^୰୮൫ ^^൯ , an interpolated HR filter dataset ^^^ is obtained, where, in ^^^ , the elevationcoverage and the azimuth coverage are the same^^ and ^^ in the inputrespectively.

[0062] In this disclosure, the structure of an interpolation model ^^୧୬^^୰୮is generally denoted as a list of model parameters and / or a list of basis functions. A typical model may be provided in the form of the data list ^^ ൌ^^^, ^^^where ^^ denotes a set of model parameters, and ^^ denotes a set of basis functions, which could be a set of parameters and / or information that is sufficient to construct the basis functions. To simplify notation, the sub- and / or superscripts may be omitted when they are not specifically needed. An exemplary model for spatial interpolation can be found in PCT / EP2023 / 054586 of which the disclosure is incorporated by reference .

[0063] In a summary, the input of the spatial interpolation is the input incomplete HR filter dataset ^^ having HR filters in the range ( ^^) of elevation angle and in the range ( ^^) of azimuth angle, and the output of the spatial interpolation is an interpolated HR filter dataset ^^^ having HR filters in the range ( ^^) of elevation angle and in the range ( ^^) of azimuthIn other words, the spatial interpolation does not change the covered region of the HR filters in the input HR filter dataset. It only changes the density of the HR filters in the covered region.

[0064] 2. Spatial Extrapolation

[0065] In FIG. 8B and / or 8C, the HR filters in the input HR filter dataset ^^ cover the entire region of the elevation angle and the azimuth angle even though there aremissing HR filters within the covered region due to the low density of the HR filters. As explained above, in order to generate these missing HR filters within the covered region, the spatial interpolation may be performed.

[0066] On the contrary, in FIG.8D, the HR filters in the input HR filter dataset ^^ do notcover the entire region of the elevation angle and the azimuth angle. In other words, in FIG.8D, the HR filters cover only a portion of the entire region. To generate these missing HR filters that are outside the covered region, spatial extrapolation may be performed.

[0067] Diffusion Probabilistic Models (DPM) have achieved state-of-the-art results in density estimation as well as in sample quality. Developed upon DPM, a Latent Diffusion Model (LDM) significantly lowers the computational demands of training DPMs while retaining the quality in high-resolution data synthesis. These models learn the underlying distribution of data and have shown excellent inductive biases for image data. Therefore, in some embodiments, an image outpainting, a.k.a., “image extrapolation,” approach using the LDM, which is disclosed in Reference [1] below, is adopted for performing the spatial extrapolation. The LDM may be used to explore the spatial distribution of HR filters.

[0068] FIG. 9A shows a block diagram of a process 900 for performing the spatial extrapolation using a machine learning (ML) model, e.g., an LDM, according to some embodiments. As shown in FIG. 9A, the process 900 includes (1) a first step 902 -- a pre- processing operation 912 and a compression operation 914 --, (2) a second step 904 -- an extrapolation operation --, and (3) a third step 906 -- a decompression operation 922 and a post- processing operation 924.

[0069] The pre-processing operation 912 is to fit an input HR filter dataset to the data structure suitable for using the ML model for spatial extrapolation. FIG. 9B shows a block diagram of the pre-processing operation 912, according to some embodiments. As shown in FIG. 9B, the pre-processing operation 912 may include a slicing operation 952 and a masking operation 954.

[0070] The purpose of the slicing operation 952 is to reduce the amount of HR filter data to be processed at a time such that the extrapolation operation of the HR filter data does not cause huge demands in computation time and energy resources. The input to the slicing operation 952 may be either (1) the original incomplete HR filter dataset ^^ in case the resolution of the original HR filter dataset is high enough or (2) the spatiallyHR filter dataset ^^^ in case the resolution of the original HR filter dataset ^^ is not high enough. Moreover,HR filters might be resampled in the time domain, ifFor simple explanation purpose, the spatially interpolated HR filter dataset ^^^ will be used as an example of the input of the spatialoperation 952.

[0071] The interpolated HR filter dataset ^^^ is a three-dimensional (3D) data, i.e., ^^^ ∈ℝெഇൈெഝൈே, where ^^ఏis a set of the angles to which HR filters in the^^^ of the azimuth angles to which HR filters in the dataset ^^^ correspond,each HR filter in the dataset ^^^. During the slicing952, the dimension of the dataset ^^^ is reduced to a lowerFor example, the 3D dimension ofthe dataset ^^^ may beto a 2D dimension during the slicing operation 952.

[0072] There are different ways to reduce the dimension of the dataset ^^^. These differentways are illustrated in FIGS. 10A-10D. FIG. 10A shows a cube 1002illustrates the 3D interpolated HR filter dataset ^^^. As shown in FIG. 10A, ^^^ corresponds to data in 3D – i.e.,^^ఏൈ ^^థൈ ^^. To reduce the dimension of ^^^, themay be sliced in any one of the ^^ఏ,^^థ, ^^.

[0073] According to some embodiments, the direction of the slicing may be determined based on where the missing HR filters are located. For example, if the missing HR filters are located in certain elevation angle ranges, the dimension of ^^^ may be sliced in ^^థ direction1006. In another example, if the missing HR filters arein certainranges, the dimension of ^^^ may be sliced in ^^ఏ direction 1004. In further example, if the missing HRfilters arein certain elevation angle ranges and in certain azimuth angle ranges, the dimension of ^^^ may be sliced in ^^ direction 1008.

[0074] 10B shows a result of slicing the dimension of ^^^ in the ^^థdirection 1006. As shown in FIG. 10B, slicing the dimension of ^^^ in the ^^థresult in thereduced two-dimensional (2D) HR filter data,having ^^థ slices of data, each ofwhich has a size of ^^ ఏ ൈ ^^ , i.e., ^ ^^^^^ெഝ∈ ℝഇൈே: ^^థ ൌ 1, … ,. Here, note that thedimension of the 2D HR filter data^^ఏindicates the range of the elevation angle that is covered by the HR filters in ^^^^^ഝ is less than the full range ^^ఏ.

[0075] FIG. 10C shows a result of slicing theof ^^^ in the ^^ఏdirection 1004. As shown in FIG.10C, slicing the dimension of ^^^ in the ^^ఏ1004 would result in the reduced 2D HR filter data, denoted by ^^^^ ^^ఏ slices of data, each of which has a size of^^ థ ൈ ^^, i.e., ^ ^^^^^ഇ ∈ ℝெ ഝൈே: ^^ఏ ൌ 1, … , ^^ఏ^. Here, note that the dimension of the 2D HRfilter dataas ^^థൈ ^^, not ^^థൈ ^^. ^^థindicates the range of the azimuth angle that is covered by theinthe full range ^^థ.

[0076] FIG.10D shows a result of slicing the dimension ofin the ^^ direction 1008. As shown in FIG. 10D, slicing the dimension of ^^^ in in the ^^1008 would result in the reduced 2D HR filter data, denoted by ^^^^^^ slices of data, each of which has a size of^^ ఏ ൈ ^^ థ, i.e., ^ ^^^^^ ∈ ℝெ ഇൈெ ഝ:, ^^^. Here, note that the dimension of the 2D HR filter^^ఏൈ ^^థ, not ^^ఏൈ ^^థ. ^^ఏindicates the range of the elevation angle that is covered by the HR^^^^^ isthe full range ^^ఏ and ^^ థ indicates the range ofthe azimuth angle that isby the HR filters in ^^^^^ isfull range ^^థ.

[0077] Each slice is then normalized betweenvalue range [-1, 1]slice’s maximum value and minimum value. The normalization removes the level differences of filters among slices. Therefore, the normalization factors are stored and used in the post-processing operation for level recovery.

[0078] After performing the slicing operation 952, the masking operation 954 may be performed on the output of the slicing operation 952 – i.e., ^^^^ ൌ ^ ^^, ^^, ^^^^୪^ഝ , ^^^^୰^ഝ : ^^థ ൌ1, … , ^^ ^ , wher ^^୪ / ୰ ெ^ ഇൈே ^ഝ ^ ^^୪^ ^ഇ ^୰థ e ^^ ∈ ℝ , or ^^ ൌ ^ ^^, ^^, ^^ , ^^^ഇ :∈the missing HR filters that is outside of the covered region. The masking operation 954 may be performed differently based on the direction of the slicing operation 952.

[0080] For example, in case the dimension of ^^^ is sliced in the ^^థdirection 1006, the output of the slicing operation 952 would∈ ℝெ ഇ, aswith respectFIG.10B. In this case, during the maskingHR filters having HR filter values of 0 are added to the region(s) of the elevation angle, corresponding to the missing HR filters. For example, in case ^^^^ has missiగ గ^ഝ ng HR filters in the elevation angle ranges of [െ ଶ ,െ ଷ] and[గଷ,గଶ], HR filters having the HR filter values of 0 may be added to the ^^^^^ഝ in the elevationangle ranges of [െగ,െగ] and [గ,గ],^^^^^ଶଷ ଷ ଶ resulting in ^^^ഝ . Morethe output of themasking operation 954 wouldൌ ^ ^^,ഝ , ^^^^୰,^^^^ഝ : ^^థ ൌ 1, … , ^^థ^, where ^^^^୪ / ୰^ഝ ∈ℝெ ഇൈேand ^^ ∈ ℝெ ഇൈேis the0. 1 indicates the data is available and 0 indicates the data is missing. Here, note that, after the masking operation 954, the dimension of the HR filter dataset is changed from ℝெ ഇൈேto ℝெഇൈே, where ^^ఏin ℝெ ഇൈேcorresponds to a full range of the elevation angle.another example, in case the dimension of ^^^ is sliced in the ^^ఏdirection 1004, the output of the slicing operation 952 would include ^^^∈ ℝெ ൈே, as explained withto FIG.10C. In this case, during the masking operation 954, HR filters having HR filter values of 0 are added to the region(s) in the azimuth angle, corresponding to the missing HR filters. For example, in case ^^^^ hasହగ^ഇ missing HR filters in the azimuth angle ranges of [െ ^^,െ ^ ] and[ହగ, ^^], HRthe HR filter values of 0 may be added to the ^^^^ in theangle ranges of [െ ^^,െହగ ହగ^] and [ ^ , ^^], resulting in ^^^^^^^ి^ഇ. More specifically, the output of themasking operation 954 would be ^^^ൌ ^ ^^, ^^, ^^,, ^^^^୰,^ ^^ഇ : ^^ఏ ൌ 1, … , ^^ ^, where ^^^ ഇ ∈ℝெഝൈேand ^^ ∈ ℝெ ഝൈே. Here,thefrom ℝெ ഝൈேto ℝெഝൈே, where ^^థin ℝெഝൈேcorresponds to a full range of the azimuth angle.

[0082] In a summary, according to some embodiments, the output of the pre-processing operation 912 is ℱ^ ^ ^ ^୍൫ ^^^൯ ൌ ^ ^^, ^^, ^^, ^^^^୪, ^^ഝ , ^^^^୰, ^^ഝ , : ^^థ ൌ 1, … , ^^థ^ in case the missing HR filterscorrespond to somein case the missing HR filters correspond to some regions of the azimuth angle, the output of the pre-processing operation 912 is ℱ୍൫ ^^^൯ ൌ ^ ^^, ^^, ^^, ^^^^୪,^^^^ഇ , ^^^^୰,^^^^ഇ , : ^^ఏ ൌ 1, … , ^^ఏ^ . For simple explanation purpose, in theHR filters correspond to some regions of the elevation angle. But the embodiments below are equally applicable to the scenario where the missing HR filters correspond to some regions of the azimuth angle, or the missing HR filters correspond to some regions of both the elevation angle and the azimuth angle.

[0083] After performing the pre-processing operation 912, the compression operation 914, e.g., ^^^⋅^, may be performed. During the compression operation 914, a compression model may be applied on ℱ୍൫^ ^^^^൯ , meaning ^^ ^ℱ୍൫^ ^^^^൯^ , and its latent representation is ^^୍൫ ^^^൯ ൌ^ ^^, ^^, ^^′^ ^^^,ഝ , ^^ ^ ^^^^୰,^^^^ഝ, ^^థ^. To achieve high quality dataHR filter slices. Note that the concealer mask contains only ones and zeros, and thus a simple down-sampling may substitute the compression operation 914.

[0084] After performing the step s902 -- the pre-processing operation 912 and the compression operation 914 --, the process 900 may proceed to step s904.

[0085] In the step s904, the latent representation ^^൫ ^^^൯ passes through the model ^^^^^୰ୟthat generates a second latent representation ^^^൫ ^^^൯. ^^^൫ ^^^൯ carries the inferred informationabout the HR filters at the missing elevationexample, in case the missing HR filters correspond to the elevation angle ranges of [െగ గ గ గଶ ,െଷ] and [ଷ,ଶ], during the masking operation 954, HR filters having the HR filter values of 0 areinput HR filter dataset in the elevation angle ranges of [െగ,െగ] and [గ,గଶଷ ଷ ଶ], and, in the step s904, these 0 values are changed to real meaning values.

[0086] ^^^^^୰ୟ are generative models for predicting the missing data in ^^^ through areverse process conditioned on the concealer mask ^^. The reverse processa prior, which is typically chosen as a standard normal distribution. Details regarding the reverse process is provided in Reference [1] of which the disclosure is hereby incorporated by reference.

[0087] Then the process 900 may proceed to step s906.

[0088] The step s906 comprises a decompressing operation 922, e.g., ^^^⋅^, and a post processing operation 924. During the decompressing operation 922, ^^^൫ ^^^൯ is decompressed,thereby obtaining a set of HR filters at the complete set of azimuth angles ^ ^^, ^ ^^ in the impulse response space – obtaining ^^ ^ ^^^൫ ^^^൯^.

[0089] After performing the922, the process 900 may proceed to the post processing operation 924, e.g., ℱି^^⋅^, which may include time-domain resample, denormalization using theetc. One or more functions of thepost-processing operation 924 may be optional.

[0090] The output of the process 900 may be a complete dataset ^^^, which covers thecomplete range of the elevation angles -- from െ ^^ / 2 to ^^ / 2, without any missing HR filters or with a reduced number of missing HR filters. Alternatively, during the process 900, ^^^ could be modeled for interpolation as a continuous function of elevation and azimuth,by^^୧୬^^୰୮൫ ^^^൯. The new interpolation model of the complete dataset ^^୧୬^^୰୮൫ ^^^൯. In furthervariation, the output of the process 900 may be another complete dataset ^^^ that is generatedfrom ^^୧୬^^୰୮൫ ^^^൯.

[0091] 3. Training the ML model ^^^^ ^^ ^^ ^^ ^^

[0092] A step-by-step description of the training procedure for obtaining the model ^^^^^୰ୟis provided below.

[0093] In order to make the model ^^^^^୰ୟto learn the underlying spatial and temporal distribution of HR filters on a full sphere such that missing HR filters can be generated using existing HR filters, each of the HR filter datasets used for training the model ^^^^^୰ୟneeds to satisfy two requirements: (1) Requirement 1 – The training dataset’s spatial sampling points must cover the full sphere; and (2) Requirement 2 – The training dataset’s spatial and temporal sampling resolution must be high enough.

[0094] Let^^^^: ^^ ൌ 1, … , ^^^denotes a set of training HR filter datasets, where ^^ is the total number of HR filter datasets included in the set. To meet the Requirement 1, each ^^^must be a complete dataset, meaning that HR filters in each ^^^must cover the full range of the elevation angle -- ^^ ∈ ^െగ గଶ ,ଶ^ -- and the full range of the azimuth angle -- ^^ ∈^0, 2 ^^^. To meet the Requirement 2,situation, the spatial sampling resolution of each ^^^should meet the minimum audible angle (MAA) requirement. Otherwise, the spatial interpolation described above may be applied and the resulting interpolated HR filter dataset ^^^^may be used in the training. An example of the spatial sampling resolution appropriate for the training HR filter datasets is 2 degrees for both the elevation and azimuth angles. Note that, for the purpose of simplifying the notation, in the description below, ^^^may be used to indicate both the original dataset and the interpolated dataset.

[0095] As disclosed below, the training procedure may include a pre-processing operation, a compression operation, and a modelling operation.

[0096] Pre-processing Operation in the ML Model Training

[0097] The purpose of the pre-processing operation in training the model ^^^^^୰ୟis to fit an initial training HR filter dataset to the data structure suitable for training the model ^^^^^୰ୟvia spatial interpolation and time domain resampling while improving knowledge retention via masking. A block diagram of the pre-processing operation is shown in FIG. 11. As shown in FIG.11, the pro-processing operation includes a slicing operation 1102and a masking operation 1104.

[0098] Slicing Operation 1102

[0099] Like the slicing operation for the spatial extrapolation, the purpose of the slicing operation in training the model ^^^^^୰ୟis to provide training data that provides sufficient spatial and temporal details but does not require a huge demand in computation time and energy resources. The method of performing the slicing operation for the model training is similar to the method of performing the slicing operation for the spatial extrapolation.

[0100] The difference is that, in the spatial extrapolation, the slicing operation is performed on an incomplete HR filter dataset, e.g., an HR filter dataset including missing HR filters in certain elevation angle ranges, while, in the model training, a complete HR filter dataset, i.e., an HR filter dataset that includes HR filters covering the full range of the elevation and azimuth angles, is used as required by the Requirement 1 above.

[0101] In the slicing operation of the model training, the direction of the slicing may be determined based on whether the model will be used for generating missing HR filters in (i) a certain elevation angle range, (ii) a certain azimuth angle range, or (iii) certain elevation and azimuth angle ranges.

[0102] For example, if the missing HR filters are in a certain elevation angle range, the slicing operation may be performed in the ^^థdimension, as illustrated in FIG.10B. In another example, if the missing HR filters are inazimuth angle range, the slicing operation may be performed in the ^^ఏdimension, as illustrated in FIG.10C. In further example, if the missing HR filters are in certain azimuth and elevation angle ranges, the slicing operation may be performed in the ^^ dimension, as illustrated in FIG.10D.

[0103] Masking Operation 1104

[0104] The purpose of the masking operation is to create an input HR filter dataset for training the model where the input HR filter dataset is a complete HR filter dataset with maskedHR filters at certain angles, e.g., the HR filters having masking value(s). This masked HR filter dataset helps the model ^^^^^୰ୟto learn the underlying distribution of input data -- i.e., an input HR filter dataset with missing HR filters -- when there are missing samples, i.e., the missing HR filters) in the input data.

[0105] In training the model ^^^^^୰ୟ, the masking operation may be performed by applying masks of certain shapes at different positions on a slice of HR filters, e.g., ^^^^ഝ, and by producing a group of masked versions of the slice, denoted by ^ ^^^൫^^൯^ഝ: ^^ ൌ 1, … , ^^^, where ^^^is the ^^-th mask, and ^^ is the number of concealer masks.

[0106] FIG. 12 illustrates a simplified example of the masking operation according to some embodiments. In FIG.12, a slice of HR filters corresponding to ^^ ൌ 30°, which is defined in the two-dimensions of ^^ ൈ ^^, is shown. Each small square shown in FIG.12 indicates an HR filter in the two dimensions. For example, the top left square in the grid of squares may indicate an HR filter at ^^ ൌ 30° and ^^ ൌ 0°. Each HR filter may be associated with an HR filter value. In FIG.12, during the masking operation, a first mask 1202 having values 0 and a second mask 1204 having values 0 are applied to HR filters in certain elevation angles (e.g., ∈ ^െగ ଷగଶ ,െ଼^ and଼columns to be 0.

[0107] The shape and / or the size of each make can be selected such that it covers the area where the missing HR filters may exist. For simple explanation purpose, it is assumed that the missing HR filters exist at certain elevation angle range, where the area could be represented by a column of length ^^ and a width corresponding to a certain azimuth angle range.

[0108] Alternatively, the shape and / or the size of each mask can be random. More specifically, in some embodiments, synthetic column masks each having a random width are generated at random positions. The width of the mask may be specified by a uniform distribution within the range^^^୫, ^^୫^. The value of ^^୫may represent the minimum number of elevation angles where HR filters are likely to be missing, and the value of ^^୫may represent the maximum number of elevation angles where HR filters are likely to be missing. An example of the range is [30, 256] for ^^ఏൌ 512 corresponding to a width roughly between 10 to 90 degrees in the elevation angle. The position of the mask may be specified by a uniform distribution conditionedon the width of the mask.

[0109] The generated masks may then be applied on each slice. The output of the masking process is ^ ^^^൫^^൯, ^^ : ^^ ൌ 1, … , ^^; ^^ ൌ 0, … , ^^^ , where ^^^^^^ ^୪,൫^^൯^ ^ ^^ ൌ ^ ^^, ^^, ^ ^^^,^ഝ : ^^థ ൌmask. ^^^୰,൫^^൯^,^ഝ is a slice of right HR filters at the ^^థ-th azimuth angle with the ^^-th mask. Aof masks is ^^ ൌ 1000.

[0110] Compression Operation in the ML Model Training

[0111] The compression operation is to convert the pre-processed data (e.g., ℱ^^^^^^^^)from the original representation in the impulse response space intorepresentation in a latent space to save computational cost in training the model and performing an inference process using the model.

[0112] Either the slice of the masked HR filters, e.g., ^^^൫େ^൯^ഝ, or the concealer masks, e.g., ^^^, has the same data structure as its original slice, e.g., ^^^^ഝ, which is a two-dimensional data of size ^^ఏൈ ^^ . Thus, according to some embodiments, the compression operation may be performed for each of the slices of the masked HR filters and the masks.

[0113] Any one of various techniques that are currently available may be used for performing the compression operation. For example, in some embodiments, the method of described in in Reference [1], the disclosure of which is incorporated by reference, is used to train an encoding / decoding model which provides a good compression rate as well as a good reconstruction. This encoding / decoding model may be used to perform the compression operation. Note that the concealer mask contains only ones and zeros, and a simple down- sampling might be already enough.

[0114] One example of a scale factor used for this compression operation is 8 given a size of ^^ఏൈ ^^ ൌ 512 ൈ 512. The output of the compression operation in the model training is alatent representation of ℱ^^^^^^^^, denoted by ^^^^^^^^^^ൌ^ ^^, ^^, , ^^′^ ^^^^, ^^ ^ ^^^୪,൫^^൯^,^ഝ , ^,^ഝ ^ , , ^, where the size of each slice is scaled down by thescale

[0115] Modeling Operation in the ML Model Training

[0116] According to some embodiments, the latent representation ^^^^^^^^^^may bemodeled using an ML model, e.g., a DM such as an LDM. One exemplary way of modeling the latent representation is described in Reference [1], the disclosure of which is herein incorporated by reference. https: / / github.com / Stability-AI / stablediffusion provides an example of implementation of the modeling. Via the modeling, the model ^^^^^୰ୟused for performing the spatial extrapolation of HR filters can be obtained.

[0117] ^^^^^୰ୟare generative models that are trained through a forward process or a diffusion process conditioned on masks, where the approximate posterior is fixed to a Markov chain that gradually adds Gaussian noise according to a variance schedule to a slice of HR filters. Details regarding the diffusion process that obtains the model is provided in Reference [1] of which the disclosure is hereby incorporated by reference.

[0118] In a summary, the process of performing the modeling involves two main components: 1) a data compression / decompression component, which converts the input HR filters between the original representation in the impulse response space and an efficient, low- dimensional representation in a latent space, and 2) a generative modeling component, which models the distribution of HR filters in its latent space. The low-dimension latent representation of HR filters abstracts away imperceptible details while retains the capability of reconstructing perceptible details. The generative model learns general features of HR filters in the latent space.

[0119] FIG. 13 shows a process 1300 of generating a first set of head-related, HR, filters for producing spatial audio signals, according to some embodiments. The first set of HR filters corresponds to a first spatial range. The process 1300 may begin with step s1302. The step s1302 comprises obtaining a second set of HR filters corresponding to a second spatial range, wherein the second spatial range is a portion of the first spatial range. Step s1304 comprises generating a third set of HR filters by adding to the second set of HR filters a subset of HR filters corresponding to a third spatial range, wherein the third spatial range does not overlap the second spatial range. Step s1306 comprises, using a machine learning, ML, model, generating the first set of HR filters corresponding to the first spatial range based on the third set of HR filters, wherein the first spatial range includes the second and third spatial ranges.

[0120] In some embodiments, the first spatial range is equal to a sum of the second spatial range and the third spatial range.

[0121] In some embodiments, an HR filter is associated with a coordinate defined at least by an elevation angle and an azimuth angle, and each of the first, second, and third spatial ranges is a different spatial range of the elevation angle and / or the azimuth angle.

[0122] In some embodiments, each HR filter included in the subset of HR filters corresponding to the third spatial range is associated with an HR filter value, and the HR filter values are the same.

[0123] In some embodiments, each of the HR filter values is 0.

[0124] In some embodiments, the process 1300 comprises obtaining D-dimensional filter data corresponding to the second set of HR filters; and converting the D-dimensional filter data into ^^ slices of (D-1)-dimensional filter data, wherein ^^ and D are positive integers.

[0125] In some embodiments, ^^ is equal to one of: a number of elevation angles to which the second set of HR filters corresponds, or a number of azimuth angles to which the second set of HR filters corresponds, or the length of an HR filter included in the second set of HR filters.

[0126] In some embodiments, the D-dimensional filter data is three dimensional filter data that is defined by an elevation angle, an azimuth angle, and a filter tap, and ^^ is equal to the number of azimuth angles to which the second set of HR filters corresponds.

[0127] In some embodiments, adding the subset of HR filters corresponding to the third spatial range to the second set of HR filters comprises adding to each of the ^^ slices of the (D-1)- dimensional filter data the subset of HR filters corresponding to the third spatial range.

[0128] In some embodiments, the process 1900 comprises obtaining a fourth set of HR filters corresponding to the second spatial range, wherein the number of HR filters included in the fourth set is lower than the number of HR filters included in the second set of HR filters, and adding a plurality of HR filters to the fourth set of HR filters, thereby obtaining the second set of HR filters, wherein each of the plurality of HR filters is determined based on two or more HR filters included in the fourth set of HR filters.

[0129] In some embodiments, each of the plurality of HR filters is determined based on an interpolation of two or more HR filters included in the fourth set.

[0130] In some embodiments, generating the first set of HR filters using the ML model comprises converting the HR filter values of the subset of HR filters included in the third set of HR filters from 0 to non-zero values.

[0131] In some embodiments, the ML model is trained by: obtaining at least one set of D-dimensional training HR filter data; and converting the D-dimensional training HR filter data into M slices of (D-1)-dimensional training HR filter data, wherein M and D are positive integers.

[0132] In some embodiments, each of the M slices of the (D-1) dimensional training HR filter data corresponds to the first spatial range, and the ML model is trained by: (i) selecting a group of HR filter values included in each of the M slices of the (D-1) dimensional training HR filter data; and (ii) changing the selected group of HR filter values to be the same, thereby generating masked HR filter values for each of the M slices.

[0133] In some embodiments, the group of HR filter values included in each of the M slices of the (D-1) dimensional training HR filter data is randomly selected.

[0134] In some embodiments, the process 1900 comprises repeatedly performing the steps (i) and (ii) of embodiment A14 for a preset number of times.

[0135] In some embodiments, the ML model is trained by performing a forward process or a diffusion process conditioned on the masked HR filter values.

[0136] FIG.14 is a block diagram of an apparatus 1400, according to some embodiments, for performing the methods described above (e.g., process 1200 shown in FIG. 12). Apparatus 1400 may be included in any one or more of: server 214, audio rendering unit 212, or headphone 206. As shown in FIG.14, apparatus 1400 may comprise: processing circuitry (PC) 1402, which may include one or more processors (P) 1455 (e.g., a general purpose microprocessor and / or one or more other processors, such as an application specific integrated circuit (ASIC), field- programmable gate arrays (FPGAs), and the like), which processors may be co-located in a single housing or in a single data center or may be geographically distributed (i.e., apparatus 1400 may be a distributed computing apparatus); at least one network interface 1448, each network interface 1448 comprises a transmitter (Tx) 1445 and a receiver (Rx) 1447 for enabling apparatus 1400 to transmit data to and receive data from other nodes connected to a network 210 (e.g., an Internet Protocol (IP) network) to which network interface 1448 is connected (directly or indirectly) (e.g., network interface 1448 may be wirelessly connected to the network 210, in which case network interface 1448 is connected to an antenna arrangement); and one or more storage units (a.k.a., “data storage system”) 1408, which may include one or more non-volatile storage devices and / or one or more volatile storage devices. In embodiments where PC 1402 includes a programmable processor, a computer program product (CPP) 1441 may be provided. CPP 1441 includes a computer readable medium (CRM) 1442 storing a computer program (CP) 1443 comprisingcomputer readable instructions (CRI) 1444. CRM 1442 may be a non-transitory computer readable medium, such as, magnetic media (e.g., a hard disk), optical media, memory devices (e.g., random access memory, flash memory), and the like. In some embodiments, the CRI 1444 of computer program 1443 is configured such that when executed by PC 1402, the CRI causes apparatus 1400 to perform steps described herein (e.g., steps described herein with reference to the flow charts). In other embodiments, apparatus 1400 may be configured to perform steps described herein without the need for code. That is, for example, PC 1402 may consist merely of one or more ASICs. Hence, the features of the embodiments described herein may be implemented in hardware and / or software.

[0137] While various embodiments are described herein, it should be understood that they have been presented by way of example only, and not limitation. Thus, the breadth and scope of this disclosure should not be limited by any of the above described exemplary embodiments. Moreover, any combination of the above-described elements in all possible variations thereof is encompassed by the disclosure unless otherwise indicated herein or otherwise clearly contradicted by context.

[0138] As used herein transmitting a message “to” or “toward” an intended recipient encompasses transmitting the message directly to the intended recipient or transmitting the message indirectly to the intended recipient (i.e., one or more other nodes are used to relay the message from the source node to the intended recipient). Likewise, as used herein receiving a message “from” a sender encompasses receiving the message directly from the sender or indirectly from the sender (i.e., one or more nodes are used to relay the message from the sender to the receiving node). Further, as used herein “a” means “at least one” or “one or more.”

[0139] Additionally, while the processes described above and illustrated in the drawings are shown as a sequence of steps, this was done solely for the sake of illustration. Accordingly, it is contemplated that some steps may be added, some steps may be omitted, the order of the steps may be re-arranged, and some steps may be performed in parallel.

[0140] Reference List [1] R. Rombach, A. Blattmann, D. Lorenz, P. Esser and B. Ommer, "High-Resolution Image Synthesis with Latent Diffusion Models," in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.[2] L. Liu, Y. Ren, Z. Lin and Z. Zhao, "Pseudo Numerical Methods for Diffusion Models on Manifolds," in Proceeding of International Conference on Learning Representations (ICLR), 2022. [3] V. R. Algazi, R. O. Duda, D. M. Thompson and C. Avendano, “The CIPIC HRTF Database,” in IEEE Workshop on Applications of Signal Processing to Audio and Electroacoustics, Mohonk Mountain House, New Paltz, NY, 2001.

Claims

CLAIMS 1. A method (1300) of generating a first set of head-related, HR, filters for producing spatial audio signals, wherein the first set of HR filters corresponds to a first spatial range, the method comprising: obtaining (s1302) a second set of HR filters corresponding to a second spatial range, wherein the second spatial range is a portion of the first spatial range; generating (s1304) a third set of HR filters by adding to the second set of HR filters a subset of HR filters corresponding to a third spatial range, wherein the third spatial range does not overlap the second spatial range; and using a machine learning, ML, model, generating (s1306) the first set of HR filters corresponding to the first spatial range based on the third set of HR filters, wherein the first spatial range includes the second and third spatial ranges.

2. The method of claim 1, wherein the first spatial range is equal to a sum of the second spatial range and the third spatial range.

3. The method of at least one of claims 1 and 2, wherein an HR filter is associated with a coordinate defined at least by an elevation angle and an azimuth angle, and each of the first, second, and third spatial ranges is a different spatial range of the elevation angle and / or the azimuth angle.

4. The method of at least one of claims 1-3, wherein each HR filter included in the subset of HR filters corresponding to the third spatial range is associated with an HR filter value, and the HR filter values are the same.

5. The method of claim 4, wherein each of the HR filter values is 0.

6. The method of at least one of claims 1-5, comprising:obtaining D-dimensional filter data corresponding to the second set of HR filters; converting the D-dimensional filter data into ^^ slices of (D-1)-dimensional filter data, wherein ^^ and D are positive integers.

7. The method of claim 6, wherein ^^ is equal to one of: a number of elevation angles to which the second set of HR filters corresponds, or a number of azimuth angles to which the second set of HR filters corresponds, or the length of an HR filter included in the second set of HR filters.

8. The method of claim 7, wherein the D-dimensional filter data is three dimensional filter data that is defined by an elevation angle, an azimuth angle, and a filter tap, and ^^ is equal to the number of azimuth angles to which the second set of HR filters corresponds.

9. The method of at least one of claims 6-8, wherein adding the subset of HR filters corresponding to the third spatial range to the second set of HR filters comprises adding to each of the ^^ slices of the (D-1)-dimensional filter data the subset of HR filters corresponding to the third spatial range.

10. The method of at least one of claims 1-9, comprising: obtaining a fourth set of HR filters corresponding to the second spatial range, wherein the number of HR filters included in the fourth set is lower than the number of HR filters included in the second set of HR filters, and adding a plurality of HR filters to the fourth set of HR filters, thereby obtaining the second set of HR filters, wherein each of the plurality of HR filters is determined based on two or more HR filters included in the fourth set of HR filters.

11. The method of claim 10, wherein each of the plurality of HR filters is determined based on an interpolation of two or more HR filters included in the fourth set.

12. The method of at least one of claims 1-11, wherein generating the first set of HR filters using the ML model comprises converting the HR filter values of the subset of HR filters included in the third set of HR filters from 0 to non-zero values.

12. The method of at least one of claims 1-11, wherein the ML model is trained by: obtaining at least one set of D-dimensional training HR filter data; and converting the D-dimensional training HR filter data into M slices of (D-1)-dimensional training HR filter data, wherein M and D are positive integers.

13. The method of claim 12, wherein each of the M slices of the (D-1) dimensional training HR filter data corresponds to the first spatial range, and the ML model is trained by: (i) selecting a group of HR filter values included in each of the M slices of the (D-1) dimensional training HR filter data; and (ii) changing the selected group of HR filter values to be the same, thereby generating masked HR filter values for each of the M slices.

14. The method of claim 13, wherein the group of HR filter values included in each of the M slices of the (D-1) dimensional training HR filter data is randomly selected.

15. The method of at least one of claims 13 and 14, the method comprising: repeatedly performing the steps (i) and (ii) of claim 13 for a preset number of times.

16. The method of at least one of claims 13-15, wherein the ML model is trained by performing a forward process or a diffusion process conditioned on the masked HR filter values.

17. A computer program (1400) comprising instructions (1444) which when executed by processing circuitry (1402) cause the processing circuitry to perform the method of at least one of claims 1-16.

18. A carrier containing the computer program of claim 17, wherein the carrier is one of an electronic signal, an optical signal, a radio signal, and a computer readable storage medium.

19. An apparatus (1400) for generating a first set of head-related, HR, filters for producing spatial audio signals, wherein the first set of HR filters corresponds to a first spatial range, the apparatus being configured to: obtain (s1302) a second set of HR filters corresponding to a second spatial range, wherein the second spatial range is a portion of the first spatial range; generate (s1304) a third set of HR filters by adding to the second set of HR filters a subset of HR filters corresponding to a third spatial range, wherein the third spatial range does not overlap the second spatial range; and using a machine learning, ML, model, generate (s1306) the first set of HR filters corresponding to the first spatial range based on the third set of HR filters, wherein the first spatial range includes the second and third spatial ranges.

20. The apparatus of claim 19, wherein the apparatus is configured to perform the method of at least one of claims 2-16.

21. An apparatus (1400) comprising: a processing circuitry (1402); and a memory (1441), said memory containing instructions executable by said processing circuitry, whereby the apparatus is operative to perform the method of at least one of claims 1- 16.

Images