Error correction for head-related filters

Abstract

To provide a method for generating a final corrected head-related (HR) filter dataset.SOLUTION: A process 800 for obtaining a first corrected HR filter dataset H'I includes: Step s802 of obtaining an initial HR filter dataset HI; Step s804 of obtaining an extracted HR filter dataset HX extracted from the initial HR filter dataset HI; Step S806 of obtaining a model MX of the extracted HR filter dataset HX; Step s808 of generating a modelled HR filter dataset HM using the model MX; Step s810 of selecting for correcting one or more HR filters that are included in the initial HR filter dataset HI; and Step s812 of generating a first corrected HR filter dataset H'I by correcting the selected one or more HR filters.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] Embodiments relating to error correction of head-related (HR) filters are disclosed. [Background technology]

[0002] Figure 1 shows sound waves propagating towards the listener from the direction of arrival (DOA), specified by an elevation-azimuth pair in spherical coordinates. Before reaching the listener's left and right eardrums, the sound waves interact with the listener's upper torso, head, outer ear, and surrounding materials. This interaction results in temporal and spectral changes in the waveforms reaching the left and right eardrums, some of which are DOA-dependent.

[0003] The human auditory system has learned to interpret these variations in order to infer the various spatial properties of sound waves themselves, as well as the acoustic environment in which the listener is located. This ability is called spatial hearing, and it relates to how humans evaluate spatial cues embedded in binaural signals—sound signals in the right and left ear canals—to infer the location of auditory events induced by sound events (physical sound sources) and the acoustic properties of the physical environment in which the human is located (e.g., a small room, a tiled bathroom, an auditorium, a room without windows (cave)). This human ability, spatial hearing, can be leveraged to create virtual spatial audio scenes by reintroducing spatial cues into binaural signals, which would lead to a specific spatial perception of sound.

[0004] The main spatial cues include 1) angular-relational cues: binaural cues, i.e., interaural level difference (ILD) and interaural time difference (ITD), as well as monaural (or spectral) cues, and 2) distance-relational cues: intensity and direct-to-reverberation (D / R) energy ratio. The mathematical representation of the short-duration DOA-dependent temporal and spectral changes of a waveform (e.g., 1-5 milliseconds) is the so-called head-related (HR) filter. The frequency-domain (FD) representation of these filters is the so-called head-related transfer function (HRTF), and the time-domain (TD) representation of the filter is the head-related impulse response (HRIR).

[0005] Figure 2 shows an example of the ITD and spectral cue of a sound wave propagating towards the listener. The two plots show the magnitude response of a pair of HR filters acquired at an elevation angle of 0 degrees and an azimuth angle of 40 degrees (data is from the CIPIC database: subject ID 28. The database is publicly available and can be accessed via the link https: / / www.ece.ucdavis.edu / cipic / spatial-sound / hrtf-data / ). HR filter-based binaural rendering techniques are gradually becoming established, where a spatial audio scene is generated by directly filtering the audio source signal with a pair of HR filters at a desired location. This technique is particularly attractive for many emerging applications, such as extended reality including virtual reality (VR), augmented reality (AR), and mixed reality (MR), and mobile communication systems where headsets are typically used.

[0006] An HR filter dataset, sometimes called an HR filter database, is a collection of left and right HR filters sampled at specific spherical angles or directions (elevation and azimuth) around the listener and other relational metadata, often obtained by acoustic measurements. Three steps—binaural recording, reference recording, and post-processing—are involved in obtaining the HR filters in the HR filter dataset.

[0007] Step 1: Binaural Recording

[0008] Binaural recordings aimed at obtaining HR filtering are typically performed in an anechoic chamber. The recording setup consists of a loudspeaker system, an in-ear binaural microphone system, mechanical systems for loudspeaker and listener positioning, and several auxiliary devices and software.

[0009] Figure 3 shows a simplified setup for HR-filtered binaural recording. The object to be listened to (e.g., an artificial head, mannequin, or human object) is positioned at the center of the mechanical system such that the center of the object's head is at the measurement origin (0,0,0). An excitation signal is generated and reproduced through a sound emitter, e.g., a loudspeaker, positioned on a sphere of a constant radius. The position can be denoted by (θ,φ,r), where θ corresponds to the elevation angle, φ corresponds to the azimuth angle, and r corresponds to the radius, which is the distance from the center of the listener's head to the sound emitter. The signal reaching the two ears is recorded by in-ear microphones. This measurement is repeated while varying the spatial position of the excitation signal relative to the listener by varying the positions of the listener, the sound emitter, or both in different dimensions. Figure 4 shows an example of a sampling grid on a sphere, where the dots indicate the positions where recordings were made.

[0010] Step 2: Reference Recording

[0011] A raw binaural recording includes not only the HR impulse responses, but also the impulse responses of the entire recording system including loudspeakers, binaural microphones, AD / DA converters, and amplifiers. A reference recording is then made separately for each in-ear microphone. The recording protocol is similar to the binaural recording by removing the subject and placing the microphone at position (0,0,0).

[0012] Step 3: Post-processing

[0013] Post-processing: Free Field Equalization

[0014] A common procedure for removing the influence of the recording system from raw binaural recordings is free field equalization. y l / r Let (t;θ,Φ,r) denote the signal recorded at the left / right in-ear microphone when the excitation signal is emitted in direction (θ,Φ,r). Note that the radius r is omitted in the rest of the present disclosure for simplicity, since r is usually constant in HR filter measurements. Let s(t) denote the excitation signal. In that case, y l / r (t;θ,Φ) is the excitation signal convolved with an unknown impulse response, i.e., y l / r (t;θ,Φ)=s(t)*h l / r (t;θ,Φ)*g l / r described as (t). h l / r (t;θ,Φ) denotes the left / right ear HR impulse response, and g l / r (t) denotes the impulse response of the recording system using the left / right in-ear microphone. x l / r Let (t) be x l / r the reference recording which is (t)=s(t)*g l / r (t).

[0015] The Fourier transform (FT) of each recording is performed, and Y l / r (f;θ,Φ) and X l / r(f) is produced, where f is the frequency. The spectral response of the reference recording is normalized to some extent, for example, Using spectral splitting with TIFF0007936908000001.tif12170, the spectral response of each binaural recording is excluded, where λ is a regularization parameter to avoid computational noise, which may be frequency-dependent. The resulting transfer function H l / r (f;θ,Φ) is then transformed into the time domain and truncated to a certain length that typically covers the acoustic effects of the ears, head, and torso.

[0016] Post-processing: Diffuse field equalization

[0017] Diffuse field equalization attempts to remove all commonalities within a set of recordings. This involves normalizing the measurements with respect to the mean across all directions, with some level of frequency-dependent / independent regularization. Such a mean could be the mean of the magnitude response across all incident directions, or the mean of the magnitude squared response across all directions (power mean). One purpose of diffuse field equalization is to provide sound quality consistency across HR filters in an HR filter dataset. Another purpose is to compensate for any general response of windowing, or any other unwanted effects from common post-processing across all directions.

[0018] Post-processing: Low-frequency compensation

[0019] Due to the limited bandwidth of loudspeakers and the low-frequency limitations of anechoic chambers, frequencies below approximately 200 Hz cannot be reliably measured. However, theoretically, if the size of the human head is much smaller than the wavelength, the low-frequency response (<200 Hz) should be close to 1 (unity), or the low-frequency gain should be approximately 0 dB. Therefore, low-frequency compensation or correction is often applied to obtain a proper low-frequency response. For example, a low-frequency model may be employed to extend the flat frequency response and linear phase response below 400 Hz. For example, some numerically simulated data may be extrapolated to the low-frequency content of measured data. Clearly, the purpose of low-frequency compensation is to ensure natural bass in binauralized audio.

[0020] Estimated HR filters are often provided as finite impulse response (FIR) filters. Currently, HR filters are typically used directly in their original form by binaural audio renderers, or pairs of HR filters may be converted to interaural transfer functions (ITF) or modified ITFs to prevent abrupt spectral peaks. Alternatively, HR filters can be described by parametric representations. Such parameterized HR filters readily integrate with parametric multichannel audio coders, such as Moving Picture Expert Group (MPEG) surround and Spatial Audio Object Coding (SAOC).

[0021] The performance of a binaural audio renderer is subjectively evaluated through listening tests, where the judgment is typically an overall assessment of perceived spatial quality along with perceived sound quality. "Authentic" reproduction occurs when the binauralized audio in the listener's two ear canals corresponds very well to what the same listener would have at the point where the sound is picked up by the listener's ears. To achieve such "authentic" reproduction, an "authentic" HR filter is required. [Overview of the Initiative]

[0022] Considerable effort has been expended in setting up HR filter datasets using mannequins or head and torso simulators, such as KEMAR (Knowles Electronics Mannequin for Acoustics Research), or measurements on human subjects. Currently, several publicly available datasets exist, and it is common for a set of HR filters selected from these publicly available datasets to be used directly in binaural audio renderers. However, variability is an inherent part of the HR filter measurement process, and noise or measurement errors are unavoidable. Such measurement errors have a strong negative impact on the spatial perception of binauralized audio when used directly in binaural audio rendering. Examples of such errors include mismatch errors and errors caused by non-HR reflections.

[0023] Mismatch error

[0024] In HR filter measurements, especially for human subjects, a special chair is used. This special chair is typically designed with a headrest and backrest to provide a reference position for the subject's head relative to the speaker, aiming to minimize head movement during measurement. However, slight tilts in the subject's head or the chair's vertical axis of rotation often occur, causing positional mismatch between the speaker and the head. Such mismatches result in a discontinuity in the time to arrival (TOA) of the signal at the eardrum, or in other words, a frequency-independent time delay in the HR filter, known as the onset delay. This discontinuity in onset delay implies a discontinuity in the ITD. For example, the ITD of an HR filter at an azimuth angle of 0 degrees is considered to be 0. However, when mismatch occurs, the ITD deviates from 0. In audio scenes where the source moves along a vertical line in front of the listener, even small deviations of around ±1 sample can be perceived as instability (left / right wobble) in renderers using HR filters with this mismatch error. Furthermore, phase discontinuities in adjacent frames caused by onset delay discontinuities can also be perceived.

[0025] Rendering spatial audio sources that lead to a convincing spatial perception of sound in arbitrary locations in space requires pairs of HR filters at corresponding locations, and therefore a set of HR filters at finely sampled locations on a sphere. The spatial resolution of the HR filter set used in the renderer determines the spatial resolution of the rendered sound source. Using a coarsely sampled HR filter set on a 2D sphere will result in virtual reality (VR), augmented reality (AR), mixed reality (MR), and / or extended reality (XR) users typically reporting spatial discontinuities in moving sounds. Such spatial discontinuities lead to audio-video synchronization errors that significantly reduce the sense of immersion.

[0026] Obtaining an HR filter set on a denser grid on a sphere may solve the problem. However, estimating the HR filter set from input / output measurements on a fine grid that satisfies the minimum audible angle (MAA) requirement can be extremely time-consuming and tedious for both the subject and the experimenter. To improve rendering without increasing the measurement resolution, angle interpolation techniques may be used.

[0027] The nearest neighbor method is one of the techniques typically used by binaural audio renderers for HR filter angle interpolation. This method assumes that the HR filter at each sampled location affects only an area up to a certain finite distance. In such a method, the HR filter at an unsampled location is approximated as a weighted average of the HR filters at sampled locations within a certain cutoff distance from the unsampled location or from a given number of nearest points on a linear two-dimensional (2D) grid. This method is efficient for inferring spatial relational information about missing HR filters, assuming a sparsely sampled HR filter dataset. However, this method is sensitive to discontinuities in onset delay, which can lead to object locations in a broad sense.

[0028] non-HR reflex

[0029] Non-HR reflections are another source of error in HR filter measurements. As explained above, HR filter measurements are generally performed in an anechoic chamber. Furthermore, the mechanical setup is always carefully designed to have the least influence on the incident acoustic wave, for example, the sides of the loudspeaker are wrapped in acoustic absorbers, the support structure of the chair is covered with acoustic absorbers, etc. However, non-HR reflections can still occur and be captured in the recording. Such non-HR impulse responses appear in the resulting HR filter, reducing its authenticity and potentially disabling ILD cues in certain frequency bands, creating a perceptible "auxiliary" source somewhere other than the desired location.

[0030] As explained above, the original HR filter set available for use in binaural rendering is generally FIR filters. These FIR filters are typically several milliseconds in length, where the time span of each filter can be divided into three continuous time domains: a preactive domain, an active domain, and a postactive domain. In the preactive and postactive domains, the filter taps are zero or very close to zero due to estimated noise and contribute little to binauralization. The active domain contains the filter taps that represent the actual binauralization. These filter taps are strong at the beginning of the active domain but taper off, decreasing to near zero towards the end of the domain.

[0031] The filters used in binaural audio rendering can be the original filter or extracted from the original filter across different sub-regions of the filter's total time span. Ideally, extracting only the active regions of the filters and estimating the ITD between the active regions of the left and right filters would be beneficial, as it saves memory and makes the implementation of the filtering operation more efficient. The filter set extracted across the active regions is called the zero-time delay HR filter set and its data representation includes the ITD between the left and right HR filters.

[0032] Currently, several challenges exist. For example, obtaining a good estimate of what constitutes the active region of each filter is not trivial, especially when the filters include pre-ringing effects arising from the filter estimation.

[0033] If errors exist in either the HR filter measurement process or the subregion extraction process, it is necessary to detect and correct both types of errors in the HR filter beforehand in order to guarantee the performance of the binaural audio renderer, before using the filter in binaural audio rendering.

[0034] Therefore, in one embodiment, the final corrected head-relationship (HR) filtered dataset H fc A method is provided for generating the first corrected HR filter dataset H'. I This includes obtaining the first corrected HR filter dataset H'. I Obtaining the initial HR filter dataset H I To obtain the initial HR filter dataset H I Extracted from, the extracted HR filter dataset H X This includes obtaining the first corrected HR filter dataset H'. I Obtaining the extracted HR filter dataset H XModel M X To obtain and Model M X Using the modeled HR filter dataset H M This further includes generating the first corrected HR filter dataset H'. I Obtaining the modeled HR filter dataset H M and the extracted HR filter dataset H X Based on this, the initial HR filter dataset H I Selecting one or more HR filters contained within for correction, and correcting the selected one or more HR filters, results in the first corrected HR filter dataset H' I This further includes generating the final corrected HR filter dataset H. fc However, the first corrected HR filter dataset H' I Either this method is used for the first corrected HR filter dataset H' I Using the final corrected HR filter set H fc This further includes generating , or any of the following.

[0035] In another embodiment, a computer program is provided which, when executed by a processing circuit, includes instructions that cause the processing circuit to perform the method described above.

[0036] In another embodiment, the final corrected head-relationship (HR) filtered dataset H fc An apparatus for generating the first corrected HR filter dataset H' is provided. This apparatus generates the first corrected HR filter dataset H' I It is configured to obtain the first corrected HR filter dataset H'. I Obtaining the initial HR filter dataset H I To obtain the initial HR filter dataset H I Extracted from, the extracted HR filter dataset H X This includes obtaining the first corrected HR filter dataset H'. IObtaining the extracted HR filter dataset H X Model M X To obtain and Model M X Using the modeled HR filter dataset H M This further includes generating the first corrected HR filter dataset H'. I Obtaining the modeled HR filter dataset H M and the extracted HR filter dataset H X Based on this, the initial HR filter dataset H I Selecting one or more HR filters contained within for correction, and correcting the selected one or more HR filters, results in the first corrected HR filter dataset H' I This further includes generating the final corrected HR filter dataset H. fc However, the first corrected HR filter dataset H' I Either this device is, or this device is, the first corrected HR filter dataset H' I Using the final corrected HR filter set H fc It is either further configured to generate or

[0037] In another embodiment, a device is provided. This device comprises a memory and a processing circuit coupled to the memory, and is configured to carry out the method described above.

[0038] Embodiments of this disclosure offer at least the following advantages:

[0039] (1) In an HR filter set, smooth out additive stochastic noise and sporadic errors such as sporadic reflections that are not related to HR filtering.

[0040] (2) To improve the HR filter set through an iterative modeling process followed by an error detection-classification-correction process.

[0041] The accompanying drawings incorporated herein and forming part of the specification illustrate various embodiments. [Brief explanation of the drawing]

[0042] [Figure 1] This diagram shows sound waves propagating from the direction of arrival (DOA), specified by an elevation-azimuth pair in a spherical coordinate system, towards the listener. [Figure 2] This figure shows an example of ITD and spectral cue of sound waves propagating towards the listener. [Figure 3] This diagram shows a simplified setup for HR filter binaural recording. [Figure 4] This figure shows an example of a sampling grid on a 2D sphere. [Figure 5] This figure shows a process according to one embodiment. [Figure 6] This figure shows a process according to one embodiment. [Figure 7A] This is a diagram showing a system in several embodiments. [Figure 7B] This is a diagram showing a system in several embodiments. [Figure 8] This figure shows a process according to one embodiment. [Figure 9] This figure shows an apparatus according to one embodiment. [Modes for carrying out the invention]

[0043] Each left and right HR filter in an HR filter set sampled across a set of spherical angles is considered to be the sum of a true HR filter and a measured / extracted error filter. This applies whether the HR filter set uses the measured FIR HR filter set directly or filters from an HR filter set extracted across subregions of filters with different time spans, where the ITD values ​​between the left and right filters are.

[0044] To detect and correct errors in the acquired HR filter set, a model of the HR filter set on a unit sphere is evaluated that forces relatively smooth transitions between spatially close HR filters. By minimizing modeling errors with respect to the filter set, the model may filter out or remove several types of measurement / extraction errors in the filter. Examples of error types that may be filtered out or removed include (1) errors in the form of sporadic reflections in the HR filter that are not part of the HR filtering process, and (2) errors that are additive stochastic noise, particularly prominent at higher frequencies.

[0045] Once the model is acquired, individual HR filters with relatively large modeling errors are identified, and the type of error is classified. Errors belonging to several error classes can be corrected. When an error is correctable, it is corrected for those particular HR filters, and the set of HR filters is updated with the corrected filters. Iterating this modeling and error detection-classification-correction procedure several times can produce a much improved set of HR filters. As mentioned above, one particularly important class of correctable errors is the onset delay error class. The onset delay error class arises from the measurement and / or extraction process when filters are extracted from a sub-region of the filter's time span.

[0046] Embodiments of the present disclosure are applicable to all types of HR filter sets (or filters) that include an original HR filter set and HR filters extracted from subregions of the filter with different time spans.

[0047] Data variables and their notation

[0048] A typical data structure is shown as a data sequence and a list of other data structures. In embodiments of this disclosure, the basic HR filter dataset H is a dataset containing M HR filters sampled at elevation and azimuth angles {(θ[m],Φ[m]):m=1,...,M}, where θ and Φ are elevation and azimuth angles, respectively, and m is the index.

[0049] The dataset H is the data list H = {θ, Φ, H}. l ,H r It can be provided in the form of}.

[0050] θ={θ[m]:m=1,...,M} represents the sequence of elevation angles.

[0051] φ={φ[m]:m=1,...,M} represents the sequence of azimuth angles.

[0052] H l ={h l [m]:m=1,...,M} represents a set of left HR filters, where h l [m]=[h l [1;m],...,h l [N l ;m]] is of length N l This is an FIR filter.

[0053] H r ={h r [m]:m=1,...,M} represents a set of right HR filters, where h r [m]=[h r [1;m],...,h r [N r ;m]] is of length N r This is an FIR filter.

[0054] The length of the left HR filter and the length of the right HR filter are different or the same (i.e., N l =N r ) is possible.

[0055] In some embodiments, H may further include a data sequence of onset delays that indicate the start of the active region of the filter. For example, the extended HR filter dataset is H={θ,Φ,H l ,H r ,τ l ,τ r}, where τ l ={τ l [m]:m=1,...,M} denotes the sequence of onset delays of the left HR filter, and τ r ={τ r [m]:m=1,...,M} denotes the sequence of onset delays of the right HR filter.

[0056] H may also include a data sequence of ITDs derived from the onset delays of the left and right HR filters, that is, H={θ,Φ,H l ,H r ,τ l ,τ r ,τ ITD}, where τ ITD ={τ ITD [m]:m=1,...,M} denotes the sequence of ITDs.

[0057] In some embodiments, H may include the data sequence of ITDs derived from the onset delays of the left and right HR filters, but may not include the sequence of onset delays of the left HR filter and the sequence of onset delays of the right HR filter, that is, H={θ,Φ,H l ,H r ,τ ITD}.

[0058] In some embodiments of the present disclosure, at least four different HR filter datasets may be used, namely, the original dataset H0, the initial HR filter dataset H I , the extracted dataset H X , and the model-generated dataset H M .

[0059] The original dataset H0 is, It may include TIFF0007936908000002.tif6170. However, in some embodiments, the dataset is It may also contain TIFF0007936908000003.tif6170. In other words, H0 is It can be represented as TIFF0007936908000004.tif6170. Also, the original dataset H0 is the ITD data sequence τ ITD It may or may not include it.

[0060] Initial HR filter dataset H I This could be a set of filters that are or will be iteratively error-corrected. Initial HR filter dataset H I From H0 It may be initialized by TIFF0007936908000005.tif6170, for example, The filename is TIFF0007936908000006.tif6170. This is the initial HR filtered dataset H. I Also, the length of the left and right filters to be extracted. TIFF0007936908000007.tif6170 and the onset delay sequences for the left and right filters to be used in extraction. It can be initialized by specifying the extraction method as TIFF0007936908000008.tif6170. TIFF0007936908000009.tif6170 can be initialized as follows:

[0061] (1) When available, the onset delay sequence obtained from H0,

[0062] (2) The left and right onset delay sequences obtained by estimating the onset of each filter in TIFF0007936908000010.tif6170, and / or

[0063] (3) Single sample onset delay sequence TIFF0007936908000011.tif7170 specifies the desired left onset delay to be used for all left HR filters, TIFF0007936908000012.tif5170 specifies the desired right onset delay to be used for all right HR filters. With this option, filters may be extracted with fixed left and right onset delays. Furthermore, when the original HR filter set is to be extracted, the initial HR filter data set H I Filter extraction specification for is TIFF0007936908000013.tif6170.

[0064] As described above, H I is May be represented as TIFF0007936908000014.tif7170. During the iterative error correction process, TIFF0007936908000015.tif6170 may be updated as needed. Also, as described above, TIFF0007936908000016.tif7170 may be used in the extraction process.

[0065] In some embodiments, the extracted HR filter data set H X is May be represented as TIFF0007936908000017.tif6170, where TIFF0007936908000018.tif6170 is Left and right sequences of the filters (filter set) extracted from TIFF0007936908000020.tif6170 according to TIFF0007936908000019.tif6170. TIFF0007936908000020.tif6170.

[0066] For example, TIFF0007936908000021.tif6170 and The file is TIFF0007936908000022.tif8170, and here, TIFF0007936908000023.tif6170 is, This is the extracted FIR filter for TIFF0007936908000024.tif6170. The filename is TIFF0007936908000025.tif6170.

[0067] TIFF0007936908000026.tif5170 and The file is TIFF0007936908000027.tif6170, and here, This is the extracted FIR filter for TIFF0007936908000028.tif5170. The filename is TIFF0007936908000029.tif5170.

[0068] In iterative model-based error correction, H X This is Model M X It can be modeled as follows: Model M X This is the extracted HR filter dataset H X This could be a function that models the spatial variation of the filter inside. Model M X A detailed explanation is provided below. Model-generated dataset H M H X Model M approximates X This is a filter set generated by H. M teeth, It can be represented as TIFF0007936908000030.tif6170. H X The filter inside and H M The modeling error between the filters can be used to determine which filters should be classified and which filters should undergo error correction.

[0069] When the iterative error correction process stops, the process output is (1)(H fc(2) Final corrected HR filter dataset (H fc The extracted HR filter dataset H' was extracted from ). X The generated model M' X (3) Model M' in (2) at the elevation angle θ and azimuth angle Φ specified in output specification O. X A new modeled HR filter dataset H' generated from M As one of these, it may be delivered according to output specification O.

[0070] Overview of iterative modeling-based error correction processes

[0071] Figure 5 shows an iterative modeling-based error correction method 500 for improving an extracted set of HR filters, according to several embodiments.

[0072] The inputs to method 500 can be the original HR filter dataset H0, the extraction specification X, and the output specification O.

[0073] The original HR filter dataset H0 can be obtained by loading an existing HR filter dataset from a file into H0.

[0074] Extraction specification X is H I Used to initialize, and later H I From H X This should be used for an iterative loop to extract This can be obtained by specifying the desired value for TIFF0007936908000031.tif7170. As explained above, TIFF0007936908000032.tif6170 specifies the desired length of the left and right extracted filters. This defines the time instance from which TIFF0007936908000033.tif6170 is extracted. This defines the time instance from which TIFF0007936908000034.tif5170 is extracted.

[0075] When the desired extracted HR filter set is the original HR filter set, or an HR filter set in which the left and right filters are all extracted with the same onset delay, All elements in TIFF0007936908000035.tif6170 have the same value, i.e., They can be set to TIFF0007936908000036.tif7170, or those elements for all M elevation and azimuth angles TIFF0007936908000037.tif7170 can be specified as a sequence of length 1.

[0076] When the desired extracted HR filter is a zero-time onset delay HR filter set, The filename is TIFF0007936908000038.tif7170, where each element corresponds to the onset delay of each HR filter. Some datasets (for example, the CIPIC dataset) are: This provides onset delay information that can be used directly as TIFF0007936908000039.tif6170. However, the majority of the dataset does not provide such information. When such information is not provided in the original dataset, the onset delay needs to be estimated for each HR filter. Many different methods exist for performing this onset estimation. One or more examples of onset estimation algorithms are described in PCT / EP2020 / 079042.

[0077] Output specification O is the desired output dataset, TIFF0007936908000040.tif4170 shows the type, and the desired angle {θ} if necessary. D ,Φ D This can be obtained by specifying the sequence}. The output dataset TIFF0007936908000041.tif5170 is the final corrected HR filtered dataset H fc Is this the case, or is it the final corrected HR filter dataset H fc Extracted from, the extracted HR filter dataset H' X Model M' X Is it the Model M'? X A new modeled HR filter dataset H' generated from M It can be shown whether this is the case. {θ D ,Φ D} can be determined to obtain better rendering quality using the model-generated HR filter dataset, either by taking it directly from H0 or by some other means, for example, based on a more densely sampled spherical grid, and potentially also based on model performance.

[0078] As described above, in embodiments of this disclosure, the output is an improved HR filter dataset (i.e., the final corrected HR filter dataset H fc ) and here the HR filter of the improved HR filter dataset is H I It can be stored in the same format, or represented by an improved HR filter dataset or a model of a model-generated HR filter dataset.

[0079] After obtaining the input as described above, an iterative modeling-based error correction method 500 may be performed. Method 500 may include an initialization process 502, an iterative loop process 504, and an output process 506.

[0080] In some embodiments, the iterative loop process 504 may include three processes: a modeling process 512, a model error detection and classification process 514, and an HR filter dataset error correction process 516.

[0081] Method 500 can be operated offline or within a binaural audio renderer with respect to loading an HR filter dataset into that renderer.

[0082] Details of the Iterative Model-Based Error Correction Method 500

[0083] 1. Initialization process

[0084] In initialization process 502, some or all of the data variables required in the iterative loop are initialized. The data variables required in the iterative loop are (1) the initial HR filter dataset H0 obtained from the original HR filter dataset H0 and the extraction specification X. I (2) Error threshold T ε (3) Repeat condition C I , and (4) List L of classifiers Classifiers It may include one or more of the following.

[0085] 1.1 Initial HR Filtered Dataset H I

[0086] In some embodiments, The file is TIFF0007936908000042.tif7170, and here, TIFF0007936908000043.tif6170 is initialized with the corresponding data structure in the original HR filter dataset H0, TIFF0007936908000044.tif6170 is initialized with the corresponding data structure in extraction specification X.

[0087] 1.2 Error threshold T ε

[0088] T εThis can correspond to a threshold for the modeling error of an HR filter. The threshold can be used to select an HR filter to classify. For example, if the modeling error associated with an HR filter is greater than the threshold, that HR filter may be selected for classification. In one example, the threshold is 0.10.

[0089] 1.3 Iteration Condition C I

[0090] C I This can correspond to a logical expression that controls an iterative loop. The iterative loop can continue until this logical expression becomes false. The logical expression can be constructed such that it becomes false when there are no more HR filters to correct, or when the number of iterations reaches a specified maximum value.

[0091] 1.4 List of Classifiers L Classifiers

[0092] L Classifiers This may be a list of classifiers used to handle the classification of modeling errors and the final HR filter correction. Each item in the list may include one or more of the following: (1) a classification ID, (2) a classification method for determining whether the modeling error belongs to a particular class, and (3) flags indicating whether the classified error can be corrected.

[0093] If the classified error can be corrected, the corresponding item may also include a correction method for correcting the classified error, provided a class-specific correction data structure is provided. This class-specific correction data structure is for the classified filter It may be included in the items within TIFF0007936908000045.tif6170.

[0094] 2. Iterative Loop Process

[0095] In the iterative loop process 504, the following three subprocesses, namely (1) the modeling subprocess 512, (2) the model error detection and classification subprocess 514, and (3) the HR filter set error correction subprocess 516, are performed according to the logical expression C I This can be repeated until the result is false.

[0096] 2.1 Modeling

[0097] As shown in Figure 6, the modeling process 512 may include three steps, namely steps s602, s604, and s606.

[0098] Step s602 is performed in the first iteration of the loop, with the initial HR filter dataset H I From this, the extracted HR filter dataset H X This includes obtaining the extracted HR filter dataset H. Step 604 is to obtain the extracted HR filter dataset H. X Model M X This includes obtaining the Model M. Step s606 is X HR filter dataset H generated from M This includes obtaining it.

[0099] 2.1.1 Step s602 - Extracted HR filter dataset H X To obtain

[0100] Step s602 is to perform the initial HR filter dataset H I From this, the extracted HR filter dataset H X This includes obtaining the following. In some embodiments, The file is TIFF0007936908000046.tif6170, where θ0 and Φ0 are H I It is obtained as the corresponding data structure in, TIFF0007936908000047.tif6170 is H I in According to TIFF0007936908000048.tif6170, H I in Extracted from TIFF0007936908000049.tif6170, here, TIFF0007936908000050.tif6170 and The file is TIFF0007936908000051.tif8170, and here, TIFF0007936908000052.tif6170 is, This is the extracted FIR filter for TIFF0007936908000053.tif6170.

[0101] Similarly, in some embodiments, TIFF0007936908000054.tif5170 and The file is TIFF0007936908000055.tif6170, and here, TIFF0007936908000056.tif5170 is, This is the extracted FIR filter for TIFF0007936908000057.tif5170.

[0102] 2.1.2 Step s604 - Model M X To obtain

[0103] Step s604 is to extract the HR filter dataset H X Model M X This includes obtaining the following. In some embodiments, TIFF0007936908000058.tif6170 is modeled separately. For the sake of simplicity of notation in this disclosure, subscripts and superscripts are omitted where not particularly necessary.

[0104] The spatial variation of the filter in HR filter set H is due to the elevation angle and azimuth angle. It can be modeled as TIFF0007936908000059.tif5170. In general form, the model It can be represented by TIFF0007936908000060.tif6170, where f is A containing all model parameters and all basis functions containing TIFF0007936908000061.tif4170 can be a linear or nonlinear function. The basis functions can be learnable or predefined. As an example, in a linear model with P basis functions, this function is It can be explicitly represented by TIFF0007936908000062.tif14170, where A=[α1,...,α P ] is the model parameter set, where α p =[α1,...,α N ] T The p-th This is the model parameter vector for TIFF0007936908000063.tif6170. [·] T The symbol indicates the transpose operator. TIFF0007936908000064.tif7170 is a sequence of basis functions.

[0105] Note that TIFF0007936908000065.tif5170 uses θ and φ here instead of spatial variables to distinguish them from fixed-space sampling points. Whether a linear or nonlinear model is used, TIFF0007936908000066.tif5170 is a regularization term, i.e. A set of A that minimizes the selected loss function L that may include TIFF0007936908000067.tif9170, where, Assuming TIFF0007936908000068.tif6170 is used, this is an approximation of the HR filter at the sampled angle (θ[m], Φ[m]).

[0106] One example of such a loss function is the squared error loss, i.e. The filename is TIFF0007936908000069.tif13170.

[0107] In linear models, TIFF0007936908000070.tif5170 can be obtained through linear least squares estimation. In nonlinear models, TIFF0007936908000071.tif5170 can be estimated through an iterative gradient-based method. One or more examples of modeling algorithms are described in PCT / EP2020 / 079042.

[0108] The fundamental relationship between HR filters and DOA is usually considered continuous. However, as explained above, noise or measurement errors are unavoidable in the HR filter measurement process, and discontinuities appear in the measurement, for example. To avoid overfitting the model to noise or errors in the HR filter set, two strategies can be applied: 1) carefully designing the basis function so that its smoothness is maximized while being rich enough to capture the intrinsic spatial variation of the HR filter, and 2) applying some regularization to the loss function to enforce smoothness.

[0109] The model representation of the extracted HR filter is: It includes a modeling function f that determines the relationship with TIFF0007936908000072.tif5170. This can be shown by TIFF0007936908000073.tif7170. The HR filter vector in TIFF0007936908000074.tif7170 can be calculated.

[0110] H X If this is a zero-time delay HR filter dataset, This is the difference between TIFF0007936908000075.tif6170 and TIFF0007936908000075.tif6170. As a function of TIFF0007936908000076.tif6170, it can be modeled separately. Similarly, a model of the set of onset delays τ is It can be represented by TIFF0007936908000077.tif5170, where g can be a linear or nonlinear function with β containing all model parameters and B containing all basis functions. The basis functions can be learnable or predefined. As an example, in a linear model, this function is: Given by TIFF0007936908000078.tif15170, where β q This is the qth These are the model coefficients for TIFF0007936908000079.tif6170, where Q is the number of basis functions.

[0111] Similar to the HR filter, TIFF0007936908000080.tif6170 minimizes the selected loss function. It can be obtained as TIFF0007936908000081.tif6170. One example of such a loss function is the squared error loss, i.e. TIFF0007936908000082.tif13170, and here, TIFF0007936908000083.tif5170 is an approximation of the delay τ at the sampled angle (θ[m],Φ[m]), assuming β and B.

[0112] The model representation of delay is, TIFF0007936908000084.tif6170 and basis function B, It includes a modeling function g that describes the relationship with TIFF0007936908000085.tif6170. This can be shown by TIFF0007936908000086.tif6170.

[0113] M X The extracted HR filter dataset H shown by XThe model representation is, It may contain a model representation of TIFF0007936908000087.tif7170. When applicable, M X This is the onset delay of the left HR filter and the right HR filter. It may also include TIFF0007936908000088.tif7170. Within iterative loops, the onset-delayed model may not be necessary.

[0114] 2.1.3 Step s606 - Model M X Using the HR filter model dataset H M To generate

[0115] In step s606, TIFF0007936908000089.tif6170 is at a given location specified by a sequence of sampled angles {θ,Φ} It can be generated using TIFF0007936908000090.tif8170, where θ={θ[m]:m=1,...,M} and Φ={Φ[m]:m=1,...,M}. The sampled angles {θ,Φ} are, It can be obtained directly from TIFF0007936908000091.tif6170.

[0116] In some embodiments, TIFF0007936908000092.tif6170 and Each of TIFF0007936908000093.tif5170 will be processed by performing the following three steps. It can be generated from TIFF0007936908000094.tif7170.

[0117] For each m in {1,...,M}, 1. Obtain the spherical angles θ[m] and Φ[m] from the sampled angle sequences θ and φ. 2.M X inside Using TIFF0007936908000095.tif7170, the modeling function f and Using TIFF0007936908000096.tif5170 in (θ[m],Φ[m]) Calculate TIFF0007936908000097.tif6170. In the case of a linear model, This can be calculated using TIFF0007936908000098.tif7170. 3.M X inside Using TIFF0007936908000099.tif7170, the modeling function f and Using TIFF0007936908000100.tif5170 in (θ[m],Φ[m]) Calculate TIFF0007936908000101.tif5170. In the case of a linear model, Calculated by TIFF0007936908000102.tif6170.

[0118] Similarly, Each of TIFF0007936908000103.tif6170 can be modified by performing the following two steps: It can be generated from TIFF0007936908000104.tif7170. As an alternative or addition, TIFF0007936908000105.tif6170 will be created by performing the following two steps: It can be generated from TIFF0007936908000106.tif7170.

[0119] For each m in {1,...,M}, 1. Obtain the spherical angles θ[m] and Φ[m] from the sampled angle sequences θ and φ. 2.M X Internal delay set model M τ Using the model function g, Using TIFF0007936908000107.tif6170 and basis function B, delay τ at (θ[m],Φ[m]) M Calculate [m]. In the case of a linear model, τ M [m] is This can be calculated using TIFF0007936908000108.tif15170.

[0120] The generation of delayed datasets is not required in the iterative loop process. In other words, the generation of delayed datasets is an optional step in the iterative loop process.

[0121] 2.2 Model Error Detection and Classification Process

[0122] As shown in Figure 6, the model error detection and classification process 514 may include steps s608 and s610.

[0123] Step s608 is the index of the HR filter to be classified. This may include obtaining TIFF0007936908000109.tif7170. The HR filter to be classified is the error threshold T ε It has a modeling error exceeding [a certain value].

[0124] Step s610 is the classified HR filter This may include obtaining TIFF0007936908000110.tif6170.

[0125] 2.2.1 Step s608 - Index of HR filter to be classified Obtain TIFF0007936908000111.tif7170

[0126] The list of HR filter indices to be classified is (1) H M (2) Evaluate the modeling error of all HR filters inside, and the error threshold T ε Find a modeling error that exceeds (3) error threshold Tε The index of the HR filter having a modeling error exceeding It can be obtained by placing it in TIFF0007936908000112.tif7170. There may be separate index lists for the left HR filter and the right HR filter.

[0127] H M The evaluation of the modeling error of all HR filters inside is H X and H M These can be obtained by using the left and right HR filter sequences. For example, the left and right modeling errors can be calculated as follows:

[0128] (1) Left index list i is empty l =[] and right index list i r =[] initializes the array.

[0129] (2) For each m in {1,...,M}, (2-1) Evaluate TIFF0007936908000113.tif6170. These errors are normalized modeling errors, i.e., It can be evaluated as TIFF0007936908000114.tif13170, where, TIFF0007936908000115.tif5170 is a normalization function. As an example, in the L2 loss function, the left normalized TIFF0007936908000116.tif6170 is, The error is given by TIFF0007936908000117.tif11170 or some other form of modeling error. (2-2) TIFF0007936908000118.tif7170

[0130] (3) Set TIFF0007936908000119.tif7170.

[0131] example) When TIFF0007936908000120.tif6170, and Each of the errors associated with TIFF0007936908000121.tif7170 is T ε If it is greater than, TIFF0007936908000122.tif7170 may contain index values ​​of 1, 3, and 5.

[0132] 2.2.2 Step s610 - Classified HR filter Obtain TIFF0007936908000123.tif6170

[0133] Classified HR filters TIFF0007936908000124.tif6170 is a list of classification items, where each classification item may contain one or more of the following:

[0134] (1) Extracted HR filter dataset H X A filter ID containing information about the index of a specific HR filter included, and whether that specific HR filter is a left HR filter or a right HR filter.

[0135] (2) A classification ID that identifies the class of modeling error associated with a particular HR filter and whether the error of the class is correctable, and

[0136] (3) A class-specific correction data structure having the correction information required by the correction method for that error class.

[0137] In the example provided above, The filename is TIFF0007936908000125.tif6170, It is TIFF0007936908000126.tif6170. Also, Each of the errors associated with TIFF0007936908000127.tif7170 is T ε Larger than, TIFF0007936908000128.tif7170 contains index values ​​1, 3, and 5.

[0138] In the example above, If the error associated with TIFF0007936908000129.tif6170 belongs to correctable error class #E1, The error associated with TIFF0007936908000130.tif6170 belongs to the uncorrectable error class #E2. The error associated with TIFF0007936908000131.tif6170 belongs to correctable error class #E3, and the classified filter TIFF0007936908000132.tif6170 is item L1 classified And item L2 classified And item L3 classified L1 classified It includes an L1 filter ID that has information about the index value of [1] and an indication that the HR filter is a left HR filter, an L1 classification ID that identifies error class #E1 and indicates that #E1 is correctable, and a class-specific correction data structure.

[0139] Similarly, L2 classified This includes an L2 filter ID that has information about the index value of [3] and an indication that the HR filter is a left HR filter, and an L2 classification ID that identifies error class #E2 and indicates that #E2 is not correctable.

[0140] Similarly, L3 classified It includes an L3 filter ID that has information about the index value of [5] and an indication that the HR filter is a left HR filter, an L3 classification ID that identifies error class #E3 and indicates that #E3 is correctable, and a class-specific correction data structure.

[0141] For some classes of HR filter modeling errors, there may be classification methods that can determine with a high degree of reliability whether the modeling error belongs to one of those classes. For some of these classes, there may be methods for correcting the HR filter to fix the error. Correction methods for different classes of modeling errors can vary considerably, and therefore, in some embodiments, those correction methods are provided for those classes.

[0142] To manage these classes of modeling errors that can be classified, we have a list L of classifications. Classifications The following may be used. Each item in that list may contain one or more of the following:

[0143] (1) Classification ID,

[0144] (2) A classification method for determining whether the modeling error belongs to that class, and

[0145] (3) A flag indicating whether the classified error can be corrected.

[0146] If the extracted HR filter corresponding to the classification ID can be corrected, L Classifications The items inside are generally classified HR filters When a class-specific correction data structure is provided within TIFF0007936908000133.tif6170, it may also include a correction method for correcting the extracted HR filter.

[0147] One class of particular interest is the class of delay errors. A characteristic of delay errors is that the normalized modeling error of the extracted filter is relatively large, but there exists a shift τ0 that may be a fractional sample shift, and therefore this shift is H I When applied to the corresponding initial HR filter within, this essentially extracts the HR filter as H MThis will match the corresponding HR filter inside and significantly reduce the normalized modeling error. In the frequency domain, the delay error is characterized by the fact that the difference in magnitude between the extracted and modeled HR filters is small, and the difference in the unwrapped phases of the two HR filters should be approximately linear with the slope of -τ0.

[0148] A method used to classify this type of modeling error, when it finds a τ0 that satisfies such conditions, the method has a classified HR filter with classification information. Add classification items to TIFF0007936908000134.tif6170. The class-specific correction data structure for this class in the classification items must include shift τ0. Note that for the set of HR filters that go through the classification step, it has already been established that their normalized modeling errors exceed a certain threshold.

[0149] Several methods can be used to classify this type of modeling error. Two embodiments are provided, one in the time domain and the other in the frequency domain. In one embodiment, the following method operating in the time domain is used, namely looping over a predetermined sequence of shifts.

[0150] This method involves a predetermined sequence of shifts {τ} which may be fractional. k The loop is performed over =-τ1+(k-1)Δτ:k=1,...,K}, obtaining the shifted extracted HR filters and evaluating the normalized modeling error of each of those shifted HR filters. This method also has the shift τ with the smallest normalized modeling error. p This method can find that if the minimum normalized modeling error is sufficiently small (controlled by a threshold), then τ0 = τ p Set the classified HR filter which has classification information. TIFF0007936908000135.tif6170, add a classification item. Fractional shifting can introduce greater complexity, as fractional shifting requires resampling of the extracted HR filter.

[0151] Before presenting the embodiments in the frequency domain, the following notation is required. F X denotes the vector of the Fast Fourier Transform (FFT) of the extracted HR filter h for the first ordered ω values in the range 0≦ω≦π, and F x denotes the vector of the Fast Fourier Transform (FFT) of the extracted HR filter h M denotes the same vector for the corresponding modeled filter h M . abs(F X ) denotes the vector of the absolute values of the elements of the F X vector, and abs(F M ) denotes the vector of the absolute values of the elements of the F M vector.

[0152] In another embodiment, the following method operating in the frequency domain is used. When the normalized norm of the difference vector abs(F X )-abs(F M ) falls below a certain threshold, it is clear that large modeling errors are due to FFT angle or phase differences, and these angle differences are most often caused by delay errors. In this method, the first step is to identify HR filters in the list of HR filters to be classified that satisfy this condition. When such an HR filter is found, the unwrapped angle / phase difference, i.e., angle(F X )-angle(F M ), is modeled as -τ0ω (the phase difference caused by the delay error of τ0), and the normalized TIFF0007936908000136.tif5170, τ0 that minimizes it is obtained. As a final verification before classifying this modeling error as a delay error, it is confirmed that the modeling error for the corrected extracted filter is below a certain threshold.

[0153] 2.3 HR Filter Set Error Correction Process

[0154] As shown in Figure 6, in the first iteration of the loop, the HR filter set error correction process 516 corrects the first corrected HR filter dataset H' I This may include obtaining H'. In some embodiments, H' I The classified HR filter This can be obtained by going through the items in TIFF0007936908000137.tif6170 and iterating over each item that can be corrected, and by performing the following steps.

[0155] (1) Obtain the filter index and the filter ID which contains information about whether the corresponding HR filter is a left HR filter or a right HR filter.

[0156] (2) H to be corrected I To retrieve the HR filter within, use the retrieved filter ID.

[0157] (3) Obtain the classification ID, and put that classification ID into list L Classifications To use to obtain a correction function from,

[0158] (4) Obtain a class-specific correction data structure that has the correction information required by the correction function for that class.

[0159] (5) Using the acquired class-specific correction data structure, H I This involves applying the acquired correction function to the acquired HR filter. This may involve taking extra filter data samples from the original HR filter dataset H0, or extrapolating the HR filter data by, for example, adding zeros, and updating the corresponding onset delay if necessary.

[0160] If condition C1 is still true after step 516, then H I is H' I It is set to equal to and steps 512, 514, and 516 are repeated.

[0161] 3. Output Process

[0162] As shown in Figure 6, the output process 506 may output one of the following output datasets based on the output designation O. The output designation O is the desired output dataset. TIFF0007936908000138.tif5170 and / or the desired angle {θ D ,Φ D A sequence of} can be specified.

[0163] (1) Final corrected HR filter dataset H fc Here, H fc This is the corrected HR filtered dataset generated when step 516 was last performed (for example, if there is only one iteration of the loop, H fc =H' I (is),

[0164] (2) Final corrected HR filter dataset H fc Model M' generated from the extracted HR filter dataset extracted from X ,

[0165] (3) Model M' at elevation angle θ0 and azimuth angle Φ0 as specified in output specification O X A new modeled HR filter dataset H' generated from M .

[0166] Figure 7A shows an exemplary system 700 according to several embodiments. System 700 comprises a preprocessor 702 and an audio renderer 704. The preprocessor 702 and the audio renderer 704 may be contained within the same entity or within different entities. Also, different modules contained within the preprocessor 702 (e.g., 712 and 714) may be contained within the same entity or within different entities, and different modules contained within the audio renderer 704 (e.g., 716 and 718) may be contained within the same entity or within different entities.

[0167] The preprocessor 702 comprises an HR filter correction module 712 and memory 714. The HR filter correction module 712 may be configured to implement a modeling-based error correction method 500 (shown in Figure 5) to improve the set of HR filters. Thus, the input to the HR filter correction module 712 may be the input to the error correction method 500, i.e., the original HR filter dataset H0, extraction specification X, and output specification O. As described above, the output of the error correction method 500 is the final corrected HR filter dataset H fc Model M' represents X This is possible. Therefore, in one embodiment, the HR filter correction module 712 is model M' X The output may be an HR filter model representation 720. The output HR filter model representation 720 can be stored in memory 714.

[0168] The audio renderer 704 comprises an HR filter generator 716 and a binaural renderer 718. The HR filter generator 716 may read an HR filter model representation 720 from memory 714 and receive rendering metadata 722. Using the HR filter model representation 720 and rendering metadata 722, the HR filter generator 716 may generate and output a complete HR filter representation 724. The HR filter representation 724 may correspond to one or more HR filters generated using the HR filter model representation 720 at one or more given spatial angles indicated in the metadata 722. Using the received HR filter representation 724, the binaural renderer 718 may generate a binaural audio signal 726.

[0169] Figure 7B shows an exemplary system 750 according to several embodiments. System 750 comprises a preprocessor 752 and an audio renderer 754. The preprocessor 752 and the audio renderer 754 may be contained within the same entity or within different entities. Also, different modules contained within the preprocessor 752 (e.g., 762 and 764) may be contained within the same entity or within different entities, and different modules contained within the audio renderer 754 (e.g., 766 and 768) may be contained within the same entity or within different entities.

[0170] The preprocessor 752 comprises an HR filter correction module 762 and a memory 764. The HR filter correction module 762 may be configured to implement a modeling-based error correction method 500 (shown in Figure 5) for improving a set of HR filters. Thus, the input to the HR filter correction module 762 may be the input to the error correction method 500, i.e., the original HR filter dataset H0, extraction specification X, and output specification O. As described above, the output of the error correction method 500 (i.e., the output of the HR filter correction module 762) may be the final corrected HR filter dataset generated when step s516 was last performed. The final corrected HR filter dataset 770 may be stored in the memory 764.

[0171] The audio renderer 754 comprises an HR filter extractor 766 and a binaural renderer 768. The HR filter extractor 766 may read the final corrected HR filter dataset 770 from memory 764 and receive rendering metadata 772. Using the final corrected HR filter dataset 770 and rendering metadata 772, the HR filter extractor 766 may output a complete HR filter representation 774. The complete HR filter representation 774 may correspond to one or more HR filters obtained (e.g., extracted or interpolated) from the final corrected HR filter dataset 770 at one or more given spatial angles indicated in the metadata 772. Using the received complete HR filter representation 774, the binaural renderer 768 may generate a binaural audio signal 776.

[0172] Figure 8 shows the final corrected head-relationship (HR) filtered dataset H fc Process 800 is for producing the first corrected HR filter dataset (H' I Step s801 may begin with step s801, which includes obtaining the initial HR filtered dataset H. Step s801 includes steps s802-s812. Step s802 is the initial HR filtered dataset H Icomprises obtaining. Step s804 comprises an initial HR filter data set H I extracted therefrom, an extracted HR filter data set H X comprises obtaining. Step s806 comprises the extracted HR filter data set H X model M X comprises obtaining. Step s808 comprises the model M X using, a modeled HR filter data set H M comprises generating. Step s810 comprises selecting one or more HR filters included in the initial HR filter data set H I the initial HR filter data set for correction. Step s812 comprises generating a first corrected HR filter data set H' by correcting the selected one or more HR filters I comprises generating. If the condition is not satisfied (i.e., the logical expression C I is true), the process 800 may further comprise step s814 which comprises generating a final corrected HR filter data set H using the first corrected HR filter data set H' I using, a final corrected HR filter data set H fc comprises generating. However, if the condition is satisfied, the final corrected HR filter data set H fc is the first corrected HR filter data set H' I .

[0173] In some embodiments, the extracted HR filter data set H X obtaining comprises obtaining a filter extraction designation X, and based on X, H X comprises obtaining.

[0174] In some embodiments, X includes a filter length value N that identifies a filter length, and a set of delay values τ that identify a starting point for extraction.

[0175] In some embodiments, obtaining a model involves modeling the spatial variation of HR filters contained in the extracted HR filter dataset as a function of elevation and azimuth angles.

[0176] In some embodiments, the HR filters included in the modeled HR filter dataset are generated by calculating the HR filters using the model for each of several sampled angles.

[0177] In some embodiments, selecting one or more HR filters included in the initial HR filter dataset for correction is H X For each HR filter included, i) H M This includes: ii) calculating an error value for the HR filter based on the corresponding HR filter contained therein; and ii) determining whether or not to select the HR filter based at least in part on the calculated error value.

[0178] In some embodiments, H I teeth, Includes TIFF0007936908000139.tif6170 and H X teeth, Includes TIFF0007936908000140.tif6170. Also, H M teeth, Includes TIFF0007936908000141.tif6170. Includes TIFF0007936908000142.tif6170, where m is TIFF0007936908000143.tif6170HR is the index of the filter, where 1 ≤ m ≤ M, and M is TIFF0007936908000144.tif6170HR is a positive integer representing the number of filters, Includes TIFF0007936908000145.tif6170. TIFF0007936908000146.tif6170 The error values ​​for each HR filter are for the corresponding m. It can be calculated based on the difference between it and TIFF0007936908000147.tif6170. Includes TIFF0007936908000148.tif5170, where m is TIFF0007936908000149.tif5170HR is the index of the filter, where 1 ≤ m ≤ M, and M is TIFF0007936908000150.tif5170HR is a positive integer representing the number of filters, Includes TIFF0007936908000151.tif5170. TIFF0007936908000152.tif5170 The error values ​​for each HR filter are for the corresponding m It is calculated based on the difference between it and TIFF0007936908000153.tif5170.

[0179] In some embodiments, TIFF0007936908000154.tif6170 The error values ​​for each HR filter are for the corresponding m. Calculated based on TIFF0007936908000155.tif14170, TIFF0007936908000156.tif5170 The error values ​​for each HR filter are for the corresponding m. Calculated based on TIFF0007936908000157.tif11170.

[0180] In some embodiments, the method is I) TIFF0007936908000158.tif6170 Compare the error value for each HR filter with the threshold, and for each error value exceeding the threshold, (i) in relation to the error value, Identify the TIFF0007936908000159.tif6170HR filter, and (ii) TIFF0007936908000160.tif6170 Adding an HR filter identifier that also identifies the corresponding HR filter to the filter classification list, and adding an HR filter identifier to the filter classification list that identifies the filter to be classified, and II) TIFF0007936908000161.tif5170 Compare the error value for each HR filter with the threshold, and for each error value that exceeds the threshold, (i) related to the error value, Identify the TIFF0007936908000162.tif5170HR filter, and (ii) TIFF0007936908000163.tif5170 Adding an HR filter identifier that also identifies the corresponding HR filter to the filter classification list, further comprising adding an HR filter identifier to the filter classification list that identifies the filter to be classified by the filter classification list.

[0181] In some embodiments, the method is as follows: The process further includes finding one or more HR filter identifiers in the filter classification list that identify correctable HR filters contained in any of TIFF0007936908000164.tif6170. The step of selecting one or more filters contained in the initial HR filter dataset for correction is each (i) (ii) Selecting one or more HR filters for correction, which are contained in any of TIFF0007936908000165.tif6170 and identified by one or more HR filter identifiers found above.

[0182] In some embodiments, the method further includes determining whether each HR filter identified by an HR filter identifier included in the classification list is correctable, and determining whether an HR filter is correctable includes determining a modeling error class for the HR filter and determining whether the determined modeling error class is correctable.

[0183] In some embodiments, the method further includes, as a result of determining that an HR filter is correctable, adding to the correctable list i) an HR filter identifier that identifies the HR filter and ii) a correction data structure or a pointer to a correction data structure, the correction data structure containing information for use when correcting the HR filter.

[0184] In some embodiments, correcting one or more selected HR filters is done by each of the following: (i) (ii) Find one or more HR filters contained in any of TIFF0007936908000166.tif6170 and identified by an HR filter identifier contained in the correctable list, and correct the one or more HR filters found above using the corresponding correction data structure.

[0185] In some embodiments, the process involves a first corrected HR filter dataset H' I Using the final corrected HR filter set H fc To generate (step s814) the first corrected HR filter dataset H' I Using the final corrected HR filter set H fc Generating the first corrected HR filter dataset H' IFrom this, obtain a second extracted HR filter dataset, obtain a second model for the second extracted HR filter dataset, generate a second modeled HR filter dataset using the second model, and based on the second extracted HR filter dataset and the second modeled HR filter dataset, H' I A method comprising selecting one or more filters contained therein for correction, and generating a second corrected HR filter dataset by correcting the selected one or more HR filters, wherein the method uses the second corrected HR filter dataset to obtain a final corrected HR filter set H fc This further includes generating the final corrected HR filter dataset H fc However, this is the second corrected HR-filtered dataset.

[0186] In some embodiments, the process involves (1) the final corrected HR filter dataset H fc (2) Final corrected HR filtered dataset H fc The process further includes outputting one or more of the models generated from the extracted HR filter datasets extracted from (3)(2), and the new modeled HR filter datasets generated from the models generated in (2). For example, in some embodiments, the process outputs the final corrected HR filter dataset H fc , or the final corrected HR filter dataset H fc This includes outputting a model generated from the extracted HR filter dataset.

[0187] In some embodiments, H X This is the extracted HR filter Includes TIFF0007936908000167.tif6170, H M This is a modeled HR filter. Includes TIFF0007936908000168.tif6170. In some embodiments, H I Selecting one or more HR filters included for correction means that for each shift included in the set of shifts (e.g., τ1, τ2), the shifts (e.g., τ1, τ2) and the extracted HR filters Shifted extracted HR filter using TIFF0007936908000169.tif6170 This involves obtaining TIFF0007936908000170.tif7170 and calculating the errors associated with each of the shifted extracted HR filters. The selection may further involve identifying the errors that satisfy the criteria from the calculated errors and classifying the errors associated with the extracted HR filters and the modeled HR filters as classes of correctable delay errors based on the identified errors.

[0188] In some embodiments, H I Selecting one or more HR filters included for correction further involves finding the shift used to obtain the shifted extracted HR filter related to the minimum error, and using the shift found above as a correction parameter for the class of delay error.

[0189] In some embodiments, the extracted HR filter dataset H X This includes the extracted HR filters and the modeled HR filter dataset H M This includes the modeled HR filter and the initial HR filter dataset H ISelecting one or more HR filters included for correction (s810) includes, in the frequency domain, calculating a measure of the difference between the magnitude of the extracted HR filter and the magnitude of the modeled HR filter; comparing the calculated measure to a threshold to determine whether the calculated measure is less than or equal to the threshold; and classifying, at least based on the comparison, the errors associated with the extracted HR filter and the modeled HR filter as classes of correctable delay errors.

[0190] In some embodiments, the initial HR filter dataset H I Selecting one or more HR filters included for correction (s810) further includes: calculating the unwrapped angle and / or phase difference between the extracted HR filter and the modeled HR filter at the sample frequency; determining a linear model that models the calculated difference; determining a modeling error based on the calculated difference and value of the linear model at the sample frequency; comparing the modeling error with an error threshold; and using the parameters of the linear model as correction parameters for a class of delay error, at least based on the comparison between the modeling error and the error threshold.

[0191] In some embodiments, H X This is the extracted HR filter Includes TIFF0007936908000171.tif6170, H M This is a modeled HR filter. Includes TIFF0007936908000172.tif6170. In some embodiments, H ISelecting one or more HR filters included for correction involves calculating the difference between the phase of the extracted HR filter and the phase of the modeled HR filter, and for each shift included in the set of shifts (e.g., τ1, τ2), calculating the error between the calculated phase difference and a comparison value (e.g., -τ0ω) calculated based on the shift. The selection may further involve identifying the smallest error among the calculated errors and classifying, based on the smallest error, the errors associated with the extracted HR filter and the modeled HR filter as classes of correctable delay errors.

[0192] In some embodiments, H I Selecting one or more HR filters included for correction further involves finding the shift used to compute the comparison value that produces the minimum error, and using the shift found above as a correction parameter in a class-specific correction data structure for the class of delay error.

[0193] In some embodiments, the filter classification list is such that the one or more HR filter identifiers are Includes an indicator showing TIFF0007936908000173.tif6170.

[0194] In some embodiments, identifying an error that satisfies the conditions includes identifying the smallest error among the calculated errors.

[0195] Figure 9 is a block diagram of the apparatus 900 according to several embodiments for implementing the preprocessor 702 or 752 shown in Figures 7a and 7b. As described above, the preprocessor 702 or 752 may include an HR filter correction module 712 or 762 which can be configured to perform the HR filter correction described herein. As shown in Figure 9, the apparatus 900 includes a processing circuit (PC) 902 which may include one or more processors (P) 955 (e.g., a general-purpose microprocessor and / or one or more other processors such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), etc.), wherein the processors may be co-site in a single housing or in a single data center or geographically distributed (i.e., the apparatus 900 may be a distributed computing apparatus), and at least one network interface 948, each network interface 948 being a network interface The PC 902 may include at least one network interface 948 and one or more storage units (also known as “data storage systems”) 908 which may include one or more non-volatile storage devices and / or one or more volatile storage devices. In embodiments in which the PC 902 includes a programmable processor, a computer program product (CPP) 941 may be provided. The CPP 941 includes a computer-readable medium (CRM) 942 which stores a computer program (CP) 943 comprising computer-readable instructions (CRI) 944.CRM942 can be a non-temporary computer-readable medium, such as a magnetic medium (e.g., a hard disk), an optical medium, or a memory device (e.g., random access memory, flash memory). In some embodiments, the CRI944 of the computer program 943, when executed by PC902, is configured to cause the device 900 to perform the steps described herein (e.g., the steps described herein with reference to a flowchart). In other embodiments, the device 900 may be configured to perform the steps described herein without the need for code. That is, for example, PC902 may consist of simply one or more ASICs. Thus, features of the embodiments described herein can be implemented in hardware and / or software.

[0196] While various embodiments have been described herein, it should be understood that these embodiments are presented only as examples and not as limitations. Therefore, the breadth and scope of this disclosure should not be limited by any of the exemplary embodiments described above. Furthermore, unless otherwise indicated herein or explicitly rejected by the context, any combination of all possible variations of the elements described above is encompassed by this disclosure.

[0197] Furthermore, the process and message flow described above and shown in the diagrams are presented as a sequence of steps, but this is done for illustrative purposes only. Therefore, it is intended that some steps may be added, some steps may be omitted, the order of the steps may be rearranged, and some steps may be performed in parallel.

Claims

1. A method for audio rendering, wherein the method is Initial HR filtered dataset H Ｉ To obtain (s802), The aforementioned initial HR filter dataset H Ｉ Extracted from, the extracted HR filter dataset H Ｘ This involves obtaining the extracted HR filter dataset H Ｘ However, the extracted HR filter dataset H includes the extracted HR filter. Ｘ To obtain (s804), The extracted HR filter dataset H Ｘ Model M Ｘ To obtain (s806), the aforementioned model M Ｘ to generate a modeled HR filter dataset H Ｍ , wherein the modeled HR filter dataset H Ｍ includes a modeled HR filter, generating the modeled HR filter dataset H Ｍ (s808), and Based on the modeled HR filter dataset H M, the initial HR filter dataset H Ｉ Selecting one or more HR filters included therein for correction, and the selection is Obtaining a set of time shifts, For each time shift included in the set of time shifts, the shifted extracted HR filter is obtained using the time shift. Calculate the error associated with each of the shifted extracted HR filters, This includes, and the selection is based on the error, the initial HR filter dataset H Ｉ This involves selecting one or more HR filters included for correction, or selecting one or more HR filters (s810), By correcting the selected one or more HR filters, the first corrected HR filter dataset H' is obtained. Ｉ To generate (s812), The first corrected HR filter dataset H' Ｉ Based on this, the final corrected HR-filtered dataset H ｆｃ To generate the final corrected HR filter dataset H ｆｃ However, the first corrected HR filter dataset H' Ｉ That is, Methods that include...

2. The extracted HR filter dataset H Ｘ Obtaining (s804) means obtaining the filter extraction specification X, and based on X, the extracted HR filter dataset H Ｘ The method according to claim 1, which includes obtaining

3. The method according to claim 2, wherein X includes a set of filter length values ​​N that identify the filter length and a set of delay values ​​τ that identify the starting point for extraction.

4. The aforementioned Model M Ｘ Obtaining (s806) the extracted HR filter dataset H as a function of elevation angle and azimuth angle Ｘ The method according to any one of claims 1 to 3, comprising modeling the spatial variation of the HR filter contained therein.

5. The modeled HR filter dataset H Ｍ The HR filter contained therein, in each of the multiple sampled angles, the Model M Ｘ The method according to any one of claims 1 to 4, which is produced by calculating an HR filter using

6. The aforementioned initial HR filter dataset H Ｉ Selecting one or more of the aforementioned HR filters for correction (s810) The extracted HR filter dataset H Ｘ For each HR filter included therein, i) the modeled HR filter dataset H Ｍ ii) calculating an error value for the HR filter based on the corresponding HR filter contained therein, and ii) deciding whether or not to select the HR filter based at least in part on the calculated error value. The method according to any one of claims 1 to 5, including the method described in any one of claims 1 to 5.

7. The aforementioned initial HR filter dataset H Ｉ but, and The extracted HR filter dataset H Ｘ but, and The modeled HR filter dataset H Ｍ but, and The extracted left above This includes, where m is the extracted left The index of the HR filter is such that 1 ≤ m ≤ M, and M is the extracted left A positive integer representing the number of HR filters contained within, The model shown on the left above Includes, The extracted left above The error value for each HR filter contained therein is for the corresponding m It is calculated based on the difference between the two, The extracted on the right This includes, where m is the extracted right The index of the HR filter contained therein, where 1 ≤ m ≤ M, and M is the extracted on the right. A positive integer representing the number of HR filters contained within, The aforementioned model on the right Includes, The extracted on the right The error value for each HR filter contained therein is for the corresponding m It is calculated based on the difference between the two. The method according to claim 6.

8. The extracted left above The error values ​​for each HR filter included therein are Calculated based on the extracted right above The error values ​​for each HR filter included therein are Calculated based on, The method according to claim 7.

9. The aforementioned method, The extracted left above The error value for each HR filter included therein is compared with a threshold value, For each error value that exceeds the threshold, (i) in relation to the error value, the extracted left Identify the HR filter contained therein, (ii) the left Adding an HR filter identifier that also identifies the corresponding HR filter contained therein to the first filter classification list, wherein the first filter classification list identifies the filter to be classified, The extracted on the right The error value for each HR filter included therein is compared with a threshold value, For each error value that exceeds the threshold, (i) in relation to the error value, the extracted on the right Identify the HR filter contained therein, and (ii) the right Adding an HR filter identifier to a second filter classification list that also identifies the corresponding HR filter contained therein, wherein the second filter classification list identifies the filter to be classified, and further, the HR filter identifier added to the second filter classification list is the same as or different from the first filter classification list. The method according to claim 7 or 8, further comprising:

10. The aforementioned method, Each of them is the extracted left above or the extracted right Find one or more HR filter identifiers in the first filter classification list or the second filter classification list that identify a correctable HR filter included in any of the above. It further includes, The step (s810) of selecting one or more filters included in the initial HR filter dataset for correction, each of which (i) the left or the above right The method of claim 9, comprising (ii) selecting one or more HR filters for correction, which are included in any of the above and identified by the one or more HR filter identifiers found above.

11. The above method is used to obtain the first corrected HR filter dataset H' Ｉ Using the final corrected HR filter dataset H ｆｃ Further including generating, The first corrected HR filter dataset H' Ｉ Using the final corrected HR filter dataset H ｆｃ To generate The first corrected HR filter dataset H' Ｉ From this, obtain the second extracted HR filter dataset, Obtaining a second model from the second extracted HR filter dataset, Using the second model described above, a second modeled HR filter dataset is generated, The aforementioned H' Ｉ Selecting one or more filters included for correction, A second corrected HR filter dataset is generated by correcting one or more of the selected HR filters. Based on the second corrected HR filter dataset, the final corrected HR filter dataset H ｆｃ To generate the final corrected HR filter dataset H ｆｃ However, the second corrected HR filter dataset is, The method according to any one of claims 1 to 10, including the method described in any one of claims 1 to 10.

12. The method described above is (1) The final corrected HR filter dataset H ｆｃ (2) The final corrected HR filter dataset H ｆｃ Output one or more of the following: a model generated from the extracted HR filter dataset extracted from (3)(2), and a new modeled HR filter dataset generated from the model generated in (2). The method according to any one of claims 1 to 11, further comprising:

13. Final corrected head-relationship (HR) filtered dataset H ｆｃ An audio renderer for generating, wherein the audio renderer is Initial HR filtered dataset H Ｉ To obtain (s802), The aforementioned initial HR filter dataset H Ｉ Extracted from, the extracted HR filter dataset H Ｘ This involves obtaining the extracted HR filter dataset H Ｘ However, the extracted HR filter dataset H includes the extracted HR filter. Ｘ To obtain (s804), The extracted HR filter dataset H Ｘ Model M Ｘ To obtain (s806), The aforementioned Model M Ｘ Using the modeled HR filter dataset H Ｍ This involves generating the modeled HR filter dataset H Ｍ However, the modeled HR filter dataset H includes the modeled HR filter. Ｍ To generate (s808), Based on the modeled HR filter dataset H M, the initial HR filter dataset H Ｉ Selecting one or more HR filters included therein for correction, and the selection is Obtaining a set of time shifts, For each time shift included in the set of time shifts, the shifted extracted HR filter is obtained using the time shift. Calculate the error associated with each of the shifted extracted HR filters, This includes, and the selection is based on the error, the initial HR filter dataset H Ｉ This involves selecting one or more HR filters included for correction, or selecting one or more HR filters (s810), By correcting the selected one or more HR filters, the first corrected HR filter dataset H' is obtained. Ｉ To generate (s812), It is configured to do so, The audio renderer processes the first corrected HR filter dataset H' Ｉ Based on the above, the final corrected HR filter dataset H ｆｃ It is further configured to generate the final corrected HR filter dataset H ｆｃ However, the first corrected HR filter dataset H' Ｉ It is an audio renderer.

14. The audio renderer according to claim 13, further configured to perform the method described in any one of claims 2 to 12.

Images