Method for creating linear interpolated head-related transfer function

Abstract

Systems, devices, and methods are described for determining an adapted pair of "coupled" left / right ear head related transfer functions (HRTFs) from a pair of original left / right ear head related transfer functions (HRTFs), where the inter-ear delay of the coupled HRTFs is formed using an all-pass filter that provides the correct inter-ear delay at low frequencies. The all-pass filter is adapted to limit inter-ear phase differences at high frequencies. In addition, a low complexity process for rapidly generating a suitable all-pass filter is described.

Description

Cross Reference to Related Applications

[0001] This application claims priority to U.S. Provisional Application No. 63 / 455,539, filed March 29, 2023, U.S. Provisional Application No. 63 / 595,752, filed November 2, 2023, and U.S. Provisional Application No. 63 / 567,376, filed March 19, 2024, the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0002] The present disclosure relates to creating modified head-related transfer functions (HRTFs) from original HRTFs. BACKGROUND

[0003] Unless otherwise indicated herein, the methods described in this section are not prior art to the claims in this application and are not admitted to be prior art by virtue of their inclusion in this section.

[0004] Binaural audio signals include two audio channels intended to be played back to a listener through respective two (left and right) ears. Binaural playback can be achieved via loudspeakers placed near each ear or through headphones, including over-ear and in-ear headphones.

[0005] Binaural signals can be generated by processing a source audio signal with a pair of head-related transfer function (HRTF) filter responses. HRTF responses can be defined in a variety of ways, including in terms of time-domain impulse responses or in terms of frequency-domain responses. HRTF responses are typically grouped in pairs so as to provide a response for each ear transducer.

[0006] When used to process an audio signal, a pair of HRTF filters can be used to provide a listener with an approximation of the experience that would be produced (at each ear) when the audio signal is presented from a particular direction of arrival. Different pairs of HRTF filters will produce different illusions of sound source direction.

[0007] A pair of reference HRTF filters associated with a particular direction of arrival can be determined by measuring the acoustic transfer functions from a sound source (located at a distance in the same direction) to each ear of a listener. Alternatively, reference HRTF filters can be determined by other means, including numerical simulation or acoustic measurements on a mannequin.

[0008] A pair of modified HRTF filters can differ from a pair of acoustically measured HRTF filters, but still provide a listener with a desired sound impression from the same direction. In particular, the phase difference between the high frequency portions of the left and right modified HRTF filters can differ significantly from the phase difference between the high frequency portions of the left and right reference HRTF filters, but without a significant loss of experience perceived by the listener. This is possible because, in the high frequency range, interaural phase differences are essentially irrelevant to the perception of the listener.

[0009] The HRTF set function is a function that determines a left ear HRTF filter and a right ear HRTF filter given an arrival direction: (1) In Equation 1, HRTFsetis the HRTF set function An arrival direction is provided in the form of a 3D unit vector and the function returns a pair of left / right HRTF filters.

[0010] With these and other considerations in mind, the disclosure presented herein was made. SUMMARY

[0011] Techniques for processing audio signals are described. Various examples described herein provide systems, methods, and / or devices for creating and using modified HRTF filters with alternative high frequency phase responses.

[0012] According to some example embodiments, an audio processing method for a control system comprising one or more processors can involve obtaining, by the control system, a first set of head related transfer functions (HRTFs) and transforming, by the control system, the first set of HRTFs to a second set of HRTFs. In some example embodiments, the transforming can involve replacing delay components of the first set of HRTFs with all-pass filters in the second set of HRTFs. In some example embodiments, the transforming can involve adjusting a phase response of each of the all-pass filters in the second set of HRTFs such that: each interaural phase response is substantially linear within a frequency below an associated threshold frequency of the corresponding all-pass filter, and each phase response reduces an interaural phase difference within a frequency above the associated threshold frequency of the corresponding all-pass filter.

[0013] In some example embodiments, the method can involve outputting the second set of HRTFs. According to some example embodiments, outputting the second set of HRTFs can involve storing the second set of HRTFs, transmitting the second set of HRTFs to a device configured to process audio data, providing the second set of HRTFs for further processing, or a combination thereof.

[0014] According to some example embodiments, the method can involve defining, by the control system, a set of base filters based on the second set of HRTFs. A number of members in the set of base filters can be less than a number of members in the second set of HRTFs. In some example embodiments, the method can involve obtaining, by the control system, a bitstream of input audio data in an input audio format, and combining, by the control system, the input audio data with one or more base filters of the set of base filters to produce left audio data and right audio data.

[0015] In some example embodiments, the method can involve outputting, by the control system, the left audio data and the right audio data. According to some example embodiments, outputting the left audio data and the right audio data can involve storing the left audio data and the right audio data, transmitting the left audio data and the right audio data, providing, by the control system, the left audio data and the right audio data to a set of loudspeakers for playback, providing the left audio data and the right audio data for further processing, or a combination thereof.

[0016] According to some example embodiments, the transforming can further involve obtaining a left ear HRTF and a right ear HRTF from the first set of HRTFs, identifying a left ear non-delayed impulse response and a left ear delay from each of the left ear HRTFs, and identifying a right ear non-delayed impulse response and a right ear delay from each of the right ear HRTFs. In some example embodiments, the transforming can further involve producing a left ear allpass filter, each of the left ear allpass filters based at least in part on an instance of the left ear delay. According to some example embodiments, the transforming can further involve producing a right ear allpass filter, each of the right ear allpass filters based at least in part on an instance of the right ear delay. In some example embodiments, the transforming can further involve combining instances of the left ear non-delayed impulse response with corresponding instances of the left ear allpass filter and combining instances of the right ear non-delayed impulse response with corresponding instances of the right ear allpass filter to produce pairs of HRTFs of the second set of HRTFs.

[0017] In some example embodiments, the method can further involve producing a modified left ear delay value based on one or more of the extracted left ear delays and producing a modified right ear delay value based on one or more of the extracted right ear delays. The left ear allpass filter can be based on the modified left ear delay value, and the right ear allpass filter can be based on the modified right ear delay value.

[0018] According to some example embodiments, generating instances of the modified left ear delay values and right ear delay values can involve determining a difference between the extracted left ear delay and the extracted right ear delay. In some such example embodiments, generating instances of the modified left ear delay values and right ear delay values can involve determining a maximum expected difference between the extracted left ear delay and the extracted right ear delay. According to some example embodiments, the difference between the extracted left ear delay and the extracted right ear delay can be equal to a difference between corresponding modified left ear delay values and modified right ear delay values.

[0019] In some example embodiments, the modified left ear delay values and the modified right ear delay values can correspond to a smoothing function. According to some example embodiments, each pair of the modified left ear delay values and the modified right ear delay values can include a lower delay value and a higher delay value. In some examples, a delay variation of the lower delay value can be less than a delay variation of the higher delay value. In some example embodiments, the non-delayed impulse responses can be minimum phase filter responses.

[0020] According to some example embodiments, extracting each left ear non-delayed impulse response, each right ear non-delayed impulse response, each left ear delay, and each right ear delay from each of the left ear HRTF and the right ear HRTF can involve determining a frequency response of an original HRTF filter in the first HRTF set, determining an amplitude response of the original HRTF filter, and determining a minimum phase frequency response of a new non-delayed minimum phase filter. In some such example embodiments, extracting each left ear non-delayed impulse response, each right ear non-delayed impulse response, each left ear delay, and each right ear delay from each of the left ear HRTF and the right ear HRTF can involve determining a phase response of the original HRTF filter and a phase response of the new non-delayed minimum phase filter, and determining a delay associated with the original HRTF filter based at least in part on the phase response of the original HRTF filter and the phase response of the new non-delayed minimum phase filter. In some such example embodiments, determining the minimum phase frequency response can involve implementing a Hilbert transform involving the amplitude response of the original HRTF filter. According to some example embodiments, determining the delay associated with the original HRTF filter can also be based at least in part on a delay measurement frequency in a range of 300 Hz to 1600 Hz.

[0021] In some example embodiments, a number of members in the base filter set can be at least one order of magnitude less than a number of members in the second HRTF set. According to some example embodiments, the all-pass phase response can deviate from a linear ramp phase response and can smoothly approach zero phase at frequencies higher than the threshold frequency.

[0022] According to some example embodiments, the control system can correspond to at least a portion of a codec for immersive voice and audio services (IVAS).

[0023] According to some further embodiments, one or more non-transitory computer- readable media can store instructions that, when executed by one or more processors, cause the one or more processors to perform operations of any of the methods disclosed herein.

[0024] According to some additional example embodiments, an audio processor device can be configured to process input audio data. In some example embodiments, the audio processor device can include a computer unit and a receiver unit configured to receive the input audio data. According to some example embodiments, the computer unit can be configured to retrieve a first set of head-related transfer functions (HRTFs) and transform the first set of HRTFs to a second set of HRTFs. In some example embodiments, the transforming can involve replacing delay components of the first set of HRTFs with all-pass filters in the second set of HRTFs. In some example embodiments, the transforming can involve adjusting a phase response of each of the all-pass filters in the second set of HRTFs such that: each interaural phase response is substantially linear within a frequency lower than an associated threshold frequency of a corresponding all-pass filter, and each phase response reduces an interaural phase difference within a frequency higher than the associated threshold frequency of the corresponding all-pass filter.

[0025] In some example embodiments, the computer unit can be configured to output the second set of HRTFs. According to some example embodiments, outputting the second set of HRTFs can involve storing the second set of HRTFs, transmitting the second set of HRTFs to a device configured to process audio data, providing the second set of HRTFs for further processing, or a combination thereof.

[0026] According to some example embodiments, the computer unit can be further configured to define a set of basis filters based on the second set of HRTFs. A number of members in the set of basis filters can be less than a number of members in the second set of HRTFs. In some example embodiments, the computer unit can be further configured to obtain a bitstream of input audio data in an input audio format and combine the input audio data with one or more basis filters in the set of basis filters to produce left audio data and right audio data.

[0027] In some example embodiments, the computer unit can be further configured to output the left audio data and the right audio data. According to some example embodiments, outputting the left audio data and the right audio data can involve storing the left audio data and the right audio data, transmitting the left audio data and the right audio data, providing the left audio data and the right audio data by the control system to a set of loudspeakers for playback, providing the left audio data and the right audio data for further processing, or a combination thereof.

[0028] According to some example embodiments, the audio processor device can include a storage device configured to store the first HRTF, the second HRTF, the left audio data, the right audio data, the input audio data, or a combination thereof. In some such example embodiments, the storage device can include a random access memory, a read only memory, a non-transitory computer readable medium, or a combination thereof.

[0029] In some example embodiments, the audio processor device can correspond to at least a portion of a codec for immersive voice and audio services (IVAS).

[0030] Embodiments described herein can be generally described as technology, where the term “technology” can refer to system(s), device(s), method(s), computer readable instruction(s), module(s), component(s), hardware logic, and / or operation(s) as suggested by the context of application herein.

[0031] Features and benefits described in the Specification are provided without admitting that any of the corresponding features and benefits are essential, alone or in any combination, to the practice of the claimed subject matter. Accordingly, the summary is intended to provide a brief overview of some embodiments of the technology and their BRIEF DESCRIPTION OF DRAWINGS

[0032] Embodiments of the application will now be described, by way of example only, with reference to the accompanying drawings: Figure 1A is a block diagram illustrating an example of components of an apparatus capable of implementing various aspects of the present disclosure; Figure 1B illustrates a schematic block diagram of an example device architecture that can be used to implement various aspects of the present disclosure; Figure 1C illustrates a schematic block diagram of an example CPU that can be used to implement various aspects of the present disclosure, implemented in the device architecture of Figure 1B Figure 1D ​is a block diagram of an immersive voice and audio services (IVAS) encoder / decoder (“codec”) framework for encoding and decoding IVAS bitstreams according to one or more embodiments; Figure 1E is a diagram showing a listener head-centered Cartesian coordinate system; Figure 2 is a plot showing a left ear reference HRTF; Figure 3 is a plot showing a right ear reference HRTF; Figure 4 is a plot showing a right ear HRTF filter with delay removed; Figure 5 is a plot showing the phase response of two alternative filters; Figure 6 is a plot showing the phase response of two alternative filters; Figure 7 is a plot showing a left ear modified HRTF; Figure 8 is a plot showing a right ear modified HRTF; Figure 9 is a plot showing a left ear HRTF with added delay; Figure 10 is a plot showing a right ear modified HRTF; Figure 11 is a diagram showing a modified HRTF filter; Figure 12 is a diagram showing forming an all-pass impulse response with an associated delay; Figure 13 is a diagram showing converting a library of original HRTFs to a more compact HRTF basis set; Figure 14 is a diagram showing a compact HRTF basis set for efficiently computing HRTFs; Figure 15 is a diagram showing a compact HRTF basis set for processing scene-based audio signals; Figure 16 shows additional details of the HRTF transform block in Figures 13 to 15 according to some embodiments; Figure 17 shows additional details of the HRTF transform sub-block in Figure 16 according to some embodiments; Figure 18 , Figure 19 and Figure 20 shows that the HRTF transform block in Figure 17Examples of functions implemented by the delay processing block in the Figure 21 is a flowchart outlining one example of a method that can be performed by an apparatus or system such as the apparatus or system disclosed herein. DETAILED DESCRIPTION

[0033] The present disclosure relates to creating modified HRTFs from original HRTFs such that the modified HRTFs can be more efficiently approximated by linear mixing while preserving psychoacoustic properties of the original HRTFs. Described herein are techniques related to processing HRTF filters for producing modified HRTF filters based on linear interpolation suitable for use in a set of filters. In the following description, for the purposes of explanation, numerous examples and specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be apparent to one skilled in the art that the present disclosure can include some or all of these examples, alone or in combination with other features not expressly described, as defined by the claims, and can further include modifications and equivalents of the features and concepts described herein.

[0034] In the following description, various systems, devices, methods, processes and procedures are detailed. Although certain steps can be described in a certain order, this order is primarily for convenience and clarity. Certain steps can be performed more than once, can occur before or after other steps (even if those steps are otherwise described in another order), and can occur in parallel with other steps. Only when a first step must be completed before a second step can begin does the second step follow the first step. This will be specifically noted when it is not clear from the context.

[0035] In this document, the terms“and”,“or” and“and / or” are used. Such terms are to be read as having an inclusive meaning. For example,“A and B” can mean at least one of the following:“both A and B”,“at least A and at least B”. As another example,“A or B” can mean at least one of the following:“at least A”,“at least B”,“both A and B”,“at least A and at least B”. As another example,“A and / or B” can mean at least one of the following:“A and B”,“A or B”. When it is intended that the exclusive OR be used, this will be specifically noted (e.g.,“either A or B”,“at most one of A and B”).

[0036] The term "includes" and its variants are to be read as open-ended terms that mean "includes, but is not limited to." The term "one example implementation" and "an example implementation" are to be understood as "at least one example implementation." The term "another implementation" is to be understood as "at least one other example implementation." The term "determining" is to be understood as obtaining, receiving, calculating, estimating, approximating, predicting, or ascertaining. In addition, to the extent possible, all

[0037] This document describes various processing functions associated with structures such as blocks, elements, components, circuits, etc. Generally, these structures can be implemented by a processor controlled by one or more computer programs.

[0038] Various acronyms that can appear throughout this disclosure and associated claims and / or drawings are listed below. Other commonly used acronyms and technical terms can be excluded from this list for brevity. Accordingly, a short list of acronyms is provided below for the reader's quick reference.

[0039] IVAS - Immersive Voice and Audio Service HRTF - Head Related Transfer Function LPC - Linear Predictive Coding CLDFB - Complex Low Delay Filter Bank SBA - Scene Based Audio SPAR - Spatial Reconstruction, a spatial audio coding technique DirAC - Directional Audio Coding, another spatial audio coding technique MD - Metadata BS - Bitstream HOA - Higher Order Ambisonics FOA - First Order Ambisonics MDFT - Modified Discrete Fourier Transform MDCT - Modified Discrete Cosine Transform Figure 1A is a block diagram illustrating an example of components of an apparatus capable of implementing various aspects of the present disclosure. As with other figures provided herein, Figure 1AThe types and number of elements shown are provided by way of example only. Other implementations can include additional, fewer, and / or different types and / or numbers of elements. According to some examples, the apparatus 101 can be or can include a device configured to perform at least some of the methods disclosed herein, such as a smart audio device, a laptop computer, a cellular telephone, a tablet device, a smart home hub, etc. In some such implementations, the apparatus 101 can be or can include a server configured to perform at least some of the methods disclosed herein.

[0040] In this example, the apparatus 101 includes an interface system 105 and a control system 110. In some implementations, the interface system 105 can be configured to provide a first set of HRTFs to the control system 110. In some examples, the interface system 105 can be configured to output one or more results of the control system 110 processing the first set of HRTFs, such as a second set of HRTFs, a set of basis filters based on the second set of HRTFs, audio data (such as left ear audio data and right ear audio data) processed with one or more of the basis filters, etc.

[0041] The interface system 105 can include one or more network interfaces and / or one or more external device interfaces (such as one or more Universal Serial Bus (USB) interfaces). According to some implementations, the interface system 105 can include one or more wireless interfaces. The interface system 105 can include one or more devices for implementing a user interface, such as one or more microphones, one or more speakers, a display system, a touch sensor system, and / or a gesture sensor system. In some examples, the interface system 105 can include one or more interfaces between the control system 110 and a memory system (such as the optional memory system 115 shown in FIG. 1). However, in some instances, the control system 110 can include a memory system. Figure 1A The control system 110 can include one or more processors and / or other processing devices. For example, as shown in FIG. 1, the control system 110 can include one or more central processing units (CPUs), graphics processing units (GPUs), and / or other hardware processors.

[0042] For example, the control system 110 can include general purpose single- or multi-chip processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic, and / or discrete hardware components.

[0043] In some implementations, the control system 110 can reside in more than one device. For example, one portion of the control system 110 can reside in a device within the environment (such as a laptop computer, tablet computer, smart audio device, etc.) and another portion of the control system 110 can reside in a device outside of the environment (such as a server). In other examples, one portion of the control system 110 can reside in a device within the environment and another portion of the control system 110 can reside in one or more other devices of the environment.

[0044] In some implementations, the control system 110 can be configured to at least partially perform the methods disclosed herein. According to some examples, the control system 110 can be configured to receive a first set of HRTFs and transform the first set of HRTFs to a second set of HRTFs. The second set of HRTFs can be more efficiently approximated by linear mixing while preserving psychoacoustic properties of the first set of HRTFs. In some such examples, the transformation can involve replacing delay components of the first set of HRTFs with all-pass filters in the second set of HRTFs. According to some such examples, the transformation can involve adjusting a phase response of each of the all-pass filters in the second set of HRTFs such that: each interaural phase response is substantially linear within frequencies below an associated threshold frequency of the corresponding all-pass filter, and each phase response reduces an interaural phase difference within frequencies above the associated threshold frequency of the corresponding all-pass filter.

[0045] In some examples, the control system 110 can be configured to define a set of basis filters based on the second set of HRTFs. The number of members in the set of basis filters can be less than the number of members in the second set of HRTFs. In this context, a “member” in the second set of HRTFs is one of the HRTFs in the second set of HRTFs. Similarly, a “member” in the set of basis filters is one of the basis filters in the set of basis filters. According to some examples, the number of members in the set of basis filters can be at least one order of magnitude less than the number of members in the second set of HRTFs. For example, in some instances, the second set of HRTFs can have hundreds or thousands of members, while the set of basis filters can contain fewer than 100 members, fewer than 50 members, or even fewer than 20 members.

[0046] According to some examples, the control system 110 can be configured to receive, via the interface system 105, a bitstream of input audio data in an input audio format. The input audio format can be, for example, a high fidelity stereo audio format, an audio object based audio format (such as Dolby Atmos™), a channel based audio format, etc. In some examples, the control system 110 can be configured to combine the input audio data with one or more base filters of the base filter bank to produce left audio data and right audio data, such as left ear audio data and right ear audio data. In some such examples, the control system 110 can be configured to output, via the interface system 105, the left audio data and the right audio data. Outputting the left audio data and the right audio data can involve storing the left audio data and the right audio data, transmitting the left audio data and the right audio data, providing the left audio data and the right audio data to a set of loudspeakers for playback, providing the left audio data and the right audio data for further processing, or a combination thereof.

[0047] In some examples, the control system 110 can be configured to implement at least a portion of a codec for Immersive Voice and Audio Services (IVAS). Some examples are described herein with reference to FIG. 23.

[0048] Some or all of the methods described herein can be performed by one or more devices in accordance with instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media can include memory devices, such as those described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. The one or more non-transitory media may, for example, reside in the optional memory system 115 and / or the control system 110 shown in FIG. 1. As such, various innovative aspects of the subject matter described in this disclosure can be implemented in one or more non-transitory media having software stored thereon. For example, the software can include instructions for controlling at least one device to process audio data. The software may, for example, be executed by one or more components of a control system, such as the control system 110 of FIG. 1. Figure 1A Figure 1A In some examples, the apparatus 101 can include the optional microphone system 120 shown in FIG. 1. The optional microphone system 120 can include one or more microphones. In some implementations, the one or more microphones can be part of or associated with another device, such as a loudspeaker of a loudspeaker system, a smart audio device, etc.

[0049] In some examples, the apparatus 101 can include the optional microphone system 120 shown in FIG. 1. The optional microphone system 120 can include one or more microphones. In some implementations, the one or more microphones can be part of or associated with another device, such as a loudspeaker of a loudspeaker system, a smart audio device, etc. Figure 1A According to some implementations, the apparatus 101 can include the optional loudspeaker system 125 shown in FIG. 1. The optional loudspeaker system 125 can include one or more loudspeakers. In some implementations, the one or more loudspeakers can be part of or associated with another device, such as a microphone of a microphone system, a smart audio device, etc.

[0050] Figure 1A ​​An optional microphone system 125 is illustrated in FIG. 1. The optional microphone system 125 can include one or more microphones. Microphones can sometimes be referred to herein as "speakers." In some examples, at least some microphones of the optional microphone system 125 can be positioned arbitrarily. For example, at least some speakers of the optional microphone system 125 can be placed in locations that do not correspond to any standard prescribed speaker layout, such as Dolby 5.1, Dolby 5.1.2, Dolby 7.1, Dolby 7.1.4, Dolby 9.1, Meridian 22.2, etc. In some such examples, at least some microphones of the optional microphone system 125 can be placed in locations that are convenient for space (e.g., in locations where there is room to accommodate a microphone), but not in any standard prescribed microphone layout.

[0051] In some implementations, the apparatus 101 can include Figure 1A An optional sensor system 130 is illustrated in FIG. 1. The optional sensor system 130 can include a touch sensor system, a gesture sensor system, one or more cameras, etc.

[0052] In some implementations, the apparatus 101 can include Figure 1A An optional display system 135 is illustrated in FIG. 1. The optional display system 135 can include one or more displays, such as one or more light emitting diode (LED) displays. In some examples, the optional display system 135 can include one or more organic light emitting diode (OLED) displays. In some examples in which the apparatus 101 includes the display system 135, the sensor system 130 can include a touch sensor system and / or a gesture sensor system proximate to one or more displays of the display system 135. According to some such implementations, the control system 110 can be configured to control the display system 135 to present a graphical user interface (GUI), such as a GUI related to implementing one of the methods disclosed herein.

[0053] Figure 1B A schematic block diagram illustrating an example apparatus architecture 101 (in this example, an apparatus 101) that can be used to implement various aspects of the present disclosure is shown. Figure 1B The apparatus 101 of FIG. 1 is Figure 1A Examples of the apparatus 101 of FIG. 1. The architecture 101 includes, without limitation, server and client devices, systems, etc. that can be configured to perform the functions described with reference to Figures 11 to 17 and Figure 21As shown, architecture 101 includes a central processing unit (CPU) 141 that is capable of executing various processes according to programs stored in, for example, a read-only memory (ROM) 142 or loaded into a random access memory (RAM) 143 from, for example, a storage unit 148. CPU 141 can be, for example, an electronic processor 141. In these examples, CPU 141 is Figure 1A 141 is an example of the control system 110, and the ROM 142 and RAM 143 are examples of the memory system 115. In the RAM 143, data required when the CPU 141 executes various processes is also stored as needed. The CPU 141, ROM 142, and RAM 143 are connected to each other via a bus 144. An input / output (I / O) interface 145 is also connected to the bus 144. The bus 144 and the I / O interface 145 are Figure 1A An example of the interface system 105.

[0054] The following components are connected to the I / O interface 145: an input unit 146, which may include a keyboard, a mouse, etc.; an output unit 147, which may include a display such as a liquid crystal display (LCD) and one or more speakers; a storage unit 148, which may include a hard disk or another suitable storage device; and a communication unit 149, which may include a network interface card such as a network card (e.g., wired or wireless).

[0055] In some implementations, input unit 146 includes one or more microphones located in different locations (depending on the host device) that enable capture of audio signals in various formats (e.g., mono, stereo, spatial, immersive, and other suitable formats).

[0056] In some embodiments, output unit 147 includes a system with various numbers of speakers. Output unit 147 (depending on the capabilities of the host device) can render audio signals in various formats (e.g., mono, stereo, immersive, binaural, and other suitable formats).

[0057] In some embodiments, the communication unit 149 is configured to communicate with other devices (e.g., via a network). The driver 150 is also connected to the I / O interface 145 as needed. A removable media 151 (such as a magnetic disk, optical disk, magneto-optical disk, flash memory drive, or another suitable removable media) is installed in the drive 150 so that computer programs read therefrom are installed into the storage unit 148 as needed. Those skilled in the art will appreciate that while the device 101 is described as including the components described above, in actual applications, some of these components can be added, removed, and / or replaced, and all such modifications or changes are intended to fall within the scope of the present disclosure.

[0058] According to example embodiments of the present disclosure, the processes described above can be implemented as computer software programs or on computer-readable storage media. For example, embodiments of the present disclosure include a computer program product comprising a computer program tangibly embodied on a machine-readable medium, the computer program including program code for executing methods. In such embodiments, the computer program can be downloaded and installed from a network via the communication unit 149 and / or installed from the removable media 151, as shown. Figure 1B

[0059] Figure 1C FIG. illustrates a schematic block diagram of an example CPU 141 implemented in the device architecture 101 of Figure 1B that can be used to implement various aspects of the present disclosure. The CPU 141 includes an electronic processor 160 and a memory 161. The electronic processor 160 is electrically and / or communicatively connected to the memory 161 for bidirectional communication. The memory 161 stores encoding software 162 and decoding software 163. The memory 161 can be, for example, a ROM, a RAM, or another non-transitory computer-readable medium. The electronic processor 160 can implement the encoding software 162 stored in the memory 161 to perform, for example, the method 2100 of Figure 21 and the like. Additionally, the electronic processor 160 can implement the decoding software 163 stored in the memory 161 to perform, for example, the methods described with reference to any or all of Figures 11 to 17 and Figure 21 and the like.

[0060] In general, the various example embodiments of the present disclosure can be implemented in hardware or special-purpose circuitry (e.g., control circuitry), software, logic, or any combination thereof. For example, the units discussed above can be implemented by control circuitry (e.g., the CPU 141 and the Figure 1B ​The control circuit can perform the actions described in this disclosure, as a result of executing instructions stored in memory or otherwise accessible to the control circuit (e.g., as instructions 112 of FIG. 1). Some aspects can be implemented in hardware, while other aspects can be implemented in firmware or software such as control circuit, which can be executed by a controller, microprocessor, or other computing device. Although various aspects of example implementations of this disclosure can be illustrated and described as block diagrams, flow charts, or using some other pictorial representation, it is well understood that these blocks, apparatus, systems, techniques or methods described herein can be implemented in hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controllers, or other computing devices, or some combination thereof, as non-limiting examples.

[0061] In addition, various blocks in the flowchart can be viewed as method steps, and / or operations produced by an operating system, and / or as a couple of coupled logic circuit elements constructed to carry out the associated function(s). For example, embodiments of the present disclosure include a computer program product that comprises a computer program tangibly embodied on a machine-readable medium, the computer program including program code configured to execute the methods described above.

[0062] In the context of this disclosure, a machine-readable medium can be any tangible medium that can contain or store program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium can be non-transitory and can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0063] Computer program code for carrying out operations of the present disclosure can be written in any combination of one or more programming languages. These computer program codes can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus with control circuit, such that the program codes implement the functions / operations specified in the flowcharts and / or block diagrams when executed by the processor of the computer or the processor of other programmable data processing apparatus. The program codes can be executed entirely on a computer, partially on a computer, as a standalone software package, partially on a computer and partially on a remote computer, or entirely on a remote computer or server, or distributed on one or more remote computers and / or servers.

[0064] Example IVAS codec framework Figure 1D is a block diagram of an immersive voice and audio services (IVAS) encoder / decoder (“codec”) framework 170 for encoding and decoding IVAS bitstreams according to one or more embodiments. IVAS is expected to support a range of audio service capabilities including, but not limited to, mono-to-stereo upmix and full immersive audio encoding, decoding, and rendering. IVAS is also intended to be supported by a variety of devices, endpoints, and network nodes including, but not limited to: mobile and smart phones, electronic tablet computers, personal computers, conference phones, conference rooms, virtual reality (VR) and augmented reality (AR) devices, home theater devices, and other suitable devices.

[0065] In this example, the IVAS codec 170 includes an IVAS encoder 171 and an IVAS decoder 174. In some examples, the IVAS encoder 171, the IVAS decoder 174, or both, can be implemented by one or more instances of the control system 110, Figure 1A the CPU 141, etc. of the system 100. In some examples, the IVAS encoder 171 can be implemented by the encoding software 162 of the system 100, and the IVAS decoder 174 can be implemented by the decoding software 163 of the system 100. According to some examples, the control system implementing the IVAS encoder 171, the IVAS decoder 174, or both, can also be configured to perform some or all of the operations disclosed herein, such as the methods described with reference to one or more of Figure 1B and Figure 1C . In some examples, the IVAS encoder 171 can be implemented by the encoding software 162 of the system 100, and the IVAS decoder 174 can be implemented by the decoding software 163 of the system 100. According to some examples, the control system implementing the IVAS encoder 171, the IVAS decoder 174, or both, can also be configured to perform some or all of the operations disclosed herein, such as the methods described with reference to one or more of Figure 1C and Figure 1C . In some examples, the IVAS encoder 171 can be implemented by the encoding software 162 of the system 100, and the IVAS decoder 174 can be implemented by the decoding software 163 of the system 100. According to some examples, the control system implementing the IVAS encoder 171, the IVAS decoder 174, or both, can also be configured to perform some or all of the operations disclosed herein, such as the methods described with reference to one or more of Figures 11 to 17 and Figure 21 . In some examples, the IVAS encoder 171 can be implemented by the encoding software 162 of the system 100, and the IVAS decoder 174 can be implemented by the decoding software 163 of the system 100. According to some examples, the control system implementing the IVAS encoder 171, the IVAS decoder 174, or both, can also be configured to perform some or all of the operations disclosed herein, such as the methods described with reference to one or more of

[0066] According to this example, the IVAS encoder 171 includes a spatial encoder 172 that receives an input spatial audio (e.g., FOA, HOA) of N channels. In some implementations, the spatial encoder 172 can be configured to implement spatial reconstruction (SPAR), directional audio coding (DirAC), another spatial audio coding technique, or a combination thereof. In this example, the output of the spatial encoder 172 includes a spatial metadata (MD) bitstream (BS) and a spatial downmix of N_dmxone channel. According to this example, the spatial MD is quantized and entropy coded. In some implementations, the quantization can include fine, medium, coarse, and very coarse quantization strategies, and the entropy coding can include Huffman or arithmetic coding. In some implementations, the framework can allow no more than 3 quantization levels in a given operating mode; however, as the bit rate is reduced, the three levels generally become increasingly coarse in some such implementations to meet the bit rate requirement. According to this example, the core audio encoder 173, which can for example be based on a single-channel Enhanced Voice Services (EVS) encoding unit, is configured to encode the spatial downmix of 1-16 channels N_dmx = 1-16 channels N_dmx into an audio bitstream that is combined with the spatial MD bitstream into an IVAS encoding bitstream that is transmitted to the IVAS decoder 174.

[0067] In this example, the IVAS decoder 174 includes a core audio decoder 175 (e.g., an EVS decoder) that decodes the audio bitstream extracted from the IVAS bitstream to recover N_dmx = 1-16 audio channels. According to this example, a spatial decoder / renderer 176 (e.g., SPAR / DirAC) decodes the spatial MD bitstream extracted from the IVAS bitstream to recover the spatial MD, and uses the spatial MD and spatial upmix to synthesize / render the output audio channels for playback on various audio systems with different speaker configurations and capabilities.

[0068] Figure 1E An example is shown with respect to a coordinate system for a listener’s head. Head- related transfer function (HRTF) filters can be used to process audio signals to produce binaural audio signals, resulting in the listener’s illusion that the sound comes from a prescribed direction of arrival. The direction of arrival can be defined in terms of a unit vector, where the Cartesian coordinates can be defined as shown in Figure 1E According to the example shown in Figure 1E , the origin of the coordinate system is approximately at the center of the listener’s head 200, with the X-axis 801 pointing forward (in the direction of the listener’s nose), the Y-axis 802 pointing to the listener’s left, and the Z-axis 803 pointing upward through the top of the listener’s head.

[0069] An audio signal may be processed using HRTF filters in order to cause the listener to perceive the sound (signal ) as coming from a direction of arrival defined by a unit vector . This process involves convolving the input audio signal with a pair of HRTF filters and ​) to produce two ear signals and : (2) HRTF filter ( and ) can be obtained from the direction vector according to the following formula It turns out that: (3) In this paper, it is called HRTF set function because it is suitable for a set of Direction vector to calculate HRTF filter. HRTF set function generates this set of effective HRTF filters The vector is referred to herein as the domain of the HRTF set function.

[0070] In the explanation given below, time domain impulse responses are used to represent filter responses. It will be understood by those skilled in the art that equivalent storage and manipulation of filter responses can be performed in other domains, including but not limited to the frequency domain.

[0071] HRTF collection functions can be used to create a HRTF discrete library that defines a set of indivual Left and right ear HRTF responses of unit vectors: (4) And when the HRTF set function is evaluated in Equation 4, the HRTF discrete library can be written as: (5) It is desirable to provide a method for defining an HRTF set function, wherein each output HRTF filter generated by the HRTF set function is formed by a linear combination of basis filters. The linear HRTF set function can be defined according to Equation 6, where, Filter and The filter is calculated as: (6) According to Equation 6, a group Left-ear basis filter and Right-ear basis filter With the gain function and The defined weights are linearly combined.

[0072] In an alternative embodiment, a smaller set of basis filters and gain functions may be used to define a symmetric HRTF set function (where left ear HRTF filter for direction is the same as the right ear HRTF for direction (7) Without loss of generality, the first line of equation 7 can be examined, with the understanding that the following explanation will apply equally well to the second line of equation 7 and / or equation 6.

[0073] For a set of directions of arrival (θ ), the first line of equation 7 can be rewritten in matrix form (to simplify the equation, the subscript is also omitted in ): (8) Equation 8 can be rewritten in a simpler form as: (9) In equation 9, the column vector defines a set of left ear HRTF filter responses for a set of unit vectors (θ ), and the column vector defines a set of filter responses. In some embodiments, the goal is to determine filter responses such that the resulting HRTF filter is a close approximation to a set of original HRTF filter responses .

[0074] Various methods are known for determining suitable filters , and one example is found according to: (10) where denotes the pseudo-inverse of the matrix (as defined in equation 9).

[0075] It will be appreciated that other methods can be employed, where the goal of each method can be to minimize the size of the difference .

[0076] The number of basis filters (N ) that can be required to provide a reasonable approximation (ε ) can be very large. The difficulty with using a linear mixing process (according to equations 6, 7 or 8) is that high frequency components of the HRTF filters can often be difficult to define in the form of a linear mix.

[0077] In some embodiments, the set of original HRTF filters is modified to produce a set of modified HRTF filters where the modified HRTF filters differ from the original filters in their phase response at high frequencies. For each of the directions, the frequency response of the original and modified HRTFs can be defined using a Fourier transform: (11) frequency response function and are complex-valued functions, and thus it follows that: (12) so that the modified filters closely match the original filters at frequencies less than Hz, and the amplitude of the modified filters closely matches the original filters at higher frequencies. In various non-limiting examples, the transition frequency may be equal to about 1200 Hz, and can generally lie in a range such as, for example: In some applications, it can be desirable to reduce the number of basis functions (N) and it can be desirable to make the value of less than 1000 Hz, such as, for example, 950 Hz, 900 Hz, 850 Hz, 800 Hz, 750 Hz, 700 Hz, 650 Hz, 600 Hz, 550 Hz, or as low as 500 Hz. In other example applications, the transition frequency can lie in another range greater than 3000 Hz, such as, for example, 3050 Hz, 3100 Hz, 3150 Hz, 3200 Hz, 3250 Hz, 3300 Hz, etc.

[0078] Figure 2 , Figure 3 and Figure 4 show examples of impulse responses of HRTF filters. Figure 2 shows an impulse response 111 for a left ear HRTF filter for the following direction of arrival: which is a front-left direction. Likewise, Figure 3 shows an impulse response 211 for a right ear HRTF for the same direction of arrival. By looking at Figure 3 it can be seen that the impulse response 211 includes a delay of 0.4 ms.

[0079] Figure 4 shows a (delay-free) impulse response 311 produced by removing the 0.4 ms delay from the impulse response 211 of Figure 3 the left ear HRTF.Figure 5 An example of a graph indicating phase response versus frequency is shown. Figure 3 The 0.4 ms delay can also be defined as Figure 5 The linear phase response curve in Figure 411. In addition, Figure 5 An alternative phase response 412 is plotted in FIG, where the alternative phase curve 412 closely matches the linear phase response 411 over a frequency range between 0 Hz and 1400 Hz.

[0080] In some embodiments, Figure 3 The original impulse response 211 can be modified by removing the overall delay of 0.4 ms to produce Figure 4 The delay-free impulse response 311, and Figure 5 The phase response 412 can be applied to the impulse response 311 to produce a new filter impulse response that has the correct phase response for frequencies below 1400 Hz. Unfortunately, this may result in the new impulse response not being causal, since an additional delay of 3 ms may be added to produce the impulse response in order to implement the filter in a real-time audio process: see e.g. Figure 10 To maintain compatibility with the right ear response, this example impulse response 111 (left ear response) would also require a 3 ms delay to be added, resulting in Figure 9 The impulse response 811.

[0081] Some published examples involve modifying the original HRTF filters for both the left and right ear to provide something like Figure 5 The interaural phase difference shown in the phase response 412 is obtained without generating undesirable delays (e.g., as shown in the above reference Figure 9 and Figure 10 3 ms delay discussed above).

[0082] Figure 6 An example of a causal all-pass filter is shown. Figure 7 and Figure 8 An example of a modified HRTF that can be generated by a causal all-pass filter is shown. In some embodiments, Figure 6 The causal all-pass filters 511 and 512 can be applied to the original left ear impulse response 111 and the right ear impulse response 311, respectively, to generate Figure 7 Modified Left Ear HRTF611 and Figure 8 Modified right ear HRTF 711.

[0083] Figure 11 An example of an HRTF processing block is shown. According to some examples, Figure 11 The blocks inFigure 1A The control system 110 is implemented, for example, in accordance with instructions stored on a computer readable medium. In Figure 11 , the arrangement 100 shows an original HRTF impulse response 211 , which is received and processed by the HRTF processing block 151 to determine an overall delay 140 , i.e., the delay inherent in the impulse response 211. According to this example, the HRTF processing block 151 also produces a delay-free HRTF 311 , such that .

[0084] In the example shown, the all-pass generator 152 produces an all-pass filter impulse response 512 Figure 11 in response to the delay 140 , and the convolution process 153 combines the delay-free impulse response 311 and the all-pass response 512 to produce a modified HRTF 711 . .

[0085] The all-pass filter 512 may be defined as a function as follows: (13) where the function defines the operation of the all-pass generator 152. Of interest is the phase response of : (14) where denotes the phase response of the all-pass filter at frequency , which is produced by the all-pass generator 152 for the delay value .

[0086] Define , i.e., the all-pass phase response produced by the all-pass generator 152 when the delay is zero. This can be referred to as the zero-delay all-pass. In some embodiments, it can be required that the all-pass phase response satisfy: (15) The left side of equation 15 denotes the phase difference between the zero-delay all-pass and the all-pass filter defined for the delay . This phase difference is equivalent to the phase response 412 of Figure 5 . The right side of equation 15 denotes the linear phase slope expected at the delay . This is equivalent to the linear phase response 411 of Figure 5 .

[0087] Therefore, Equation 15 expresses the all-pass phase response 412 in this example up to The frequency should match the requirement of linear ramp phase response 411. In addition, Equation 15 defines the upper limit of the delay value range. , within the range described, the all-pass generator function is expected to be Produce effective results. Typical values ​​are (milliseconds), but in some applications, It may be some other value, such as a value between 0.6 ms and 0.8 ms, a value between 0.5 ms and 0.8 ms, a value between 0.6 ms and 0.9 ms, a value between 0.5 ms and 1.0 ms, etc.

[0088] In some embodiments, a limited set of Delay value ( ), which ranges from 0 to , and the appropriate full-pass response can be pre-calculated according to the optimization process ( ). In this case, the all-pass generator function This can be done by using a lookup table or interpolation function, This is implemented using a pre-stored all-pass response.

[0089] In some further embodiments, each of the all-pass responses (e.g., m All-pass filter ) can be defined as having An infinite impulse response (IIR) filter with a pair of conjugate poles and their corresponding pair of conjugate zeros. The basis set of filter poles , then the filter can be Defined as having extremes and zero point All-pass filter.

[0090] Therefore, when all The basis set of all-pass responses ( ) is defined as the order When the IIR filter is The full-pass response is based on The complex basis poles are completely defined: (16) The matrix in Equation 16 The complex value of can be obtained by an optimization process (such as MATLAB fminsearch function). In some examples, the optimization process can be guided by a cost function (also referred to as error function in this article), which first calculates the cost function from the matrix The base pole in the all-pass filter is calculated ( ), and then calculate the corresponding phase response according to Equation 14 ( ), and then measure how well the relative phase differences between all all-pass filter pairs match the expected delay difference. For example, the error function can be defined as: (17) Given a set The matrix of the base poles of the all-pass filter (where each all-pass filter has complex basis poles) and given a corresponding set of delay values ​​( ), a polynomial approximation can be formed so that the base poles can be defined as The polynomial approximation process can be implemented according to known methods (including but not limited to MATLAB polyfit function).

[0091] In some embodiments, the number of complex basis poles is , and can be calculated using a polynomial of order 4 as the delay According to this embodiment, for calculating the all-pass filter The process is performed through the following series of operations: 1. Given delay , calculate the base pole according to the following formula ( ):

[0092] 2. Form a pole And the corresponding zero point IIR filter:

[0093] 3. Calculate the impulse response of the IIR filter to form the all-pass response

[0094] The three steps shown above illustrate the use of polynomial functions as a convenient way to calculate the filter poles. Of course, the polynomial can only give an approximation of the "optimal" poles, and small errors in the known pole locations can result in large changes in the resulting filter response. Some alternative methods involve adjusting the polynomial values ​​(J l and J l+1) applying a non-linear function (such as Equation 18). The non-linear function can be defined such that small errors in the polynomial values (J l and J l+1 ) will no longer result in large errors in the pole locations. Further, some such examples involve computing the pole locations in the "s-domain" and then mapping them to the "z-domain." According to some examples, the mapping can be implemented using the MATLAB function "bilinear." Other choices of non-linear mapping functions can be used, and such non-linear mapping functions can produce poles in the z-domain, s-domain, or other domain.

[0095] Figure 12 Additional transformation processes that can be implemented by the allpass generator 152 of Figure 11 are shown according to some embodiments. In the example shown in Figure 12 , the allpass generator 152 receives the selected delay d 140, which is processed by a delay processing block 172 to produce a set of intermediate values 180. In this example, the intermediate values 180 are then mapped by additional non-linear processing to form a set of filter poles 182. The filter poles 182 are then processed by an allpass computation block 175 to form an allpass impulse response Figure 11 and Figure 12 of 512.

[0096] According to some examples corresponding to the blocks shown in Figure 12 , the delay processing block 172 applies the polynomial function described above to output a set of intermediate values 180, which in some examples is a set of numbers: J l In some such examples, the mapping block 173 applies a non-linear mapping process to the set of intermediate values 180, for example by implementing Equation 18, to produce an output 181, which in one example is the s-domain pole locations. In some examples, the bilinear transform block 174 converts the s-domain pole locations to produce an output 182, which in one example includes the z-domain pole locations. In the example shown in Figure 12 , the allpass computation block 175 computes an output 512, which in this example is alpha(t) (the impulse response). In alternative examples, the output 512 can be a phase response, a frequency response, or an output for any other method of defining an allpass filter response.

[0097] As will be appreciated by those skilled in the art, when using simple functions such as polynomials to produce filter poles, small deviations in the polynomial output can translate into large errors in the final allpass response when the poles are very close to the unit circle. As in Figure 12The nonlinear processing applied in to transform intermediate values ​​180 into filter poles 182 may enable a more efficient implementation of process 172 .

[0098] In some embodiments, process 172 is implemented to generate A group with an intermediate value of 180 polynomial functions. For example, ( ). Intermediate values ​​180 can then be used to generate s-plane filter poles 181 by filter pole generation block 173 in this example. For example, two intermediate values ​​can be used and To define the complex s-plane poles : (18) Alternatively, a single intermediate value (e.g., ) based on To define a single real s-plane pole .

[0099] The s-plane pole 181 may then be transformed (in this example by the transform block 174) into a z-plane pole 182. For example, this transformation may be performed by the transform block 174 using the MATLAB bilinear function: p(n) = bilinear(P(n),1,1,48000), Or this transformation: p(n) = bilinear(P(n),1,1,48000,Fp), Where Fp represents the upper frequency (as used in Equation 15), and 48000 is the sampling rate according to this embodiment. It should be understood that alternative sampling rates may be used, including but not limited to 16000, 32000, 44100, or 96000.

[0100] It will be appreciated by those skilled in the art that other nonlinear processing methods may be used to facilitate the selection of the delay 140 is mapped to a set of all-pass poles 182. In an alternative embodiment, the polynomial function applied by the delay processing block 172 can be used to define the frequency and Q of the poles, and the nonlinear mapping process applied by the mapping block 173 can convert the frequency and Q values ​​to the corresponding pole locations. In another embodiment, the nonlinear mapping process applied by the mapping block 173 can determine the z-domain pole locations, thereby eliminating the need for the bilinear transformation of the transform block 174.

[0101] It should also be understood that by forming an additional conjugate pole (for each complex pole in set 182) and by forming each filter zero as the inverse of each corresponding pole, an allpass filter response (in this example by allpass filter response block 175) can be derived, and this allpass filter will possess causality.

[0102] Figure 13 The process of generating a set of basis filters from a set of HRTFs is illustrated. In some examples, Figure 13 The blocks in can be implemented, at least in part, by a control system 110 of Figure 1A . Figure 13 An arrangement 500 is shown in which an original HRTF bank 520 is processed (in this example by HRTF transform block 521) to produce a modified HRTF bank 541. In this example, the interaural delay component inherent in the HRTF filters in the original HRTF set is replaced by an allpass filter satisfying equation 15, and the modified HRTF bank has reduced interaural phase at frequencies greater than The modified HRTF 521 is processed (in this example by basis filter generation block 522) to produce a set of basis filters 523 according to a fitting process, such as the fitting process of equation 10.

[0103] The number of members in the basis filter set 523 is less than the number of members in the modified HRTF set. In this context, a "member" in the basis filter set 523 is one of the basis filters in the basis filter set 523, and a "member" in the modified HRTF set is one of the HRTFs in the modified HRTF set. According to some examples, the number of members in the basis filter set 523 can be at least one order of magnitude less than the number of members in the modified HRTF set. For example, in some instances, the modified HRTF set can have hundreds or thousands of members, while the basis filter set 523 can contain fewer than 100 members, fewer than 50 members, or even fewer than 20 members. Thus, the basis filter set 523 forms a compact representation of the original HRTF set 520.

[0104] Figure 14 The process of generating a set of basis filters from a set of HRTFs, and using this set of basis filters to form left and right HRTF filters is illustrated. In some examples, Figure 14 The blocks in can be implemented, at least in part, by a control system 110 of Figure 1A . Figure 14An arrangement 501 is shown in which an original HRTF library 520 is processed by an HRTF transform block 521 to produce a modified HRTF library 541 which is then processed by a basis filter generation block 522 to produce a set of basis filters 523. Directions of arrival 524 (which can be defined according to spherical coordinates , unit vectors , or other forms known in the art) are processed by a weight coefficient generation block 525 to form weight coefficients 526. In some embodiments, the weight coefficients can be defined according to a spherical harmonic translation equation, and the basis filters can likewise be adapted to be compatible with the spherical harmonic translation equation, e.g., the .

[0105] In this example, a weight coefficient and basis filter combination block 527 combines the weight coefficients 526 with the basis filters 523 to form left ear HRTF filters and right ear HRTF filters (528, 529, respectively) representing modified HRTFs for specified directions of arrival. When the basis filters represent a set of symmetric HRTFs, the weight coefficient and basis filter combination block 527 can be implemented, e.g., according to Equation 7. When the basis filters represent a set of HRTFs containing asymmetries, the weight coefficient and basis filter combination block 527 can be implemented, e.g., according to Equation 6.

[0106] Figure 15 A process is illustrated of generating a set of basis filters from a set of HRTFs, and using the set of basis filters to form left and right audio signals. In some examples, Figure 15 The blocks in Figure 1A may be implemented at least in part by the control system 110 of Figure 15 An arrangement 502 is shown in which an original HRTF library 520 is processed by an HRTF transform block 521 to produce a modified HRTF library 541 which is then processed by a basis filter generation block 522 to produce a set of basis filters 523. According to this example, an audio generation block 530 produces audio signals 531 in a form associated with a scene-based audio format, such as high fidelity stereo or higher order high fidelity stereo. The audio generation block 530 can be or can include an audio decoder adapted to produce a multi-channel audio bitstream from a transmitted or stored encoded bitstream. Alternatively, the audio generation block 530 can be or can include an audio capture and / or processing device adapted to produce scene-based audio signals 531 representing a spatial audio scene.

[0107] According to this example, the audio input and base filter combination block 532 is adapted to combine the audio signal 531 with the base filter 523 to produce left and right ear audio signals (533, 534, respectively). In some examples, the audio input and base filter combination block 532 can be configured to implement a convolution process, which can be implemented according to known time-domain or frequency-domain methods, as known in the art.

[0108] Figure 16 Additional details of the HRTF transform block in Figures 13 to 15 are shown according to some embodiments. In some examples, Figure 16 the block in Figure 1A may be implemented at least in part by the control system 110 of Figures 13 to 15 . Figure 17 A more detailed view of the process in the upper portion of is shown. According to this example, each left / right HRTF pair is processed by a corresponding HRTF transform sub-block 150.

[0109] Figure 16 Additional details of the HRTF transform sub-block in Figure 17 are shown according to some embodiments. In some examples, Figure 1A the block in Figure 17 may be implemented at least in part by the control system 110 of Example delay definitions for each ear . An example of an HRTF transform sub-block 150 150 is shown, in which the L and R HRTFs (211L and 211R) are processed by left and right HRTF processing blocks 151L and 151R, respectively, to extract the non-delayed impulse responses 311L / 311R and the delays 140L / 140R. These two delays 140L / 140R are then processed by a delay processing block 138 to produce new simplified delays 141L / 141R. The simplified delays are each processed by all-pass filter generation blocks 152L and 152R to form all-pass filters 512L / 512R. According to this example, modified left and right HRTF generation blocks 153L and 153R are configured to combine the non-delayed impulse responses 311L / 311R with the all-pass filters 512L / 512R to form modified HRTF pairs 711L / 711R.

[0110] Figure 17 Figure 18 The delay processing block 138 in Figure 19 is configured to respond to the difference between the delays 140L / 140R to generate new delay values 141L / 141R. Figure 20 , Figure 17 An example of an HRTF transform sub-block 150 150 is shown, in which the L and R HRTFs (211L and 211R) are processed by left and right HRTF processing blocks 151L and 151R, respectively, to extract the non-delayed impulse responses 311L / 311R and the delays 140L / 140R. These two delays 140L / 140R are then processed by a delay processing block 138 to produce new simplified delays 141L / 141R. The simplified delays are each processed by all-pass filter generation blocks 152L and 152R to form all-pass filters 512L / 512R. According to this example, modified left and right HRTF generation blocks 153L and 153R are configured to combine the non-delayed impulse responses 311L / 311R with the all-pass filters 512L / 512R to form modified HRTF pairs 711L / 711R.Figure 18 An example of a function implemented by the delay processing block 138 is given below. For Figure 19 , Figure 20 and Figure 18 , the corresponding functions are as follows: According to Figure 19 : (19) According to Figure 20 : (20) According to Figure 20 : (21) In equation 21, denotes the maximum expected value of .

[0111] One important property of the function implemented by the delay processing block 138 is that it produces a modified delay d' L and d' R : such that the interaural delay difference between the left (L) and right (R) HRTFs is preserved.

[0112] Example MATLAB implementation The function in

[0113] In some further embodiments, the left HRTF processing block 151L and the right HRTF processing block 151R can be adapted to produce the undelayed impulse responses 311L and 311R as minimum phase filter responses, respectively.

[0114] According to some embodiments, the left HRTF processing block 151L and the right HRTF processing block 151R can be implemented according to the following steps: 1. Determine the frequency response of the original HRTF filter, e.g., as defined in equation 11 .

[0115] 2. Determine the amplitude response of the original HRTF filter:

[0116] 3. Apply a Hilbert transform to determine the frequency response of the new (undelayed) minimum phase filter according to methods known in the art:

[0117] 4. Determine the phase responses of the original filter and the undelayed filter:

[0118]

[0119] where, The function extracts the phase component of the complex frequency response, and The operation is known in the art as a method for removing discontinuities in the extracted phase response by adding multiples of at each frequency (e.g., the MATLAB unwrap() function).

[0120] 5. Determine the delay associated with the original HRTF filter as:

[0121] In some examples, the delay measurement frequency is a value in the range of 300 Hz to 1600 Hz. In a detailed example, In some other examples, may be a value in the range of 300 Hz to 1200 Hz, a value in the range of 600 Hz to 1800 Hz, a value in the range of 1000 Hz to 1400 Hz, or a value in some other frequency range.

[0122] In some embodiments, the modified HRTF response is determined using the delay and the minimum phase response as determined according to the above method according to the following steps: 1. For each of the original left ear HRTF filter and right ear HRTF filter (for a given direction of arrival), use the above procedure to determine the delay and the minimum phase response and label them as:

[0123]

[0124] 2. Determine the maximum interaural delay difference:

[0125] such that the maximum interaural delay for all directions of arrival ( ) is defined.

[0126] 3. Determine new left ear delay value and new right ear delay value and such that:

[0127] where the left ear delay value and the right ear delay value and may be determined according to equation 21.

[0128] 4. Determine all-pass filter frequency response:

[0129]

[0130] where, may be a function described in equation 13 that determines the all-pass filter phase response associated with the delay .

[0131] 5. Determine new modified HRTF filter:

[0132]

[0133] Figure 21 The following MATLAB implementation provides additional details according to the disclosed method. An example of a MATLAB function that determines a set of HRTF base filters from an existing HRTF library is shown below:

[0134]

[0135]

[0136]

[0137]

[0138]

[0139] Figure 1Ais a flowchart that outlines one example of a method that can be performed by an apparatus or system such as the apparatus or system disclosed herein. As with other methods described herein, the blocks of method 2100 need not be performed in the indicated order. In some embodiments, one or more of the blocks of method 2100 can be performed concurrently. Also, some embodiments of method 2100 can include additional blocks beyond those shown and / or described. Blocks of method 2100 can be performed by one or more devices, which can be (or can include) a control system such as the control system 110 shown and described above. Figures 13 to 16

[0140] In this example, method 2100 is an audio processing method. According to this example, block 2105 involves obtaining, by a control system, a first set of HRTFs. The first set of HRTFs can be, for example, the original HRTF library 520 in Figures 13 to 16

[0141] In this example, block 2110 involves transforming, by the control system, the first set of HRTFs into a second set of HRTFs. The second set of HRTFs can be, for example, the modified HRTF library 541 in Figure 5 According to this example, the transformation process of block 2110 involves replacing the delay components of the first set of HRTFs with allpass filters in the second set of HRTFs. In this example, the transformation process of block 2110 also involves adjusting the phase response of each of the allpass filters in the second set of HRTFs such that: each interaural phase response is substantially linear within a frequency below an associated threshold frequency of the corresponding allpass filter, and each phase response reduces an interaural phase difference within a frequency above the associated threshold frequency of the corresponding allpass filter. The threshold frequency can be, for example, a frequency at which the alternative phase curve 412 deviates from the linear phase response 411 of Figure 13

[0142] In this example, block 2115 involves outputting a result of the adjustment of the phase response of each of the allpass filters in the second set of HRTFs. Outputting the result can involve, for example, storing the result, transmitting the result, providing the result for further processing, or a combination thereof.

[0143] According to some examples, method 2100 can involve one or more of the aspects of the apparatus or system disclosed herein with reference to Figure 14 ​​​The described process, among other processes. In some such examples, the method 2100 can further involve defining, by the control system, a set of base filters based on the second set of HRTFs. The number of members in the set of base filters can be less than the number of members in the second set of HRTFs. In some examples, the number of members in the set of base filters can be at least one order of magnitude less than the number of members in the second set of HRTFs.

[0144] In some examples, the method 2100 can further involve a process such as the described process, among other processes. In some such examples, the method 2100 can further involve obtaining, by the control system, a bitstream of input audio data in an input audio format, and combining, by the control system, the input audio data with one or more base filters of the set of base filters to produce left audio data and right audio data. According to some such examples, the method 2100 can further involve outputting, by the control system, the left audio data and the right audio data. Outputting the left audio data and the right audio data can involve storing the left audio data and the right audio data, transmitting the left audio data and the right audio data, providing, by the control system, the left audio data and the right audio data to a set of loudspeakers for playback, providing the left audio data and the right audio data (e.g., to other modules implemented by the control system, to another control system, or a combination thereof) for further processing. Figure 15 Figure 16 In some examples, the method 2100 can further involve a process such as the described process, among other processes. In some such examples, the method 2100 can further involve obtaining, by the control system, a bitstream of input audio data in an input audio format, and combining, by the control system, the input audio data with one or more base filters of the set of base filters to produce left audio data and right audio data. According to some such examples, the method 2100 can further involve outputting, by the control system, the left audio data and the right audio data. Outputting the left audio data and the right audio data can involve storing the left audio data and the right audio data, transmitting the left audio data and the right audio data, providing, by the control system, the left audio data and the right audio data to a set of loudspeakers for playback, providing the left audio data and the right audio data (e.g., to other modules implemented by the control system, to another control system, or a combination thereof) for further processing.

[0145] According to some examples, the transformation process of block 2110 can involve a process such as the described process, among other processes. According to some such examples, block 2110 can involve obtaining a left ear HRTF and a right ear HRTF from the first set of HRTFs, identifying a left ear non-delayed impulse response and a left ear delay from each of the left ear HRTFs, and identifying a right ear non-delayed impulse response and a right ear delay from each of the right ear HRTFs. In some such examples, block 2110 can involve producing a left ear allpass filter (each of the left ear allpass filters based at least in part on an instance of the left ear delay), and producing a right ear allpass filter (each of the right ear allpass filters based at least in part on an instance of the right ear delay). In some such examples, block 2110 can involve combining instances of the left ear non-delayed impulse response with corresponding instances of the left ear allpass filter and combining instances of the right ear non-delayed impulse response with corresponding instances of the right ear allpass filter to produce pairs of HRTFs of the second set of HRTFs. Figure 17 Figure 20 In some examples, the method 2100 can further involve a process such as the described process, among other processes. In some such examples, the method 2100 can further involve obtaining, by the control system, a bitstream of input audio data in an input audio format, and combining, by the control system, the input audio data with one or more base filters of the set of base filters to produce left audio data and right audio data. According to some such examples, the method 2100 can further involve outputting, by the control system, the left audio data and the right audio data. Outputting the left audio data and the right audio data can involve storing the left audio data and the right audio data, transmitting the left audio data and the right audio data, providing, by the control system, the left audio data and the right audio data to a set of loudspeakers for playback, providing the left audio data and the right audio data (e.g., to other modules implemented by the control system, to another control system, or a combination thereof) for further processing.

[0146] ​​In some such examples, block 2110 can involve generating a modified left-ear delay value based on one or more of the extracted left-ear delays and generating a modified right-ear delay value based on one or more of the extracted right-ear delays. The left-ear all-pass filter and the right-ear all-pass filter can be based on the modified left-ear delay value and the modified right-ear delay value, respectively. In some examples, generating the modified left-ear delay value and the right-ear delay value can involve determining a difference between the extracted left-ear delay and the extracted right-ear delay. According to some examples, generating the modified left-ear delay value and the right-ear delay value can involve determining a maximum expected difference between the extracted left-ear delay and the extracted right-ear delay.

[0147] According to some examples, the difference between the extracted left ear delay and the extracted right ear delay can be equal to the difference between the corresponding modified left ear delay value and the modified right ear delay value. In some examples, the modified left ear delay value and the modified right ear delay value can correspond to a smooth function, such as Figure 5 According to some examples, each pair of the modified left-ear delay value and the modified right-ear delay value can include a lower delay value and a higher delay value. In some such examples, the delay variation of the lower delay value can be less than the delay variation of the higher delay value.

[0148] In some examples, the non-delayed impulse response may be a minimum phase filter response.

[0149] According to some examples, extracting each left ear non-delayed impulse response, each right ear non-delayed impulse response, each left ear delay, and each right ear delay from each of the left ear HRTF and the right ear HRTF can involve: determining a frequency response of an original HRTF filter in the first set of HRTFs; determining a magnitude response of the original HRTF filter; determining a minimum phase frequency response of a new non-delayed minimum phase filter; determining a phase response of the original HRTF filter and a phase response of the new non-delayed minimum phase filter; and determining a delay associated with the original HRTF filter based at least in part on the phase response of the original HRTF filter and the phase response of the new non-delayed minimum phase filter. In some such examples, determining the minimum phase frequency response can involve performing a Hilbert transform involving the magnitude response of the original HRTF filter. According to some examples, determining the delay associated with the original HRTF filter can also be based at least in part on a delay measurement frequency in the range of 300 Hz to 1600 Hz.

[0150] In some examples, the all-pass phase response can deviate from the linear ramp phase response and can smoothly approach zero phase within frequencies higher than the threshold frequency. Figure 1D An alternative phase curve 412 provides one such example.

[0151] According to some examples, the control system configured to implement the method 2100 is further configured to implement at least part of a codec for Immersive Voice and Audio Services (IVAS). ​ One such example is shown.

[0152] The above description illustrates various embodiments of the disclosure and examples of how aspects of the disclosure can be implemented. The above examples and embodiments should not be considered to be the only ones in which the disclosure can be implemented, but are presented in order to illustrate the flexibility and advantages of the disclosure as defined by the appended claims. Other arrangements, embodiments, implementations and equivalents, based on the above disclosures and claims, will be apparent to persons skilled in the art and can be employed without departing from the spirit and scope of the disclosure as defined by the claims.

Claims

1. A method for audio processing in a control system comprising one or more processors, the method comprising: obtaining a first set of head-related transfer functions (HRTFs) by the control system; The control system transforms the first set of HRTFs into a second set of HRTFs, wherein the transforming comprises: replacing the delayed components of the first set of HRTFs with the all-pass filters in the second set of HRTFs; The phase response of each of the all-pass filters in the second set of HRTFs is adjusted such that: each interaural phase response is substantially linear within frequencies below the associated threshold frequency of the corresponding all-pass filter, and Each phase response reduces interaural phase differences within frequencies above an associated threshold frequency of a corresponding all-pass filter; and The second set of HRTFs is output.

2. The audio processing method according to claim 1, wherein: Outputting the second set of HRTFs involves storing the second set of HRTFs, transmitting the second set of HRTFs to a device configured to process audio data, providing the second set of HRTFs for further processing, or a combination thereof.

3. The audio processing method according to claim 1 or claim 2, further comprising: defining, by the control system, a set of basis filters based on the second set of HRTFs, wherein the number of members in the base filter group is less than the number of members in the second HRTF group; obtaining, by the control system, a bitstream of input audio data in an input audio format; combining, by the control system, the input audio data with one or more base filters of the base filter bank to generate left audio data and right audio data; and The left audio data and the right audio data are output by the control system.

4. The audio processing method according to claim 3, wherein: Outputting the left audio data and the right audio data involves storing the left audio data and the right audio data, transmitting the left audio data and the right audio data, providing the left audio data and the right audio data to a set of loudspeakers for playback by the control system, providing the left audio data and the right audio data for further processing, or a combination thereof.

5. The audio processing method according to any one of claims 1 to 4, wherein: The transformation further comprises: Obtaining a left-ear HRTF and a right-ear HRTF from the first set of HRTFs; identifying a left-ear non-delayed impulse response and a left-ear delay from each of the left-ear HRTFs; identifying a right ear non-delayed impulse response and a right ear delay from each of the right ear HRTFs; generating left ear all-pass filters, each of the left ear all-pass filters being based at least in part on an instance of the left ear delay; generating right ear all-pass filters, each of the right ear all-pass filters being based at least in part on an instance of the right ear delay; and The instances of the left ear non-delayed impulse response are combined with corresponding instances of the left ear all-pass filter and the instances of the right ear non-delayed impulse response are combined with corresponding instances of the right ear all-pass filter to produce HRTF pairs for the second set of HRTFs.

6. The audio processing method of claim 5 , further comprising generating a modified left-ear delay value based on one or more of the extracted left-ear delays and generating a modified right-ear delay value based on one or more of the extracted right-ear delays, wherein The left ear all-pass filter is based on the modified left ear delay value and the right ear all-pass filter is based on the modified right ear delay value.

7. The audio processing method according to claim 6, wherein: An example of generating the modified left-ear and right-ear delay values ​​involves determining a difference between an extracted left-ear delay and an extracted right-ear delay.

8. The audio processing method according to claim 6 or claim 7, wherein: An example of generating the modified left-ear and right-ear delay values ​​involves determining a maximum expected difference between an extracted left-ear delay and an extracted right-ear delay.

9. The audio processing method according to any one of claims 6 to 8, wherein: The difference between the extracted left ear delay and the extracted right ear delay is equal to the difference between the corresponding modified left ear delay value and the modified right ear delay value.

10. The audio processing method according to any one of claims 6 to 9, wherein: The modified left-ear delay value and the modified right-ear delay value correspond to a smooth function.

11. The audio processing method according to any one of claims 6 to 10, wherein: Each pair of the modified left-ear delay value and the modified right-ear delay value comprises a lower delay value and an upper delay value, and wherein a delay variation of the lower delay value is less than a delay variation of the upper delay value.

12. The audio processing method according to any one of claims 5 to 11, wherein: The non-delayed impulse response is a minimum phase filter response.

13. The audio processing method according to any one of claims 5 to 12, wherein: Extracting each left-ear non-delayed impulse response, each right-ear non-delayed impulse response, each left-ear delay, and each right-ear delay from each of the left-ear HRTF and the right-ear HRTF involves: determining a frequency response of an original HRTF filter in the first set of HRTFs; determining a magnitude response of the original HRTF filter; determining a minimum phase frequency response of a new no-delay minimum phase filter; determining a phase response of the original HRTF filter and a phase response of the new undelayed minimum phase filter; and A delay associated with the original HRTF filter is determined based at least in part on the phase response of the original HRTF filter and the phase response of the new non-delayed minimum phase filter.

14. The audio processing method according to claim 13, wherein: Determining the minimum phase frequency response involves performing a Hilbert transform involving the magnitude response of the original HRTF filter.

15. The audio processing method according to claim 13 or claim 14, wherein: Determining the delay associated with the original HRTF filter is also based at least in part on a delay measurement frequency within a range of 300 Hz to 1600 Hz.

16. The audio processing method according to any one of claims 3 to 15, wherein: The number of members in the base filter group is at least one order of magnitude less than the number of members in the second HRTF group.

17. The audio processing method according to any one of claims 1 to 16, wherein: The all-pass phase response deviates from a linear ramp phase response and smoothly approaches zero phase within frequencies above the threshold frequency.

18. The audio processing method according to any one of claims 1 to 17, wherein: The control system corresponds to at least a portion of a codec for immersive voice and audio services (IVAS).

19. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform the operations of the method of any one of claims 1 to 18.

20. An audio processor device for processing input audio data, the audio processor device comprising: a receiver unit configured to receive the input audio data; A computer unit, the computer unit being configured to: Obtain a first set of head-related transfer functions (HRTFs); Transforming the first set of HRTFs into a second set of HRTFs, wherein the transforming comprises: replacing the delayed components of the first set of HRTFs with the all-pass filters in the second set of HRTFs; The phase response of each of the all-pass filters in the second set of HRTFs is adjusted such that: each interaural phase response is substantially linear within frequencies below the associated threshold frequency of the corresponding all-pass filter, and Each phase response reduces interaural phase differences within frequencies above an associated threshold frequency of a corresponding all-pass filter; and The second set of HRTFs is output.

21. The audio processor device of claim 20, wherein: Outputting the second set of HRTFs involves storing the second set of HRTFs, transmitting the second set of HRTFs to a device configured to process audio data, providing the second set of HRTFs for further processing, or a combination thereof.

22. An audio processor device as claimed in claim 20 or claim 21, wherein: The computer unit is further configured to: defining a set of base filters based on the second set of HRTFs, wherein the number of members in the set of base filters is less than the number of members in the second set of HRTFs; obtaining a bitstream of input audio data in an input audio format; combining the input audio data with one or more base filters in the base filter bank to generate left audio data and right audio data; and The left audio data and the right audio data are output.

23. The audio processor device of claim 22, wherein: Outputting the left audio data and the right audio data involves storing the left audio data and the right audio data, transmitting the left audio data and the right audio data, providing the left audio data and the right audio data to a set of loudspeakers for playback by the control system, providing the left audio data and the right audio data for further processing, or a combination thereof.

24. The audio processor device of claim 22 or claim 23, further comprising a storage device configured to store the first HRTF, the second HRTF, the left audio data, the right audio data, the input audio data, or a combination thereof.

25. The audio processor device of claim 24, wherein: The storage device includes one or more of a random access memory, a read-only memory, and a non-transitory computer-readable medium.

26. The audio processor device of any one of claims 20 to 25, wherein: The device corresponds to at least a portion of a codec for Immersive Voice and Audio Services (IVAS).

Images