Method and system for head-related transfer function adaptation - Patents.com

Abstract

The present disclosure provides a method and system for head-related transfer function (HRTF) adaptation, the method including performing system identification, where the system identification includes pinna identification and shadowing identification.

Description

[Technical field]

[0001] The present disclosure relates to the field of audio, and more specifically, to a method and system for head-related transfer function (HRTF) adaptation using a hybrid adaptive active noise canceller (ANC) loop. [Background technology]

[0002] In the past few years, ANC earphones have become increasingly popular because they can provide users with a relatively quiet environment in a noisy environment, reduce unwanted environmental noise, and thus bring more convenience and comfort to users.

[0003] As people's demands for user experience continue to grow, spatial audio technology (also known as 3D audio technology) has become more prominent and used. This technology allows earphones to be used to create a 3D audio experience. Applications of this technology include realizing augmented virtual reality, listening to music, and watching movies on tablets and PCs. Virtual surround earphones are a typical application of 3D audio technology. Producing surround sound through 3D audio earphones provides the same audio experience as listening to a real speaker system.

[0004] HRTFs are an advanced way of presenting 3D audio, so that sounds appear to come from specific points in 3D space, and synthesizing binaural audio. To achieve a faithful and immersive experience during binaural audio playback, HRTFs are often used as filters to describe the transmission of sound from the sound source to the listener's eardrums.

[0005] ANC earphones are another typical application that uses the HRTF from the noise source to the ear entry point (EEP) to introduce a matched amplitude but opposite phase sound wave to reduce the severity of noise pollution (such as street noise, aircraft engine noise, and office chatter).

[0006] In a word, HRTFs are highly personalized and different for each individual. Every person has a different upper body contour and ear shape, resulting in different acoustic filtering effects. In current practice, an average HRTF from a group of subjects is usually used offline and in earphones. This method of using average HRTFs has two drawbacks: 1) When the average HRTF is barely able to match the actual end user experience, the front-to-back and up-to-down confusion associated with 3D audio (the so-called confusion cone) results in a very poor sound image localization effect. 2) Modifying the non-personalized average HRTF might be a way to save effort, but it always involves undesirable audio distortions.

[0007] Existing HRTF measurements use a set of speakers mounted on a rotating semicircular ring to generate an excitation signal (e.g., an exponential sweep signal). A dummy head or an individual's head is placed in the center of the semicircular ring, and microphones are placed on the eardrums of the left and right ears of the dummy head or individual's head. However, such measurements are very difficult and time-consuming.

[0008] In addition, current ANC designs use fixed HRTF / offline HRTF or require dedicated hardware, which makes the cost much higher. Also, ANC designs with fixed HRTF have two drawbacks: 1) they cannot accurately adapt to various real-world environmental noises based on on-site calibration / measurement; and 2) they cannot achieve user personalization. For example, human differences between earphones will lead to inconsistent ANC results, and leakage will occur due to various fitting conditions of earphones and wearer's heads.

[0009] To overcome the above drawbacks of inaccurate and non-personalized HRTFs, improved solutions are needed. Summary of the Invention [Means for solving the problem]

[0010] The present disclosure provides a solution for obtaining adaptive HRTFs, for example, from far-field to near-field and from ear reference point (ERP) to ear entrance point (EEP) via adaptive ANC. In addition, in another embodiment of the present disclosure, the adaptive HRTFs are used for compensation in applications such as ANC earphone applications and 3D earphone applications. In addition, the present disclosure can provide a hybrid (feedback + feedforward) adaptive ANC to adapt to different adaptation conditions.

[0011] According to one or more aspects of the present disclosure, a method for HRTF adaptation is provided. The method includes performing a system identification. The system identification includes pinna identification and shadowing identification. The method provided according to one or more aspects of the present disclosure further includes performing a system compensation based on the adapted HRTF obtained from the system identification. The method provided according to one or more aspects of the present disclosure further includes generating an HRTF rendering matrix based on the pinna identification and the shadowing identification.

[0012] According to one or more aspects of the present disclosure, a system for HRTF adaptation is provided. The system includes a memory and a processor. The memory is configured to store computer readable instructions. The processor is configured to perform system identification upon execution of the computer readable instructions. The system identification includes pinna identification and shadowing identification.

[0013] Another embodiment of the present disclosure provides a computer-readable medium configured to perform the steps of the above-described method.

[0014] Advantageously, the method and system disclosed in the present disclosure can provide personalized HRTFs for different users, so that the users can have a better sound experience when using earphones. The present specification also provides, for example, the following items: (Item 1) A method for head-related transfer function (HRTF) adaptation comprising performing system identification, the system identification including pinna identification and shadowing identification. (Item 2) Item 10. The method of item 1, further comprising performing system compensation based on adapted HRTFs obtained from the system identification. (Item 3) 3. The method of claim 2, wherein the system compensation comprises generating an HRTF rendering matrix based on outputs of the pinna identification and the shadowing identification. (Item 4) The ear identification, Obtaining an adapted HRTF from an ear reference point (ERP) to an ear entrance point (EEP); performing curve fitting based on the adapted HRTF to obtain a compensation curve for the HRTF; Multiplying the frequency domain (FD) audio signal by the compensation curve; The method according to item 1, comprising: (Item 5) 5. The method of claim 4, wherein the compensation curve is an inverse function of the HRTF. (Item 6) 2. The method according to claim 1, wherein the pinna identification is implemented by a feedback loop of an active noise canceller (ANC) with an adaptive controller. (Item 7) 2. The method of claim 1, wherein the shadowing identification is implemented by a feedforward loop of an ANC that includes an adaptive controller. (Item 8) The shadowing identification, inputting the audio signal received from the reference microphone and the audio signal received from the error microphone into an adaptation controller to obtain an adaptation HRTF; The method according to item 1, comprising: (Item 9) 9. The method of claim 8, further comprising performing system compensation with a hybrid output of the adaptive HRTFs obtained from the pinna identification and the adaptive HRTFs obtained from the shadowing identification. (Item 10) a memory configured to store computer readable instructions; a processor configured to execute computer readable instructions to perform system identification, the system identification including pinna identification and shadowing identification; A system for head-related transfer function (HRTF) adaptation comprising: (Item 11) Item 11. The system of item 10, wherein the processor is further configured to perform system compensation based on adaptive HRTFs obtained from the system identification. (Item 12) Item 12. The system of item 10 or 11, wherein the processor is further configured to generate an HRTF rendering matrix based on the output of the pinna identification and the shadowing identification. (Item 13) The processor, Obtaining an adapted HRTF from an ear reference point (ERP) to an ear entrance point (EEP); performing curve fitting based on the adapted HRTF to obtain a compensation curve for the HRTF; Multiplying the frequency domain (FD) audio signal by the compensation curve; Item 11. The system of item 10, further configured to perform the pinna identification including: (Item 14) Item 14. The system of item 13, wherein the compensation curve is an inverse function of the HRTF. (Item 15) Item 11. The system of item 10, wherein the processor is further configured to implement the pinna identification by a feedback loop of an active noise canceller (ANC) comprising an adaptive controller. (Item 16) Item 11. The system of item 10, wherein the processor is further configured to implement the shadowing identification by a feedforward loop of an ANC comprising an adaptive controller. (Item 17) The processor, inputting the audio signal received from the reference microphone and the audio signal received from the error microphone into an adaptation controller to obtain an adaptation HRTF; Item 11. The system of item 10, further configured to perform the shadowing identification including: (Item 18) Item 18. The system of item 17, wherein the processor is further configured to perform system compensation with a hybrid output of the adaptive HRTF obtained from the pinna identification and the adaptive HRTF obtained from the shadowing identification. (Item 19) 10. A computer-readable medium comprising instructions for carrying out the method of any one of items 1 to 9.

[0015] The present disclosure can be better understood by reading the following description of non-limiting embodiments with reference to the accompanying drawings, in which parts are not necessarily to scale, with the focus being on illustrating the principles of the invention. Moreover, in the drawings, similar or identical reference numbers refer to similar or identical elements. [Brief description of the drawings]

[0016] [Figure 1] 1 shows a schematic diagram of the method and system of the present disclosure. [Diagram 2] 1 shows a schematic diagram of an ANC feedback loop according to an embodiment of the present disclosure. [Diagram 3] 1 shows a left ear transfer function (TF) curve graph measured by a method according to an embodiment of the present disclosure. [Figure 4] 1 shows a right ear TF curve graph measured by a method according to an embodiment of the present disclosure. [Diagram 5] FIG. 1 shows a schematic diagram of an ANC feedforward loop of another embodiment of the present disclosure. [Figure 6] A schematic diagram of an acoustic echo cancellation system H(Z) implemented in the frequency domain (FD) is shown. [Figure 7]A schematic diagram of the acoustic echo cancellation system H(Z) adaptation implemented in FD is shown. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] It should be understood that the following description of the embodiments is given for illustrative purposes only and is not limiting. The division in the examples of functional blocks, modules, or units shown in the drawings should not be interpreted as representing these functional blocks, and these modules or units need to be implemented as physically separated units. The functional blocks, modules, or units shown or described can be implemented as individual units, circuits, chips, functions, modules, or circuit elements. One or more functional blocks or units can also be implemented in a common circuit, chip, circuit element, or unit.

[0018] Any one or more of the processors, memories, or systems described herein include computer-executable instructions that may be compiled or interpreted from computer programs created using various programming languages ​​and / or technologies. Generally speaking, a processor (such as a microprocessor) receives and executes instructions, for example, from a memory, a computer-readable medium, etc. The processor includes a non-transitory computer-readable storage medium capable of executing instructions of a software program. The computer-readable medium may be, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof.

[0019] FIG. 1 shows a schematic diagram of the method and system of the present disclosure. As shown in FIG. 1, the present disclosure provides a method and system for HRTF adaptation. The method may include system identification and system compensation. System identification aims to determine the difference between a reference model and a user. System identification mainly focuses on pinna differences and shadowing features. That is, system identification may include pinna identification and shadowing identification. System compensation aims to compensate for the system difference between the reference model and a user using mathematical modeling techniques. For example, an HRTF rendering matrix is ​​generated based on the output of pinna identification and shadowing identification.

[0020] For system identification, we can refer to the acoustic echo canceller (AEC) in the communication system for modeling. For convenience of explanation, FIG. 6 shows a schematic diagram of the acoustic echo cancellation system H(Z) implemented in FD. The explanation of the principle will be performed later with reference to FIG. 6. In one embodiment of the present disclosure, by using an echo path identification algorithm such as the normalized least mean square (NLMS) algorithm, we can obtain the TF from the speaker (horn) spk to the internal microphone (i.e., pinna identification from ERP to EEP) and the TF from the external microphone to the internal microphone (i.e., shadowing identification from far field to EEP).

[0021] Pinna Identification (ERP to EEP) The pinna identification (ERP to EEP) of one embodiment of the present disclosure will be described below with reference to FIG. 2. FIG. 2 shows a schematic diagram of an ANC feedback loop. For convenience of understanding, the system model of pinna identification in the present disclosure will be described by taking the ANC feedback loop in FIG. 2 as an example. As shown in FIG. 2, for convenience of explanation, for example, the position of the speaker (horn) spk is defined as the ERP, the position of the error microphone is defined as the EEP, and the HRTF from the ERP to the EEP is defined as the EEP. 0The controller can be implemented as an AEC system. For a more intuitive understanding, the AEC system shown in Fig. 6 is taken as an example for explanation. For example, the controller can implement an NLMS-based adaptive algorithm.

[0022] To account for individual differences (HRTFs) between the pinna (speaker (spk) in ERP) and the ear canal (error microphone (mic) in EEP) that affect spatial fidelity, we used the feedback loop in Figure 2 to generate an HRTF compensation curve (i.e., H 0 ) must be applied separately.

[0023] Next, the process of pinna identification will be described in detail.

[0024] First, the HRTF from the ERP to the EEP (H 0 ) is obtained. This process can be implemented in two ways. One method involves capturing an arbitrary reference audio signal from an earphone spk, recording the signal by an error microphone, and then converting the signal from the time domain (TD) to FD by a Fast Fourier Transform (FFT). Another method involves using an AEC adaptation loop to obtain an adaptive HRTF (H 0 6 illustrates an adaptive AEC using NLMS for ease of understanding. Those skilled in the art can appreciate that the present disclosure can also employ AEC adaptation loops using other adaptation algorithms (such as RLS and VLMS).

[0025] Next, H 0 HRTF compensation curve by curve fitting

number

[0026] Finally, the audio signal of the FD is subjected to the HRTF compensation curve before being played through the speaker spk.

number

[0027] 3 and 4 are schematic diagrams of left-ear and right-ear HRTFs of left-ear pinna identification and right-ear pinna identification, respectively, obtained by earless measurement, measurement of different users, and artificial head measurement, respectively, in a method according to an embodiment of the present disclosure. In the earless measurement, a sound-absorbing foam is placed on the top of the ear shield of the earphone. It can be seen from FIG. 3 and FIG. 4 that the method for HRTF adaptation of the present disclosure can obtain respective HRTFs (i.e., different frequency response curves in the figure) based on earless conditions, different users, and artificial heads, so that personalized HRTF measurements of different test subjects can be implemented.

[0028] Shadowing discrimination (from far field to EEP) The system model of shadowing identification may be the same as the feedforward loop designed by ANC. Figure 5 shows a schematic diagram of an ANC feedforward loop. The system model of shadowing identification in the present disclosure takes the ANC feedforward loop in Figure 5 as an example for convenience of understanding.

[0029] Considering the binaural feedforward ANC combination, the different HRTFs between left and right ERP / EEP represent head shadowing effects that are adaptively compensated separately according to 3D audio.

[0030] Take the mono feedforward ANC shown in Figure 5 as an example. The far-field HRTF from the noise source to the reference microphone (ERP) and the near-field HRTF from the ERP to the error microphone (EEP) are shown in Figure 5. The reference microphone and the error microphone usually have the same characteristics.

[0031] Figure 5 shows the various components and signal transfer paths of a mono feedforward ANC. A reference microphone 2, located outside the earphone 1, is configured to measure the far-field HRTF. An error microphone 3, located inside the earphone 1, is configured to measure the near-field HRTF. Noise 4 entering the system is filtered by the ear shield of the earphone into a signal 5. The signal 6 reproduced by the earphone speaker is preferably the inverse signal of signal 5. P(z) in the figure represents the far-field HRTF from the noise source to the reference microphone (ERP). N(z) represents the low-pass characteristic of the ear shield of the earphone with passive isolation. H 0 represents the near-field HRTF from the earphone speaker (approximately ERP) to the error microphone (EEP). The controller may be implemented as an AEC system, for example, the AEC system of FIG. 6. Similarly, one skilled in the art may understand that FIG. 6 only shows an adaptive AEC using NLMS. The present disclosure may also use an AEC adaptation loop that uses other adaptation algorithms (such as RLS and VLMS).

[0032] Assuming that the noise source captured by the reference microphone 2 is regarded as the reference signal X(Z) and the signal captured by the error microphone 3 is regarded as the input signal Y(Z), the echo cancellation transfer function H(z) is calculated by N(z) (e.g., a low-pass filter of the FD) and H 0 It is ultimately presumed that this is related to

[0033] According to one aspect of the present disclosure, a priori estimation can be incorporated into the ANC feed-forward loop to obtain better and more stable performance based on the measurement results. In practical operation, the H obtained through a low-pass filter (e.g., cutoff frequency is 3 kHz) and a feedback loop is 0 is multiplied by the reference microphone signal X(Z) of the FD in the NLMS AEC system.

[0034] In the solution of the present disclosure, after pinna identification and shadowing identification are performed on the system, the obtained adaptive HRTF can be applied to the ANC earphone to achieve accurate and personalized HRTF measurement and adaptation during the use of the ANC earphone. Furthermore, a hybrid (feedback + feedforward) adaptive ANC earphone design can be provided to adapt to different adaptation conditions.

[0035] System Compensation For example, for 3D virtual surround earphones, the final reproduction of the customer's HRTF requires an inverse mapping to measure and map the near-field TFs and imperfect directional shadowing features to the 360-degree model. This process can be modeled as a sparsity problem in the field of statistical analysis. For example, a reference head model collects a large number of near-field and far-field measurements and is used to train a deep neural network (DNN). The data is collected in the form of impulse responses placed around the space and with various degrees and distances.

[0036] Then, during the calculation, the measured shadowing function and pinna response can be used as input to generate a 360-degree HRTF rendering matrix to achieve the system compensation effect.

[0037] In binaural hearing modeling research, HRTFs can generally be divided into two free-field spatial characteristics according to the distance from the sound source to the head center: far-field (e.g., distance greater than 1.0 m) and near-field (e.g., distance less than 1.0 m). The method of determining the source of a free-field sound mainly relies on three acoustic cues: (a) interaural time difference (ITD), (b) interaural intensity difference (ILD), and (c) acoustic filtering, i.e., spectral cues derived from the shape of the human ear, head, and body. The near-field HRTFs depend on the human body structure, especially the outer ear structure, which consists of the pinna, ear canal, and eardrum.

[0038] 6 and 7 respectively show a schematic diagram of the acoustic echo cancellation system implemented in FD and the adaptation process of H(Z) implemented in echo cancellation system. Those skilled in the art can understand that Fig. 6 and Fig. 7 are intended to help understand the technology of the present disclosure, rather than narrowly limiting the technology of the present disclosure. The following will be further described with reference to Fig. 6 and Fig. 7.

[0039] FD has become the first choice for AEC because it can implement high-order adaptive filters H(z) with high convergence speed and moderate computational complexity. Two basic modules of NLMS AEC filtering and adaptation are shown in Figures 6 and 7. Figures 6 and 7 show the case of loudspeaker-ambient-microphone (LEM) system. Those skilled in the art can understand that Figures 6 and 7 are intended to explain the basic principles through examples, without specific limitations.

[0040] In FD, AEC is usually implemented using fast convolution / correlation techniques. The cross-correlation between the error signal e(i) in TD and the reference signal x(i) is multiplied by X in FD's E(z). * (z) multiplied by (X * (z) is the conjugate of X(z). Figure 7 shows the reference signal

number

[0041] AEC versions such as those shown in Figures 6 and 7 can only control the linear part of the LEM system and typically use an additional residual echo suppression (RES) to further reduce the echo and keep it within the (linear) AEC error e(i). However, it is well known that the RES characterizes a non-linear signal processing stage and has inherent drawbacks. It can result in acoustic artifacts called musical tones, which must be avoided.

[0042] The following is a brief description of some basic mathematical principles of NLMS AEC. Those skilled in the art can understand that the following description is not a specific limitation, but is only intended to aid in understanding the basic principles of NLMS AEC.

[0043] Based on the Wiener-Khintchine and Parseval theorems, it is as follows: Φ xx (z)=X(z) X * (z)

[0044] The adaptation of the echo canceller H(z) is implemented, for example, as follows.

number

[0045] Optimal step size μ opt (z) is derived based on the relationship between E(z) and X(z), which is then fine-tuned through simulation, analysis, and practice. opt Increasing (z) will result in quick convergence, but may cause instability. opt Making (z) smaller will result in slower convergence, but this may not be suitable for practical applications.

[0046] The present disclosure further provides a system including a memory and a processor, the memory configured to store computer readable instructions, and the processor configured to perform system identification upon execution of the computer readable instructions, the system identification including pinna identification and shadowing identification.

[0047] The description of the embodiments has been given for purposes of illustration and description. Suitable modifications and variations of the embodiments may be implemented or practically obtained in light of the above description. For example, unless otherwise specified, one or more of the described methods may be performed by a suitable device and / or system combination. The method may be performed in a manner that executes stored instructions using one or more logic devices (e.g., processors) in combination with one or more additional hardware elements (storage, memory, circuitry, hardware network interfaces, etc.). The method and related operations may also be performed in various orders, in parallel, and / or simultaneously other than the order described in this application. The system is exemplary in nature and may include additional elements and / or omit elements. The inventive subject matter of the present disclosure includes all novel and non-obvious combinations of the various method and system configurations and other features, functions, and / or properties disclosed.

[0048] As used in this application, elements or steps preceded by the word "one / a" in the singular are to be understood as not excluding a plurality of such elements or steps, unless the exclusion is expressly stated. Furthermore, references to "one embodiment" or "one example" of the present disclosure are not intended to be interpreted as excluding the existence of additional embodiments that also incorporate the recited features.

Claims

1. A method for head-related transfer function (HRTF) adaptation, the method comprising: performing system identification, the system identification including pinna identification and shadowing identification; The shadowing identification, inputting the audio signal received from the reference microphone and the audio signal received from the error microphone into an adaptation controller to obtain an adaptation HRTF; Including, The method of claim 1, wherein the reference microphone is outside the earphone to measure the far-field HRTF from a noise source to a reference point on the ear, and the error microphone is inside the earphone to measure the near-field HRTF from the reference point on the ear to an ear entrance point.

2. The method of claim 1 , further comprising: performing system compensation based on an adaptive HRTF derived from the system identification.

3. The ear identification, Obtaining an adaptive HRTF from an ear reference point (ERP) to an ear entrance point (EEP); Performing curve fitting based on the adaptive HRTF to obtain a compensation curve of the HRTF; multiplying a frequency domain (FD) audio signal by said compensation curve; The method of claim 1 , comprising:

4. The method of claim 3 , wherein the compensation curve is an inverse function of the HRTF.

5. The method of claim 1 , wherein the pinna identification is implemented by a feedback loop of an active noise canceller (ANC) with an adaptive controller.

6. The method of claim 1 , wherein the shadowing identification is implemented by a feedforward loop of an ANC with an adaptive controller.

7. The method of claim 1 , further comprising: performing system compensation with a hybrid output of the adaptive HRTFs derived from the pinna identification and the adaptive HRTFs derived from the shadowing identification.

8. A system for head-related transfer function (HRTF) adaptation, comprising: a memory configured to store computer readable instructions; a processor configured to perform system identification upon execution of the computer readable instructions, the system identification including pinna identification and shadowing identification; Equipped with The shadowing identification, inputting the audio signal received from the reference microphone and the audio signal received from the error microphone into an adaptation controller to obtain an adaptation HRTF; Including, The reference microphone is outside the earphone to measure the far-field HRTF from a noise source to a reference point on the ear, and the error microphone is inside the earphone to measure the near-field HRTF from the reference point on the ear to an ear entrance point.

9. The system of claim 8 , wherein the processor is further configured to perform system compensation based on an adaptive HRTF derived from the system identification.

10. The processor, Obtaining an adaptive HRTF from an ear reference point (ERP) to an ear entrance point (EEP); Performing curve fitting based on the adaptive HRTF to obtain a compensation curve of the HRTF; multiplying a frequency domain (FD) audio signal by said compensation curve; The system of claim 8 , further configured to perform the pinna identification comprising:

11. The system of claim 10 , wherein the compensation curve is an inverse function of the HRTF.

12. The system of claim 8 , wherein the processor is further configured to implement the pinna identification by an active noise canceller (ANC) feedback loop comprising an adaptive controller.

13. The system of claim 8 , wherein the processor is further configured to implement the shadowing identification by a feed-forward loop of an ANC that comprises an adaptive controller.

14. The system of claim 8 , wherein the processor is further configured to perform system compensation with a hybrid output of the adaptive HRTFs derived from the pinna identification and the adaptive HRTFs derived from the shadowing identification.

15. A computer readable medium comprising instructions for carrying out the method of any one of claims 1 to 7.

Images