Personalized Head-Related Transfer Function Prediction Method and Apparatus Based on Sparse Measurements

Abstract

This invention provides a method and apparatus for predicting personalized head-related transfer functions (HRTFs) based on sparse measurements. The method includes: first, determining the HRTF amplitude spectrum of sparse measurements based on head-related impulse responses measured in several directions; then, processing the HRTF amplitude spectrum of the sparse measurements using an acoustic feature extraction network to obtain corresponding acoustic features; using a dimension augmentation network to augment the dimension of the target direction to obtain an augmented target direction vector; and finally, using a spectral prediction neural network to process the acoustic features and the augmented target direction vector to predict the personalized HRTF amplitude spectrum of the target. The apparatus includes: an amplitude spectrum acquisition module, an acoustic feature acquisition module, a direction vector acquisition module, and a personalized target amplitude spectrum acquisition module. Thus, given a small amount of HRTF amplitude spectrum from a new subject and the azimuth and pitch angles of the target location, a personalized target HRTF amplitude spectrum can be generated through three neural network modules, achieving prediction of personalized HRTF amplitude spectra across the entire space.

Description

Technical Field

[0001] This specification relates to the field of spatial audio technology, and more particularly to a personalized head-related transfer function prediction method and apparatus based on sparse measurements. Background Technology

[0002] Head Related Transfer Function (HRTF) plays a crucial role in spatial audio technology. HRTF characterizes the process by which sound waves travel from the sound source through physiological structures such as the head, auricle, and trunk to the eardrum. Differences in physiological parameters among individuals result in varying HRTFs, and using personalized HRTFs in spatial audio playback can effectively improve playback quality. However, personalized HRTFs typically require measurement in an anechoic environment using specialized equipment, which is very time-consuming, hindering their practical application. Currently, personalized HRTFs are widely obtained by selecting from existing HRTF databases based on the similarity of physiological parameters or by synthesizing HRTF amplitude spectra using physiological parameters. However, due to the complexity of HRTF generation, existing personalized HRTF prediction methods based on physiological parameters struggle to accurately characterize the physical processes between sound source propagation from different directions and various physiological parameters of the human body. Therefore, some methods attempt to interpolate or predict the entire spatial personalized HRTF using HRTF amplitude spectra measured in different directions.

[0003] However, currently, such personalized HRTF generation methods require HRTF measurements in multiple directions, resulting in time-consuming and memory-intensive personalized HRTF generation systems, which limits their application in spatial audio playback. Summary of the Invention

[0004] This invention describes a personalized head-related transfer function prediction method and apparatus based on sparse measurements, which can solve the above-mentioned technical problems.

[0005] According to the first aspect, a personalized head-related transfer function (HRTF) prediction method based on sparse measurements is provided. The method includes: determining the HRTF amplitude spectrum of sparse measurements based on head-related impulse responses measured in several directions; processing the HRTF amplitude spectrum of the sparse measurements using an acoustic feature extraction network to obtain corresponding acoustic features; augmenting the dimension of the target direction using a dimension augmentation network to obtain an augmented target direction vector; and processing the acoustic features and the augmented target direction vector using a spectral prediction neural network to predict the personalized HRTF amplitude spectrum of the target.

[0006] In one implementation, determining the head-related transfer function (HRTF) amplitude spectrum of sparse measurements based on head-related impulse responses measured in several directions includes: performing Fourier transform and logarithmic domain transform on the head-related impulse responses to obtain the HRTF amplitude spectrum of the sparse measurements.

[0007] In one implementation, the step of augmenting the dimension of the target direction using a dimension augmentation network to obtain an augmented target direction vector includes: converting the azimuth and pitch angles of the target direction into three-dimensional coordinates on a unit sphere in a Cartesian coordinate system; and inputting the three-dimensional coordinates into the dimension augmentation network to obtain the augmented target direction vector.

[0008] In one embodiment, the step of using a spectral prediction neural network to process the acoustic features and the augmented target direction vector to predict the target HRTF amplitude spectrum includes: concatenating the acoustic features and the target direction vector and inputting them into the spectral prediction network, and outputting the target personalized HRTF amplitude spectrum.

[0009] In one embodiment, the method further includes: given the HRTF amplitude spectrum of the sparse measurement and the azimuth and elevation angles of the target direction, the personalized HRTF amplitude spectrum of the target can be predicted by the acoustic feature extraction network, the dimension augmentation network and the spectral prediction neural network.

[0010] According to a second aspect, a personalized head-related transfer function prediction device based on sparse measurements is provided. The device includes:

[0011] The amplitude spectrum acquisition module is configured to determine the amplitude spectrum of the head correlation transfer function (HRTF) of sparse measurements based on the head correlation impulse response measured in several directions.

[0012] The acoustic feature acquisition module is configured to process the HRTF amplitude spectrum of the sparse measurement using an acoustic feature extraction network to obtain the corresponding acoustic features.

[0013] The direction vector acquisition module is configured to use a dimension augmentation network to augment the dimension of the target direction to obtain an augmented target direction vector.

[0014] The target personalized amplitude spectrum acquisition module is configured to use a spectral prediction neural network to process the acoustic features and the augmented target direction vector to predict the target personalized HRTF amplitude spectrum.

[0015] In one implementation, the amplitude spectrum acquisition module is specifically configured as follows:

[0016] The HRTF amplitude spectrum of the sparse measurement is obtained by performing Fourier transform and logarithmic domain transform on the head-related impulse response.

[0017] In one implementation, the acoustic extraction network is implemented as a convolutional neural network.

[0018] In one specific implementation, the convolutional neural network includes: several convolutional layers, several max pooling layers, several activation layers, and a fully connected output layer.

[0019] In one implementation, the direction vector acquisition module is used to convert the azimuth and pitch angles of the target direction into three-dimensional coordinates on a unit sphere in a Cartesian coordinate system; the three-dimensional coordinates are input into the dimension augmentation network to obtain the augmented target direction vector.

[0020] In one implementation, the target amplitude spectrum acquisition module is specifically configured to: concatenate the acoustic features and the target direction vector and input them into the spectrum prediction network to output the target personalized HRTF amplitude spectrum.

[0021] In the methods and apparatus provided in the embodiments of this specification, data from an existing HRTF database are used to establish a mapping model between the sparsely measured HRTF amplitude spectrum, the target orientation, and the target-personalized HRTF amplitude spectrum through deep learning methods. This allows the target-personalized HRTF amplitude spectrum to be generated by three neural network modules, given a small number of HRTF amplitude spectra of a new subject and the azimuth and elevation angles of the target orientation. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This diagram illustrates the implementation architecture of the personalized head-related transfer function prediction scheme based on sparse measurements provided in the embodiments of this specification.

[0024] Figure 2 This diagram illustrates a flowchart of a personalized head-related transfer function prediction method based on sparse measurements provided in an embodiment of this specification.

[0025] Figure 3 This diagram illustrates the structure of a personalized head-related transfer function prediction device based on sparse measurements, as provided in an embodiment of this specification. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in the embodiments of this application will be described below with reference to the accompanying drawings.

[0027] First, it should be noted that in the description of the embodiments of this application, the words "exemplary," "for example," or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design that is described as "exemplary," "for example," or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of the words "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a specific manner.

[0028] In the description of the embodiments of this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, B existing alone, and A and B existing simultaneously. Furthermore, unless otherwise stated, the term "multiple" means two or more.

[0029] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. The terms "comprising," "including," "having," and their variations all mean "including but not limited to," unless otherwise specifically emphasized.

[0030] As mentioned above, audio processing based on head-related transfer functions (HRTFs) is a key technology for spatial audio rendering. A key challenge is generating a fully spatially personalized HRTF in a short time and with minimal memory usage. To address this issue, this application proposes a personalized HRTF prediction method based on sparse measurements. This method uses deep learning to extract acoustic features representing the subject's physiological structure from the HRTF amplitude spectrum of sparse measurements. It then establishes a relationship between these acoustic features and the target personalized HRTF amplitude spectrum, enabling the prediction of the fully spatial personalized HRTF amplitude spectrum given the subject's measured HRTF in a few orientations.

[0031] Figure 1 This diagram illustrates the implementation architecture of the personalized head-related transfer function prediction scheme based on sparse measurements provided in the embodiments of this specification. Figure 1As shown, this implementation architecture involves using an acoustic feature extraction network, a dimension augmentation network, and a spectral prediction neural network to predict the personalized HRTF amplitude spectrum across the entire space. Specifically, the sparsely measured HRTF amplitude spectrum is input into the acoustic feature extraction network to obtain the corresponding acoustic features, and the three-dimensional coordinates of the target direction (or target orientation) are input into the dimension augmentation network to obtain the augmented target direction vector. The outputs of these two networks are then used as the input to the spectral neural network to obtain the personalized HRTF amplitude spectrum. It should be noted that the specific composition of the acoustic feature extraction network is not specifically limited; any network architecture capable of obtaining the corresponding acoustic features from the sparsely measured HRTF amplitude spectrum is acceptable. The number of each layer in the input, hidden, and output layers of the dimension augmentation network is not specifically limited. The same applies to the spectral prediction neural network, which will not be elaborated upon here.

[0032] Figure 2 This document illustrates a flowchart of a personalized head-related transfer function prediction method based on sparse measurements, as provided in an embodiment of this specification. Figure 2 As shown, the method includes the following steps:

[0033] Step S110: Based on the head-related impulse responses measured in several directions, determine the HRTF amplitude spectrum of the sparse measurements. It should be noted that "several" in this text refers to one or more.

[0034] For example, the head-related impulse response (HRIR) of the current subject is measured in several specified directions. Compared to HRTF measurements in more directions, this embodiment only requires measurements in several directions, thus the personalized HRTF generation system is less time-consuming and consumes less memory resources. The head-related impulse response is then transformed into the logarithmic domain after undergoing a Fourier transform to obtain the sparsely measured HRIR, and subsequently, the corresponding HRTF amplitude spectrum is obtained.

[0035] Step S120: The HRTF amplitude spectrum of the sparse measurement is processed using an acoustic feature extraction network to obtain the corresponding acoustic features.

[0036] For example, an acoustic feature extraction network consisting of a convolutional layer, a max pooling layer, an activation layer, and a fully connected output layer is constructed to extract a one-dimensional acoustic feature vector from the sparsely measured HRTF amplitude spectrum to describe the physiological structure of the current subject.

[0037] Step S130: Use a dimension augmentation network to augment the dimension of the target direction to obtain the augmented target direction vector.

[0038] For example, the dimension-enhancing network is implemented as a deep neural network consisting of a fully connected input layer, two hidden layers, and an output layer.

[0039] For example, the two-dimensional azimuth and elevation angles of the target orientation are first converted into three-dimensional coordinates on a unit sphere in the Cartesian coordinate system, and then input into a deep neural network. After passing through the input layer nodes, the target orientation vector with increased dimension is finally output, which is a one-dimensional vector.

[0040] It should be understood that there are various ways to measure the direction of a target, such as polar coordinates, Cartesian coordinates, etc., which can be selected as needed. The embodiments in this specification do not limit this.

[0041] Step S140: The acoustic features and the augmented target direction vector are processed using a spectral prediction neural network to predict the target personalized HRTF amplitude spectrum.

[0042] For example, the spectral prediction network includes a fully connected input layer, six hidden layers, and a fully connected output layer.

[0043] For example, acoustic features and dimension-enhanced direction vectors are concatenated and input into a spectral prediction neural network, and finally the output layer outputs the predicted target personalized HRTF amplitude spectrum.

[0044] In summary, in the methods disclosed in the embodiments of this specification, given a small amount of HRTF amplitude spectrum of a new subject and the azimuth and elevation angles of the target orientation, after being mapped by their respective networks, the acoustic features and the target orientation vector after dimensional augmentation are concatenated and then input into the corresponding prediction network to predict the personalized HRTF amplitude spectrum of the target.

[0045] It should be noted that the prediction of the target's personalized HRTF amplitude spectrum mentioned above uses pre-trained acoustic feature extraction networks, dimension augmentation networks, and spectral prediction neural networks. The main difference between training these three neural networks and their usage phase is that during training, the network parameters are adjusted using the error between the prediction results and the ground truth, while during usage, the parameters are directly input into the network.

[0046] This invention uses deep learning to extract acoustic features that characterize the physiological structure of subjects from sparse measurements, and then establishes the relationship between acoustic features and the target personalized HRTF amplitude spectrum, so that given the measured HRTF of a subject in a few directions, it can predict the personalized HRTF amplitude spectrum in the whole space.

[0047] This invention utilizes data from an existing HRTF database and establishes a mapping model between sparsely measured HRTF amplitude spectra, target orientation, and target-specific HRTF amplitude spectra using deep learning methods. This allows for the generation of target-specific HRTF amplitude spectra from a small number of HRTF amplitude spectra of a new subject and the azimuth and elevation angles of the target orientation via three neural network modules.

[0048] Corresponding to the method provided by the present invention, the present invention also provides an apparatus. Figure 3 This diagram illustrates the structure of a personalized head-related transfer function prediction device based on sparse measurements, as provided in an embodiment of this specification. Figure 3 As shown, the device 300 includes:

[0049] The amplitude spectrum acquisition module 310 is configured to determine the amplitude spectrum of the head correlation transfer function (HRTF) of sparse measurements based on the head correlation impulse response measured in several directions.

[0050] The acoustic feature acquisition module 320 is configured to process the HRTF amplitude spectrum of the sparse measurement using an acoustic feature extraction network to obtain the corresponding acoustic features.

[0051] The direction vector acquisition module 330 is configured to use a dimension augmentation network to augment the dimension of the target direction to obtain an augmented target direction vector.

[0052] The target personalized amplitude spectrum acquisition module 340 is configured to use a spectral prediction neural network to process acoustic features and augment the target direction vector to predict the target personalized HRTF amplitude spectrum.

[0053] In one implementation, the amplitude spectrum acquisition module 310 is specifically used to obtain the HRTF amplitude spectrum of sparse measurements by performing Fourier transform and logarithmic domain transform on the head-related impulse response.

[0054] In one implementation, the acoustic extraction network is implemented as a convolutional neural network.

[0055] In one specific implementation, the convolutional neural network includes: several convolutional layers, several max pooling layers, several activation layers, and a fully connected output layer.

[0056] In one implementation, the direction vector acquisition module 330 is used to convert the two-dimensional azimuth and pitch angles of the target direction into three-dimensional coordinates on a unit sphere in a Cartesian coordinate system; the three-dimensional coordinates are input into the dimension augmentation network to obtain the augmented target direction vector.

[0057] In one implementation, the target amplitude spectrum acquisition module 340 is used to concatenate the acoustic features and the target direction vector and input them into the spectrum prediction network to output the target personalized HRTF amplitude spectrum.

[0058] It needs to be explained that, for Figure 3 For a description of the apparatus, see also the description of the aforementioned method.

[0059] According to another embodiment, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed in a computer, causes the computer to perform a combination Figure 2 The method described.

[0060] According to another embodiment, a computing device is also provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements a combination... Figure 2 The methods described herein. Those skilled in the art will recognize that, in one or more of the examples above, the functions described in this invention can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0061] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.

Claims

1. A sparse measurement based personalized head-related transfer function prediction method, characterized in that, The method comprises: determining a sparse measured head-related transfer function (HRTF) magnitude spectrum based on head-related impulse responses measured in a plurality of directions; processing the sparse measured HRTF magnitude spectrum using an acoustic feature extraction network to obtain corresponding acoustic features; augmenting dimensions of a target direction using a dimension augmentation network to obtain an augmented target direction vector; the augmenting dimensions of the target direction using the deep neural network to obtain the augmented target direction vector comprises: converting an azimuth angle and an elevation angle of the target direction into three-dimensional coordinates on a unit sphere in a Cartesian coordinate system; inputting the three-dimensional coordinates into the dimension augmentation network to obtain the augmented target direction vector; processing the acoustic features and the augmented target direction vector using a spectral prediction neural network to predict a target personalized HRTF magnitude spectrum.

2. The method of claim 1, wherein, The determining a sparse measured head-related transfer function (HRTF) magnitude spectrum based on head-related impulse responses measured in a plurality of directions comprises: performing Fourier transform and logarithmic domain transform on the head-related impulse responses to obtain the sparse measured HRTF magnitude spectrum.

3. The method of claim 1, wherein, The acoustic extraction network is implemented as a convolutional neural network.

4. The method of claim 3, wherein, The convolutional neural network comprises a plurality of convolutional layers, a plurality of maximum value pooling layers, a plurality of activation layers, and a fully connected output layer.

5. The method of claim 1, wherein, The processing the acoustic features and the augmented target direction vector using the spectral prediction neural network to predict a target personalized HRTF magnitude spectrum comprises: concatenating the acoustic features and the target direction vector and inputting the result into the spectral prediction network to output the target personalized HRTF magnitude spectrum.

6. An apparatus for personalized head-related transfer function prediction based on sparse measurements, characterized by The apparatus comprises: a magnitude spectrum acquisition module configured to determine a sparse measured head-related transfer function (HRTF) magnitude spectrum based on head-related impulse responses measured in a plurality of directions; an acoustic feature acquisition module configured to process the sparse measured HRTF magnitude spectrum using an acoustic feature extraction network to obtain corresponding acoustic features; a direction vector acquisition module configured to augment dimensions of a target direction using a dimension augmentation network to obtain an augmented target direction vector; the direction vector acquisition module is specifically configured to: convert an azimuth angle and an elevation angle of the target direction into three-dimensional coordinates on a unit sphere in a Cartesian coordinate system; input the three-dimensional coordinates into the dimension augmentation network to obtain the augmented target direction vector; a target personalized magnitude spectrum acquisition module configured to process the acoustic features and the augmented target direction vector using a spectral prediction neural network to predict a target personalized HRTF magnitude spectrum.

7. The apparatus of claim 6, wherein, The magnitude spectrum acquisition module is specifically configured to: perform Fourier transform and logarithmic domain transform on the head-related impulse responses to obtain the sparse measured HRTF magnitude spectrum.

8. The apparatus of claim 6, wherein, The acoustic extraction network is implemented as a convolutional neural network.

Images