HRTF reconstruction device and method based on automatic encoder and spherical harmonic expansion
By predicting the HRTF spherical unfolding coefficient from ear depth pictures based on variational automatic encoder and spherical harmony unfolding method, the problems of complex and large error in HRTF measurement in the prior art are solved, and the accurate reconstruction of personalized HRTF and the improvement of audio quality are achieved.
Patent Information
- Application Number
- CN202411668122.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-21
- Publication Date
- 2025-02-28
AI Technical Summary
When obtaining individual HRTFs in the prior art, the measurement process is complicated and difficult to carry out on a large scale, and there are large errors in the methods based on human parameters, resulting in a large deviation from the calculated HRTFs from the actual situation.
Using a method based on variational automatic encoder and spherical harmony expansion, the HRTF spherical harmony expansion coefficient is predicted from ear depth pictures, and a personalized HRTF is obtained through the reconstruction module.
HRTF prediction through ear depth pictures avoids errors in measuring human parameters, improves the accuracy and audio quality of HRTF, and simplifies the measurement process.
Smart Images
Figure CN119545285A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of acoustic technology and proposes a HRTF reconstruction device and method based on an automatic encoder and spherical harmonic expansion. Background Art
[0002] The Head Related Transfer Function (HRTF) is a frequency domain acoustic transfer function from the sound source to the ears in a free field. It characterizes the process of the direct sound waves emitted by the sound source reaching the human ear after being scattered and reflected by physiological structures such as the head, auricle, and trunk. The binaural sound pressure contains various sound source localization factors such as ITD and ILD. The auditory system uses these factors to locate the sound source. Different individuals have different HRTFs due to differences in physiological parameters. Using individual HRTF processing in the process of audio playback using headphones will provide a better sense of immersion.
[0003] Usually, obtaining individual HRTF requires measurement in a full anechoic room or a semi-anechoic room, which often requires multiple speakers and a person sitting still in a fixed position. The measurement process lasts about half an hour, so HRTF measurement is difficult to carry out on a large scale.
[0004] In recent years, data resources and computer computing have continued to develop, and neural network-based methods have been widely used in the field of HRTF personalization. People have proposed using human physiological parameters or ear images to map HRTF. Only by collecting physiological parameters or ear images, personalized HRTF can be obtained, which simplifies the entire process. However, this method has the problem of difficult human parameter measurement and large errors, resulting in large deviations between the calculated HRTF and the actual situation, as well as differences between the predicted HRTF and the actual HRTF peak and valley frequencies, which cannot accurately reflect the individual's auditory characteristics, thus affecting the quality of the rendered audio. Summary of the invention
[0005] In order to solve the defects existing in the above-mentioned prior art, the present invention provides a HRTF reconstruction device and method based on variational autoencoder and spherical harmonic expansion, which can predict HRTF spherical harmonic expansion coefficients from depth images of the ear, and further reconstruct personalized HRTF.
[0006] The technical solution adopted by the device of the present invention is:
[0007] A HRTF reconstruction device based on an autoencoder and spherical harmonic expansion, comprising: a variational autoencoder for obtaining a coded representation of auricle features based on an ear image; a spherical harmonic coefficient expansion module for expanding HRTF into spherical harmonic coefficients using spherical harmonic functions; a prediction module for predicting HRTF spherical harmonic expansion coefficients based on the image coded representation output by the variational autoencoder; a reconstruction module for reconstructing the spherical harmonic expansion coefficients predicted by the prediction module into HRTF amplitudes, and then personalizing the HRTF phase according to the listener's head width, thereby obtaining a complete HRTF.
[0008] Furthermore, the variational autoencoder includes an encoder module and a decoder module, the encoder module is a probability model that maps input samples to a probability distribution, and is used to derive a compact coding representation of an ear depth image; the decoder module is used to reconstruct the ear image from the compact coding representation of the ear depth image.
[0009] Furthermore, the prediction module is composed of a neural network, which maps each frequency to be processed into an embedded vector, splices the encoded frequency information with the ear picture encoding information, and the spliced code is input into multiple layers of a one-dimensional convolutional neural network to gradually extract and analyze important features in the data; the prediction module uses the mean square error between the true spherical harmonic expansion coefficients and the predicted spherical harmonic expansion coefficients as the training loss function.
[0010] The present invention also provides a method for HRTF reconstruction based on an automatic encoder and spherical harmonic expansion, the method comprising the following steps:
[0011] (1) Using the 3D head mesh in the HRTF database, the ear depth image corresponding to each head mesh is obtained; the HRTF of each individual in the HRTF database is expanded using spherical harmonic functions to obtain the corresponding spherical harmonic expansion coefficients, and the ear depth image and the corresponding spherical harmonic expansion coefficients are combined to construct a data set;
[0012] (2) constructing the variational autoencoder, wherein the variational autoencoder includes an encoder module and a decoder module;
[0013] (3) using the ear depth image obtained in step (1) to train the variational autoencoder, and using the encoder module to obtain a compact encoding representation of the ear depth image;
[0014] (4) Construct a neural network model to predict HRTF spherical harmonic expansion coefficients based on image coding;
[0015] (5) using the compact coding representation of the ear depth image obtained in step (3) and the HRTF spherical harmonic expansion coefficients obtained in the corresponding step (1), the neural network model in step (4) is trained to obtain trained weights;
[0016] (6) Inputting the ear depth image to be tested into the encoder module to obtain a compact coding representation of the ear depth image, and then inputting the compact coding representation into the model trained in step (5) to obtain predicted spherical harmonic expansion coefficients, and reconstructing the predicted HRTF through inverse spherical harmonic transformation;
[0017] (7) The predicted HRTF is interpolated and supplemented with phase information to obtain a complete HRTF spectrum, which is then converted into a head-related impulse response (HRIR) through inverse Fourier transform.
[0018] Furthermore, in step (2), the encoder module is a probability model that maps input samples to probability distributions. The encoder module maps the input ear depth image to a probability distribution p(z|x), that is, the distribution of the latent variable z given the input data x, which is parameterized by its mean μ and logarithmic variance σ, as the output of the encoder module.
[0019] Furthermore, in step (3), the method for training the variational autoencoder is as follows: the ear depth image obtained in step (1) is input into the encoder module to obtain a compact representation of the encoded image, and then the compact encoded representation of the image is input into the decoder module to reconstruct the original input, and then the output image of the decoder module is compared with the original input image, and the difference between the two is calculated as the cost function, and the loss of image reconstruction is minimized through optimization training.
[0020] Furthermore, in step (5), the training method of the deep neural network model is as follows: first, the frequency is mapped into an embedding vector, and then the frequency coding information and the compact coding representation information of the ear image obtained in step (3) are spliced; the spliced coding information is input into the convolutional neural network to predict the HRTF spherical harmonic expansion coefficients of the ear corresponding to the current frequency; the predicted spherical harmonic coefficients are compared with the actual spherical harmonic coefficients, the mean square error is calculated and fed to the model for optimization training.
[0021] Furthermore, in step (7), the specific method of supplementing the phase information is as follows: first, the head width data of the target listener is obtained; then, from a database containing anthropometric features, an individual whose head width is closest to the target listener is selected, and a phase spectrum is extracted from the HRTF of the individual; and then the extracted phase spectrum and the predicted HRTF amplitude spectrum are synthesized into a complex HRTF spectrum.
[0022] Compared with the current prior art, the advantages of the present invention are:
[0023] (1) This scheme uses ear depth images to predict HRTF. Compared with physiological parameters, ear depth images can provide more information and avoid errors in the measurement of human body parameters.
[0024] (2) This scheme uses spherical harmonic functions to reduce the dimension of HRTF. After dimensionality reduction, the model training is easier, and in theory, the predicted spherical harmonic expansion coefficients can be used to obtain the HRTF of the entire space. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1 It is a schematic diagram of the structure of the variational autoencoder of the present invention.
[0026] Figure 2 It is the real part graph of the spherical harmonic basis function of the present invention.
[0027] Figure 3 It is a structural diagram of the convolutional neural network for predicting HRTF spherical harmonic expansion coefficients of the present invention.
[0028] Figure 4 It is an ear depth picture collected in the database of the embodiment of the present invention.
[0029] Figure 5 These are (a) the original picture and (b) the encoded and decoded output picture in the embodiment of the present invention. DETAILED DESCRIPTION
[0030] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below in conjunction with the implementation modes and the accompanying drawings.
[0031] The HRTF prediction device proposed in the present invention includes the following four modules:
[0032] Variational Autoencoders (VAE), which are used to derive the encoded representation z of 2D ear features (such as images, depth maps, or edge maps) ear ;
[0033] Spherical harmonic coefficient expansion module, used to expand HRTF into spherical harmonic coefficients using spherical harmonic functions;
[0034] The prediction module is used to represent z according to the picture encoding ear Predict HRTF spherical harmonic expansion coefficients using deep neural networks (DNN);
[0035] A reconstruction module is used to reconstruct the predicted spherical harmonic expansion coefficients into HRTF amplitudes, and then personalize the HRTF phase according to the listener's head width to obtain the complete HRTF.
[0036] In this embodiment, the variational autoencoder is used to encode and decode the ear depth image. The network structure of the encoder is as follows: Figure 1 As shown in Figure 1, it includes an encoder module and a decoder module, where the encoder module is a probabilistic model that maps input samples to a probability distribution. This encoder module obtains an approximate probability p(z|x), that is, the distribution of the latent variable z given the input data x. This distribution is parameterized by its mean μ and logarithmic variance σ, and these two outputs are the outputs of the encoder module. During the training process, a value is extracted from the distribution As a compact representation of the encoded image, it is input into the decoder module, which attempts to reconstruct the original input. Then the output of the decoder module is compared with the original input image, and the difference between the two is calculated as the cost function. After optimization training, the loss of image reconstruction is minimized.
[0037] The encoder module of the variational autoencoder VAE is used to derive a compact representation of 2D auricle features (such as pictures, depth maps or edge maps), while the decoder module is used to reconstruct the ear picture from the compact representation of the auricle features. The reconstructed image shows that the autoencoder can effectively encode the physical characteristics of the auricle.
[0038] The spherical harmonic coefficient expansion module uses spherical harmonic functions to expand the HRTF at each required frequency, and uses the obtained spherical harmonic expansion coefficients as the dimensionality reduction representation of the HRTF.
[0039] Spherical harmonics (SH) are an orthogonal basis set of spherical coordinates and are widely used in the field of spatial audio. The real spherical harmonic basis functions are defined as
[0040]
[0041] Where θ, φ are the azimuth and elevation angles, is the associated Legendre polynomial, n∈{0,1,...} and m∈{-n,...,n} represent the order and degree of the spherical harmonic function respectively. The real part of the first two order spherical harmonic basis functions is as follows Figure 2 shown.
[0042] The real spherical harmonic basis functions form a complete orthogonal basis for any square-integrable function defined in spherical coordinates. This means that any continuous spherical function f(φ,θ) can be expanded into a finite set of spherical harmonic functions
[0043]
[0044] The weight set C nm Called spherical harmonic coefficients, they carry information about how the function f(φ,θ) varies in space.
[0045] Due to the standard orthogonality of spherical harmonics, for any continuous spherical function f(φ,θ), the spherical harmonic coefficients can be calculated as follows
[0046]
[0047] For any discrete grid, the SH coefficient is usually obtained by using The system is given in matrix form as
[0048] f=Yc (4)
[0049] in
[0050]
[0051] c=[C 00 ,C 1-1 ,C 10 ,C 11 ,...C LL ] T
[0052] Y=[y 00 ,y 1-1 ,y 10 ,y 11 ,...y LL ]
[0053]
[0054] where f contains the original amplitude values in S spatial directions, c is the desired spherical harmonic expansion coefficients, and Y contains the values of the spherical harmonic basis functions up to order L in the corresponding source spatial directions.
[0055] Compute the coefficient vector c using least squares estimation
[0056] c=(Y T Y) -1 Y T f (5)
[0057] The coefficient vector c is the spherical harmonic expansion coefficient at this frequency, which can be used to represent the HRTF amplitude response at all positions at a certain frequency.
[0058] The prediction module of this embodiment uses a deep neural network that can predict the HRTF spherical harmonic expansion coefficients from ear image encoding. The model structure is as follows: Figure 3As shown. When predicting the spherical harmonic expansion coefficients at different frequencies, first, determine the range of all frequencies to be processed, map each frequency to an embedding vector, concatenate the encoded frequency information with the ear image encoding information, and the concatenated latent code is input into multiple layers of a one-dimensional convolutional neural network (CNN) for processing. The convolutional neural network gradually extracts and analyzes important features in the data through a series of convolution operations. These layers extract features step by step, capturing the complex relationship between frequency and ear features, thereby more accurately predicting the spherical harmonic expansion coefficients.
[0059] In order to evaluate the performance of the model, the mean square error (MSE) between the true spherical harmonic expansion coefficients and the predicted spherical harmonic expansion coefficients is used as the training loss function. By minimizing the mean square error, the model can continuously adjust its parameters to improve the accuracy of the prediction, making the predicted SH coefficient closer to the actual value.
[0060] The main steps of the HRTF prediction method provided in this embodiment are as follows:
[0061] (1) Obtain a 3D head mesh in the head-related transfer function (HRTF) database, project each ear model at multiple angles to obtain multiple corresponding ear depth images; use spherical harmonics to expand the HRTF of each individual in the HRTF database to obtain the corresponding spherical harmonic expansion coefficients, and combine the ear depth images with the corresponding spherical harmonic expansion coefficients to construct a data set.
[0062] (2) constructing a variational autoencoder (VAE) to derive an encoded representation of 2D auricle features (e.g., depth map);
[0063] (3) using the ear depth image obtained in step (1) to train the autoencoder in step (2), and using the encoder module of the autoencoder to derive a compact encoding representation of the ear depth image;
[0064] (4) Construct a deep neural network model to predict HRTF spherical harmonic expansion coefficients based on image coding.
[0065] (5) Using the compact coding representation obtained in step (3) and the HRTF spherical harmonic expansion coefficients obtained in the corresponding step (1), the deep neural network model in step (4) is trained to obtain the trained weights.
[0066] (6) Input the ear depth image to be tested into the encoder to obtain a compact coding representation of the ear depth image, and then input the compact coding representation and frequency coding of the image into the model trained in step (5) to obtain the predicted spherical harmonic expansion coefficients, and reconstruct the predicted HRTF through inverse spherical harmonic transformation.
[0067] (7) The predicted HRTF is interpolated and supplemented with phase information to obtain a complete HRTF spectrum, which is then converted into a head-related impulse response (HRIR) through inverse Fourier transform.
[0068] The method of the present invention is described in detail below in conjunction with specific example data.
[0069] In step (1), the dataset construction method is as follows: the public HRTF dataset Hutubs database is used, which contains 96 subjects. In addition to acoustic measurements and numerically simulated head-related impulse responses (HRIR), the database also includes anthropometric data and 58 3D head meshes. HRTF is stored in the form of SOFA files at the locations specified by the spatial sampling grid. The 3D head mesh is provided in the widely used ply format, and the anthropometric features are stored in text files.
[0070] The depth images required for the dataset are obtained from the 3D head mesh, and the images are exported mainly using Blender software. In order to obtain more images for training the autoencoder model, each ear model is projected at different angles, with the vertical and horizontal angles ranging from -15° to 15°. Some of the images are as follows: Figure 4 As shown, 49 depth images of such an ear can be obtained. In this embodiment, 2842 depth images of the left ear and the right ear are obtained from the database respectively.
[0071] In the process of preprocessing HRTF, it is necessary to use the formula given above to perform spherical harmonic decomposition on HRTF to obtain the spherical harmonic expansion coefficients of HRTF at each frequency (i.e. coefficient vector c). Finally, the ear depth image and the corresponding HRTF spherical harmonic expansion coefficients are combined to form the data set required for training.
[0072] In step (2), construct Figure 1 The variational autoencoder model shown includes an encoder module and a decoder module, where the encoder module is a probabilistic model that maps input samples to a probability distribution, and the decoder module attempts to reconstruct the original input.
[0073] In step (3), the training method of the variational autoencoder is as follows: the depth image obtained in step (1) is input into the encoder module to obtain a compact representation of the encoded image, and then the compact encoded representation of the image is input into the decoder module to reconstruct the original input, and then the output image of the decoder module is compared with the original input image, and the difference between the two is calculated as the cost function, and the loss of image reconstruction is minimized through optimization training. In this embodiment, an autoencoder model is trained for each left ear and right ear. Finally, the trained autoencoder model is used to encode each ear image in the (0°, 0°) direction to obtain a compact encoded representation z of the depth of each individual ear. ear , used for subsequent model training.
[0074] After obtaining the ear image code and the spherical harmonic expansion coefficients of the HRTF, a deep learning method is needed to map the two. In step (5), the training method of the neural network model is as follows:
[0075] Since the spherical harmonic expansion coefficients of HRTF are frequency-dependent, first, the range of all frequencies to be processed is determined, each frequency is mapped to an embedding vector, and the encoded frequency information is concatenated with the ear image encoding information. The concatenated code is input into multiple layers of a one-dimensional convolutional neural network (CNN) for processing. These layers extract features step by step to capture the complex relationship between frequency and ear features. Finally, the spherical harmonic expansion coefficients of the HRTF corresponding to the ear at that frequency are predicted, the predicted spherical harmonic coefficients are compared with the actual spherical harmonic coefficients, the loss is calculated, and optimization training is performed.
[0076] In step (7), the specific method of generating HRIR is as follows:
[0077] The predicted HRTF amplitude is upsampled to match the frequency resolution of the sound used for spatial audio. The upsampled HRTF amplitude does not include information about interaural time differences (ITD), and phase information needs to be supplemented to obtain a complete HRTF. Studies have shown that among anthropometric features, head width is highly correlated with ITD.
[0078] In order to select HRTF, first obtain the head width data of the target listener (if there are multiple test individuals, calculate the average width). Then, select the HRTF with the head width most similar to the head width of the target listener from the HRTF database (Hutubs) containing anthropometric features, extract the phase spectrum from the selected HRTF, and synthesize the phase spectrum and the predicted HRTF amplitude spectrum into a complex HRTF spectrum, which is then restored to the time domain HRIR through inverse Fourier transform.
Claims
1. A HRTF reconstruction device based on an automatic encoder and spherical harmonic expansion, characterized in that: The device includes: A variational autoencoder is used to obtain an encoded representation of the auricle features based on the ear image; Spherical harmonic coefficient expansion module, used to expand HRTF into spherical harmonic coefficients using spherical harmonic functions; A prediction module, used for predicting HRTF spherical harmonic expansion coefficients according to the picture coding representation output by the variational autoencoder; The reconstruction module is used to reconstruct the spherical harmonic expansion coefficients predicted by the prediction module into HRTF amplitudes, and then personalize the HRTF phase according to the listener's head width, so as to obtain a complete HRTF.
2. The HRTF reconstruction device based on automatic encoder and spherical harmonic expansion according to claim 1, characterized in that: The variational autoencoder includes an encoder module and a decoder module. The encoder module is a probability model that maps input samples to probability distribution and is used to derive a compact coding representation of an ear depth image; the decoder module is used to reconstruct the ear image from the compact coding representation of the ear depth image.
3. The HRTF reconstruction device based on automatic encoder and spherical harmonic expansion according to claim 1, characterized in that: The prediction module consists of a neural network that maps each frequency to be processed into an embedding vector, concatenates the encoded frequency information with the ear image encoding information, and the concatenated encoding is input into multiple layers of a one-dimensional convolutional neural network to gradually extract and analyze important features in the data; The prediction module uses the mean square error between the true spherical harmonic expansion coefficients and the predicted spherical harmonic expansion coefficients as the loss function for training.
4. A method for using a HRTF reconstruction device based on an automatic encoder and spherical harmonic expansion as claimed in claim 1, characterized in that: The method comprises the following steps: (1) Using the 3D head mesh in the HRTF database, the ear depth image corresponding to each head mesh is obtained; the HRTF of each individual in the HRTF database is expanded using spherical harmonic functions to obtain the corresponding spherical harmonic expansion coefficients, and the ear depth image and the corresponding spherical harmonic expansion coefficients are combined to construct a data set; (2) constructing the variational autoencoder, wherein the variational autoencoder includes an encoder module and a decoder module; (3) using the ear depth image obtained in step (1) to train the variational autoencoder, and using the encoder module to obtain a compact encoding representation of the ear depth image; (4) Construct a neural network model to predict HRTF spherical harmonic expansion coefficients based on image coding; (5) using the compact coding representation of the ear depth image obtained in step (3) and the HRTF spherical harmonic expansion coefficients obtained in the corresponding step (1), the neural network model in step (4) is trained to obtain trained weights; (6) Inputting the ear depth image to be tested into the encoder module to obtain a compact coding representation of the ear depth image, and then inputting the compact coding representation into the model trained in step (5) to obtain predicted spherical harmonic expansion coefficients, and reconstructing the predicted HRTF through inverse spherical harmonic transformation; (7) The predicted HRTF is interpolated and supplemented with phase information to obtain a complete HRTF spectrum, which is then converted into a head-related impulse response (HRIR) through inverse Fourier transform.
5. The method according to claim 4, characterized in that In step (2), the encoder module is a probability model that maps input samples to probability distributions. The encoder module maps the input ear depth image to a probability distribution p(z|x), that is, the distribution of the latent variable z given the input data x, which is parameterized by its mean μ and logarithmic variance σ, as the output of the encoder module.
6. The method according to claim 4, characterized in that In step (3), the method for training the variational autoencoder is specifically as follows: The ear depth image obtained in step (1) is input into the encoder module to obtain a compact representation of the encoded image, and then the compact encoded representation of the image is input into the decoder module to reconstruct the original input. The output image of the decoder module is compared with the original input image, and the difference between the two is calculated as the cost function. After optimization training, the loss of image reconstruction is minimized.
7. The method according to claim 4, characterized in that In step (5), the training method of the deep neural network model is specifically as follows: First, the frequency is mapped to an embedded vector, and then the frequency encoding information and the compact encoding representation information of the ear image obtained in step (3) are concatenated; the concatenated encoding information is input into the convolutional neural network to predict the HRTF spherical harmonic expansion coefficients of the ear at the current frequency; the predicted spherical harmonic coefficients are compared with the actual spherical harmonic coefficients, the mean square error is calculated and fed to the model for optimization training.
8. The method according to claim 4, characterized in that In step (7), the specific method of supplementing the phase information is as follows: first, the head width data of the target listener is obtained; then, from a database containing anthropometric features, an individual whose head width is closest to the target listener is selected, and a phase spectrum is extracted from the HRTF of the individual; and then the extracted phase spectrum and the predicted HRTF amplitude spectrum are synthesized into a complex HRTF spectrum.