A personalized head-related transfer function model based on hierarchical integration
Through the hierarchical integration of personalized head-related transfer function models, shared, personalized and residual models are established respectively, which solves the problems of high complexity and large errors of HRTF personalization methods, improves HRTF prediction performance, and is suitable for personalized spatial audio and virtual reality devices.
Patent Information
- Application Number
- CN202111135805.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-09-27
- Publication Date
- 2025-09-19
- Estimated Expiration
- 2041-09-27
AI Technical Summary
Existing HRTF personalization methods are highly complex, time-consuming, or have large errors in high frequencies. In addition, the dimensionality reduction process loses detailed information, affecting the spatial audio quality.
A personalized head-related transfer function model based on hierarchical integration is adopted, including a preprocessing module, a GMM-based common model module, a DNN-based personalized model module, a residual model module and a HRTF reconstruction module. A user-independent common model, a user-related personalized model and a residual model are established respectively to restore the positioning information in the personalized HRTF.
It reduces the loss of HRTF personalized spectrum and improves the prediction performance of HRTF, making it suitable for personalized spatial audio and virtual reality devices.
Smart Images
Figure CN113872565B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of signal processing in the electronics industry, and in particular to a personalized head-related transfer function model based on hierarchical integration. Background Art
[0002] The human auditory process can generally be viewed as a source-channel-receiver model, where the channel encompasses the process by which the sound source diffracts and interferes through different parts of the human body before ultimately reaching the eardrum. This can be viewed as a spatial digital filter, known as a head-related transfer function (HRTF). Therefore, HRTFs encompass all spectral features caused by the interaction between sound waves and body parts, and are highly correlated with physiological characteristics such as height, head size, and ear shape. Because each individual's physiological structure varies, HRTFs are personalized. Non-personalized HRTFs can result in undesirable effects such as upside-down inversion, front-to-back confusion, and in-head localization. Therefore, research on personalized HRTFs is crucial.
[0003] Currently, existing research on HRTF personalization can be divided into multiple methods. Theoretical or mathematical modeling analysis of the human body, such as the spherical head model, snowman model, structural model, boundary element method, finite difference time domain method, etc. However, these methods are relatively complex. Therefore, some low-complexity personalized HRTF methods have been proposed. One type is the perception-based method, which determines the parameters through audiometry experiments. This type of method requires pre-measurement of a large-scale database for matching to obtain the HRTF that best suits the target object, which is time-consuming. The method of estimating the full space of HRTF from a small data measurement set is another type of HRTF personalization method. However, most existing methods only obtain the coefficients of the linear prediction model from a small data test set and then expand it to the full space. This method has large errors in the high frequency range.
[0004] Considering the dependency of HRTFs on human physiological parameters, one approach involves first reducing the dimensionality of the HRTFs and then establishing a mapping between the physiological parameters and the reduced HRTFs. This approach loses detailed HRTF information during the dimensionality reduction and reconstruction process. This detail often relates to positioning cues in three-dimensional space, thus impacting the quality of spatial audio.
[0005] Therefore, there is an urgent need to provide a personalized method for head-related transfer function based on a pre-trained model. Summary of the Invention
[0006] (1) Technical issues to be resolved
[0007] To solve one or more of the above problems, the present invention provides a personalized head-related transfer function model based on hierarchical integration to solve the problems mentioned in the background technology.
[0008] (2) Technical solution
[0009] According to one aspect of the present invention, a personalized head-related transfer function model based on hierarchical integration is provided, comprising a preprocessing module, a GMM-based common model module, a DNN-based personalized model module, a residual model module, a personalized HRTF prediction module, and an HRTF reconstruction module. The preprocessing module is configured to normalize HRTFs and human physiological parameters. The GMM-based common model module is connected to the preprocessing module and is configured to obtain a user-independent universal HRTF. The DNN-based personalized model module is respectively connected to the preprocessing module and the GMM-based common model module and is configured to establish a mapping relationship between human physiological parameters and personalized HRTFs after removing common components. The residual model module is respectively connected to the preprocessing module, the GMM-based common model module, and the DNN-based personalized model module. The residual represents the content in the HRTF excluding the user-independent common model and personalized model outputs. The residual model module is configured to model the residual components in the HRTF excluding the user-independent common model and personalized model outputs.
[0010] Preferably, the preprocessing module branch includes a HRTF preprocessing module and a human physiological parameter preprocessing module, the HRTF preprocessing module is used to normalize the HRTF value; the human physiological parameter preprocessing module is used to normalize the human physiological parameter value.
[0011] Preferably, the DNN-based personalized model module comprises a dimensionality reduction module based on an autoencoder and a DNN-based mapping module;
[0012] The dimensionality reduction module based on the autoencoder is connected to the preprocessing module, and the dimensionality reduction module based on the autoencoder is used to reduce the dimension of the head-related transfer function;
[0013] The DNN-based mapping module is connected to the autoencoder-based dimensionality reduction module, and uses the HRTF after dimensionality reduction to establish a mapping relationship between human physiological parameters and the personalized HRTF after removing common components.
[0014] Preferably, the personalized HRTF prediction module is connected to the preprocessing module and the DNN-based personalized model module respectively, and the personalized HRTF prediction module is used to predict the personalized HRTF components of the target object.
[0015] Preferably, the HRTF reconstruction module is connected to the GMM-based common model module, the residual model module, and the personalized HRTF prediction module respectively, so as to integrate the model components of the three levels to obtain the personalized HRTF.
[0016] (3) Beneficial effects
[0017] As can be seen from the above technical solution, the present invention proposes a personalized head-related transfer function model based on hierarchical integration, which has the following beneficial effects:
[0018] (1) The personalized head-related transfer function model based on hierarchical integration provided in the present invention models the residuals and model errors generated in the dimensionality reduction process, thereby reducing the loss of HRTF personalized spectrum;
[0019] (2) The hierarchically integrated personalized head-related transfer function model in the present invention establishes a user-independent common model, a user-related personalized model, and a residual model to restore the positioning information in the personalized HRTF and improve the prediction performance of the HRTF;
[0020] (3) The present invention can be used to generate personalized spatial audio, be used in virtual reality devices, etc., and has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] Figure 1 This is a schematic diagram of the structure of the personalized head-related transfer function model based on hierarchical integration implemented in the present invention.
[0022] Figure 2 It is a structural diagram of a DNN-based personalized model implementing a hierarchical integration-based personalized head-related transfer function model in the present invention.
[0023]
Main component symbol description
[0024] 1- Preprocessing module;
[0025] 2- GMM-based shared model module;
[0026] 3-DNN-based personalized model module;
[0027] 4-Residual model module;
[0028] 5-Personalized HRTF prediction module;
[0029] 6-HRTF reconstruction module;
[0030] 11-HRTF preprocessing module;
[0031] 12-Human physiological parameter preprocessing module; DETAILED DESCRIPTION
[0032] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.
[0033] It should be noted that similar or identical parts are numbered the same in the drawings and descriptions. In the drawings, references are made for simplicity or convenience. Furthermore, implementations not depicted or described in the drawings are known to those of ordinary skill in the art. Furthermore, while examples of parameters with specific values may be provided herein, it should be understood that the parameters do not need to be exactly equal to the corresponding values, but rather may approximate the corresponding values within acceptable error tolerances or design constraints.
[0034] The present invention provides a personalized head-related transfer function model based on hierarchical integration. It constructs three levels: a common model based on GMM, a personalized model based on DNN, and a residual model. Common features, personalized features, and residual features are modeled respectively. The models at the three levels are integrated to obtain a personalized HRTF. This hierarchical model can improve the accuracy of personalized head-related transfer function estimation and has high theoretical and application value.
[0035] like Figure 1 As shown, in an exemplary embodiment of the present invention, a personalized head-related transfer function model based on hierarchical integration is proposed. Figure 1 This is a schematic diagram of the structure of the personalized head-related transfer function model based on hierarchical integration implemented in the present invention. This implementation includes: a preprocessing module 1 for normalizing HRTFs and human physiological parameters; a GMM-based shared model module 2, connected to the preprocessing module 1, for obtaining a user-independent universal HRTF; a DNN-based personalized model module 3, connected to the preprocessing module 1 and the GMM-based shared model module 2, for establishing a mapping relationship between human physiological parameters and personalized HRTFs after removing shared components; a residual model module 4, connected to the preprocessing module 1, the GMM-based shared model module 2, and the DNN-based personalized model module 3, for modeling the residual components of the HRTF excluding the user-independent shared model and personalized model outputs; a personalized HRTF prediction module 5, connected to the preprocessing module and the DNN-based personalized model module, for predicting the personalized HRTF components of the target object; and an HRTF reconstruction module 6, connected to the GMM-based shared model module 3, the residual model module 4, and the personalized HRTF prediction module 5, for integrating the three levels of model components to obtain a personalized HRTF. Each module is described in detail below.
[0036] In this embodiment, the preprocessing module 1 is used to normalize HRTF and human physiological parameters. The preprocessing module 1 includes an HRTF preprocessing module 11 for normalizing HRTF values and a human physiological parameter preprocessing module 12 for normalizing human physiological parameters.
[0037] The HRTF preprocessing module 11 is used to normalize the HRTF values and use them as input to the GMM-based common model module 2, the DNN-based personalized model module 3, and the residual model module 4. Since humans are not sensitive to the details of the HRTF phase spectrum in positioning perception, the minimum phase HRTF of both ears and the interaural time delay can better estimate the HRTF and the spatial position. , frequency point The left or right ear HRTF at can be expressed as:
[0038] (1)
[0039] in, Indicates the HRTF for the left or right ear. , is the number of spatial locations where HRTF is measured, , The number of frequency points for each HRTF spectrum. and Represents spatial position Frequency The HRTF amplitude spectrum and phase of the minimum phase at point 1 are:
[0040] (2)
[0041] The logarithmic amplitude spectrum of HRTF is closer to human auditory perception, so the logarithmic amplitude spectrum of HRTF is used to model HRTF. The HRTF is preprocessed using the standard normalization method, and the mean of the training sample is 0 and the variance is 1, which is expressed as
[0042] (3)
[0043] in, , and The HRTF logarithmic amplitude spectrum of all spatial positions at the frequency point The mean and standard deviation at . Represents the HRTF of the left or right ear after preprocessing. Assume that the HRTF of the left and right ear after preprocessing are represented as and , can be calculated based on the HRTF of the left and right ears using equations (1) to (3). The pre-processed HRTF can be expressed as: ,in, , for or .
[0044] The human physiological parameter preprocessing module 12 is used to normalize the values of human physiological parameters and use them as inputs for the DNN-based personalized model module 3 and the personalized HRTF prediction module 6. Since the values of human physiological parameters are all positive numbers, in order to preserve this characteristic, the parameter values are normalized to a number between 0 and 1 during preprocessing. User's Personal physiological characteristics , the preprocessing process can be expressed as
[0045] (4)
[0046] in, and For all users in The minimum and maximum values of individual human physiological characteristics. , is the number of users already in the database, , The number of human physiological parameters measured for each user.
[0047] The GMM-based shared model module 2 is connected to the pre-processing module 1 and is used to obtain a universal HRTF that is independent of the user. The shared model assumes that the user-independent components of the HRTF are available. Gaussian components, then the problem of solving the common model can be described as
[0048] (5)
[0049] in, is the HRTF matrix after preprocessing. are the model parameters in GMM. The first The weights of the Gaussian components, The first The probability density distribution of Gaussian components, each row of which is represented by
[0050] (6)
[0051] in, and Respectively The mean and covariance of the Gaussian components are calculated and the Expectation-Maximization algorithm (EM) is used to learn the model parameters. By mixing multiple Gaussian components with the learned parameters, a user-independent shared HRTF can be obtained, which is expressed as
[0052] (7)
[0053] The DNN-based personalized model module 3 is connected to the preprocessing module and the GMM-based shared model module, and is used to establish a mapping relationship between human physiological parameters and the personalized HRTF after removing the shared components. Figure 2 This is a schematic diagram of the structure of the DNN-based personalized model 2 implementing the hierarchical integrated personalized head-related transfer function model of the present invention. The module branch includes:
[0054] The dimensionality reduction module 31 based on the autoencoder is connected to the preprocessing module 1 and is used to reduce the dimension of the head-related transfer function. First, the user-irrelevant common HRTF components are removed to obtain the personalized HRTF spectrum, which is expressed as
[0055] (8)
[0056] in, is the output of the GMM-based common model module 2, i.e., the user-independent common HRTF component.
[0057] Then, through the encoder, the number of nodes in the last layer of the encoder is set to be lower than the number of input HRTF features, and the input HRTF is reduced in dimension to obtain the vector after HRTF dimension reduction, which is expressed as , decode the HRTF after dimension reduction to get the estimate of the complete HRTF , this process is expressed as
[0058] (9)
[0059] (10)
[0060] The DNN-based mapping module 32 is connected to the autoencoder-based dimensionality reduction module 31, and uses the HRTF after dimensionality reduction to establish a mapping relationship between human physiological parameters and personalized HRTF after removing common components. During model training, the input is the pre-processed value of the human physiological parameters of known users in the database. and location d The position information at , including azimuth and elevation, the input vector can be expressed as The output is the corresponding position d Implicit representation of HRTF at ,Right now ,in, is a set of parameters related to the model.
[0061] When training autoencoders and DNNs, a loss function is needed to measure the accuracy of the model. Considering that the HRTF logarithmic amplitude spectrum contains most of the positioning information, the loss function is designed to be the mean square error function of the weighted log-spectral distortion (LSD), which is expressed as
[0062] (11)
[0063] in, Estimated positions for DNN-based personalization models Frequency Points The implicit representation of the HRTF at . Select the standard deviation As weights to compensate for the impact of HRTF preprocessing. DNN minimizes the loss function To maximize objective performance and obtain optimal model parameters , expressed as
[0064] (12)
[0065] Therefore, the actual output of the personalized model can be expressed as (The training was performed using the PyTorch tool. Since the specific model training method is a relatively common deep neural network training method, it will not be introduced in detail here.)
[0066] The residual model module 4 is connected to the preprocessing module 1, the GMM-based common model module 2, and the DNN-based personalized model module 3, and is used to model the residual in the HRTF except for the user-independent common model and personalized model output. The residual component is the content in the HRTF except for the user-independent common model and personalized model output, and is expressed as
[0067] (13)
[0068] in,
[0069] is the output of the DNN-based personalized model module 3, as shown in formula (10).
[0070] The residual component contains model errors, spectral losses during dimensionality reduction and reconstruction, etc., which are related to the personalized characteristics of HRTF. In order to improve the accuracy, a linear regression model is used to learn the mapping relationship between personalized HRTF and residual components. The modeling problem of the linear residual model can be expressed as
[0071] (14)
[0072] in, Indicates the The HRTF estimation of each user is obtained by the personalized model in Section 2.2. Represents the model weight. After learning the linear regression model, the optimal model weight can be obtained.
[0073] Therefore, the residual component can be estimated as
[0074] (15)
[0075] The personalized HRTF prediction module 5 is connected to the pre-processing module 1 and the DNN-based personalized model module 3 to predict the personalized HRTF components of the target object. When predicting personalized HRTF, the human physiological parameters of the target user are given. , , after preprocessing according to formula (4), the forward calculation is performed through the trained DNN and decoder network to obtain the target user at any position d Estimation of personalized HRTF at , expressed as
[0076] (16)
[0077] in, is the human physiological parameter vector after preprocessing.
[0078] The HRTF reconstruction module 3 is connected to the personalized HRTF prediction module 5, the GMM-based common model module 2, and the linear residual model module 3 to integrate the three-level model components to obtain personalized HRTF. The personalized HRTF of the target user can be based on the individual physiological parameters of the target user. , integrating three levels of models, namely the shared model, the personalized model and the residual model, is expressed as
[0079] (17)
[0080] in, is the estimated pre-processed HRTF, User-independent results output by the shared model, is the user-related personalized part obtained by formula (11), is the estimated model residual.
[0081] Will Perform the inverse transformation of the preprocessing process, that is, the reverse processing according to formula (3), to obtain the minimum phase HRTF of the target user By adding the time delay caused by path transmission, the spatial position can be obtained , frequency point The complete HRTF at
[0082] (18)
[0083] in, The location of the spatial audio to be generated The distance from the target user's left or right ear, is the speed of sound waves in the air. Therefore, the left and right ear HRTF can be expressed as
[0084] (19)
[0085] Among them, the HRTF of the left ear and the right ear are The first and second half of , .
[0086] This patent proposes a personalized spatial audio generation method based on hierarchical integration, which respectively establishes a user-independent common model, a user-related personalized model, and a residual model to restore the positioning information in the personalized HRTF. First, a Gaussian mixture model is used to obtain a user-independent common model. Then, a deep neural network is used to establish a mapping between the user-related HRTF and the physiological parameters of the human body to obtain a personalized model. In order to restore the detailed information of the personalized HRTF as much as possible, linear modeling is performed on the residuals, model errors, etc. in the dimensionality reduction process of the above model. The models at the three levels are integrated to obtain personalized HRTF. Finally, the personalized HRTF can be used to generate three-dimensional spatial audio.
[0087] The present invention proposes a personalized head-related transfer function model based on hierarchical integration, which is written in Python and C. In addition, the present invention can be applied to computer terminals, handheld mobile devices or other forms of mobile devices.
[0088] It should be noted that the above definitions of the various elements are not limited to the various specific structures or shapes mentioned in the embodiments, and those skilled in the art can simply and familiarly replace them.
[0089] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.
Claims
1. A personalized head-related transfer function model based on hierarchical integration, characterized by: It includes a preprocessing module, a GMM-based common model module, a DNN-based personalized model module, a residual model module, a personalized HRTF prediction module and a HRTF reconstruction module, wherein the preprocessing module is used to normalize HRTF and human physiological parameters; the GMM-based common model module is connected to the preprocessing module, and the GMM-based common model module is used to obtain a universal HRTF that is independent of the user; the DNN-based personalized model module is respectively connected to the preprocessing module and the GMM-based common model module, and the DNN-based personalized model module is used to establish a mapping relationship between human physiological parameters and personalized HRTF after removing common components; the residual model module is respectively connected to the preprocessing module, the GMM-based common model module and the DNN-based personalized model module, and the residual represents the content in HRTF except the output of the common model and personalized model that are independent of the user, and the residual model module is used to reconstruct the HRTF. The residual components other than the user-independent common model and personalized model output are modeled; the personalized HRTF prediction module is respectively connected to the preprocessing module and the DNN-based personalized model module, and the personalized HRTF prediction module is used to predict the personalized HRTF components of the target object; the HRTF reconstruction module is respectively connected to the GMM-based common model module, the residual model module, and the personalized HRTF prediction module, and is used to integrate the three levels of model components to obtain personalized HRTF.
2. The personalized head-related transfer function model based on hierarchical integration according to claim 1, characterized in that: The pre-processing module includes a HRTF pre-processing module and a human physiological parameter pre-processing module, and the HRTF pre-processing module is used to normalize the HRTF value; The human physiological parameter preprocessing module is used to normalize the values of the human physiological parameters.
3. The personalized head-related transfer function model based on hierarchical integration according to claim 1, characterized in that: The DNN-based personalized model module includes a dimensionality reduction module based on an autoencoder and a DNN-based mapping module; The dimensionality reduction module based on the autoencoder is connected to the preprocessing module, and the dimensionality reduction module based on the autoencoder is used to reduce the dimension of the head-related transfer function; The DNN-based mapping module is connected to the autoencoder-based dimensionality reduction module, and uses the HRTF after dimensionality reduction to establish a mapping relationship between human physiological parameters and the personalized HRTF after removing common components.
4. The personalized head-related transfer function model based on hierarchical integration according to claim 1, characterized in that The personalized HRTF prediction module is connected to the preprocessing module and the DNN-based personalized model module respectively, and the personalized HRTF prediction module is used to predict the personalized HRTF components of the target object.
5. The personalized head-related transfer function model based on hierarchical integration according to claim 1, characterized in that: The HRTF reconstruction module is respectively connected to the GMM-based common model module, the residual model module, and the personalized HRTF prediction module, and is used to integrate the model components of the three levels to obtain the personalized HRTF.