A Personalized Reconstruction Method of Head-Related Transfer Function Based on Spatial Orientation Fusion and Frequency Channel Fusion

Abstract

This invention discloses a personalized reconstruction method for the head-related transfer function (HRTF) based on spatial orientation fusion and frequency channel fusion, aiming to solve the technical problem of how to quickly and accurately obtain a comprehensive personalized HRTF of a subject from a small amount of measured orientation data. It includes the following steps: preprocessing HRTF data in the CIPIC database; rearranging the three-dimensional amplitude spectra of all orientations at all pitch angles after preprocessing to obtain a two-dimensional amplitude spectrum of spatial orientation-frequency channels; retaining the amplitude values ​​of all frequencies in the spatial orientation portion of the two-dimensional amplitude spectrum, and setting the amplitude values ​​of other orientations to 0, to obtain the input dataset; establishing a neural network structure for personalized HRTF reconstruction; and inputting the preprocessed data into the neural network structure for training to form a neural network model for personalized HRTF reconstruction. The model of this invention has low complexity, exhibits good performance in terms of mean logarithmic spectral distortion and root mean square error, and has a short training time.

Description

Technical Field

[0001] This invention relates to the field of audio signal processing technology, specifically to a personalized reconstruction method for head-related transfer functions based on spatial orientation fusion and frequency channel fusion. Background Technology

[0002] Sound contains a wealth of information. Hearing is an important way for humans to perceive and acquire information about the natural environment. The auditory system can distinguish the meaning of speech and the object that makes the sound from the sound, and can extract the location information of the sound source, thereby identifying the direction from which the sound is emitted. Therefore, studying the human ability to perceive the spatial characteristics of sound has always been a research focus in the fields of acoustics and auditory psychology. This research has extremely important scientific value.

[0003] The Head-Related Transfer Function (HRTF) describes the acoustic transfer function between a point sound source in a free field and the listener's ear canal. Once a person's full-space HRTF database is obtained, convolving the temporal HRIR (the temporal form of the HRTF) of a specific spatial location with the sound signal can render the sound at that spatial location that the listener wants to hear, achieving an immersive auditory experience. However, HRTF is highly personalized because it is closely related to our physiological parameters. Therefore, researching and obtaining personalized HRTFs is of great significance.

[0004] Currently, personalized HRTFs are mainly obtained through experimental measurement, personalized methods based on physiological parameters, personalized methods based on a small number of measurements, and deep learning methods. The most accurate existing method for obtaining personalized HRTFs is laboratory measurement, which typically involves subjects undergoing HRTF measurements in a semi-ablation or fully ablation laboratory. During the measurement, subjects wear a miniature microphone at the ear canal opening or a specific location within the ear canal to receive sounds emitted from fixed speakers at different locations in the space. However, experimental measurement is usually costly and time-consuming. Existing HRTF databases include CIPIC, MIT, IRCAM, ARI, and HUTUBS. Some researchers also obtain personalized HRTFs through personalized methods based on physiological parameters, specifically physiological parameter matching methods, which are divided into subjective and objective matching. Subjective selection involves adjusting the HRTF model parameters through subjective experimental feedback from the subject to achieve an approximate personalized HRTF. Objective selection involves matching the subject's physiological parameters with existing physiological parameters in a database to find the database HRTF with the closest physiological parameters as an approximation of the subject's personalized HRTF. However, regardless of whether the subjective or objective selection method is used, the resulting HRTF is only an approximation of the personalized HRTF, not a truly personalized customization. Furthermore, current studies do not fully define the measurement points for the same physiological parameter (e.g., head width) consistently, leading to incomparable results. There is a lack of a complete, independent, representative, and minimal set of physiological parameters, which introduces errors during the measurement process and affects the accuracy of HRTF modeling. With the improvement of computing power in recent years, many researchers have attempted to use deep learning methods to establish the relationship between human physiological parameters and HRTF to synthesize HRTFs for new subjects. However, most current personalized HRTF generation methods can only generate HRTFs in specific directions. For example, building a neural network to predict the horizontal HRTF in the CIPIC database requires building separate networks for each of the 25 azimuth angles on the horizontal plane, making the personalized HRTF generation system overly complex and memory-intensive, thus limiting its application in the field of spatial audio playback. Table 1 summarizes the research and development of personalized head-related transfer function modeling methods both domestically and internationally in recent years.

[0005] Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a personalized reconstruction method of head-related transfer function based on spatial orientation fusion and frequency channel fusion, which is used to solve the technical problem of how to quickly and accurately obtain a comprehensive personalized HRTF of the subject based on a small amount of measured orientation data.

[0007] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0008] A personalized reconstruction method for the head-related transfer function based on spatial orientation fusion and frequency channel fusion is designed, including the following steps:

[0009] (1) Preprocess the HRTF data in the publicly available dataset CIPIC database;

[0010] (2) The three-dimensional amplitude spectrum of all azimuths under all pitch angles after preprocessing is rearranged to obtain the two-dimensional amplitude spectrum of the spatial azimuth-frequency channel;

[0011] (3) Retain the amplitude values ​​of all frequencies in the spatial orientation of the two-dimensional amplitude spectrum, and set the amplitude values ​​of the other orientations to 0 to obtain the input dataset;

[0012] (4) Establish a neural network structure for personalized HRTF reconstruction, including an input layer, an information fusion MLP layer, and an output layer;

[0013] (5) Input the preprocessed data in step (1) into the neural network structure described in step (4) for training to form a neural network model for personalized HRTF reconstruction.

[0014] Preferably, in step (1), the HRIR data of each pitch angle and each azimuth angle in the CIPIC library are transformed to the frequency domain by fast Fourier transform to obtain the frequency domain HRTF. The spectrum of the first 129 points in the frequency range of 0~22 kHz is selected to obtain the HRTF data of the three-dimensional amplitude spectrum of pitch angle-azimuth angle-frequency.

[0015] Preferably, in step (2), the horizontal axis of the two-dimensional amplitude spectrum of the spatial azimuth-frequency channel is the number of frequency boxes, the vertical axis is the number of spatial azimuths, the 0th to 25th rows of the two-dimensional amplitude spectrum are the 25 azimuth angles corresponding to an elevation angle of -45°, and the 25th to 50th rows are the 25 azimuth angles corresponding to an elevation angle of 39.375°, wherein the azimuth angles are ordered as -80° to 80°.

[0016] Preferably, in step (3), the partial spatial orientation includes -45° and 45° azimuth angles corresponding to an elevation angle of 0°, and 0° azimuth angle corresponding to an elevation angle of 180°.

[0017] Preferably, in step (4), the information fusion MLP layer includes multiple sub-layers with the same structure, each sub-layer including a frequency channel fusion MLP layer, a spatial orientation fusion MLP layer, and an activation function layer.

[0018] Preferably, in step (4), the frequency channel fusion MLP layer allows information fusion between different frequencies, the spatial orientation fusion MLP layer allows information fusion between different spatial locations, and there is a residual connection between the frequency channel fusion MLP layer and the spatial orientation fusion MLP layer, which are interleaved.

[0019] Preferably, in step (4), the calculation formula in the information fusion MLP layer is:

[0020]

[0021]

[0022] Where X is the input data, A is the learnable weight matrix of the spatial orientation fusion MLP layer, Z is the output result after the spatial orientation fusion MLP layer, B and C are the learnable weight matrices of the frequency channel fusion MLP layer, Y is the output result of the frequency channel fusion MLP layer, GELU is the activation function, and F is calculated using the following formula:

[0023]

[0024] Where X represents the input data. , It is a learnable weight vector.

[0025] Preferably, in step (5), the evaluation metrics of the model are logarithmic spectral distortion (LSD) and root mean square error (RMSE), and the formulas are as follows:

[0026]

[0027] Where K represents the number of frequency boxes, The number representing spatial locations, Represents azimuth. Represents the angle of elevation. and These represent the azimuth angles respectively. and elevation angle In the direction, the linear scale of the true HRTF and the reconstructed HRTF of the k-th frequency box;

[0028]

[0029] Where S represents the number of subjects, t represents time, m represents spatial orientation, and N represents the total length of the HRIR. Representing the true HRIR, This represents the predicted HRIR.

[0030] Preferably, in step (5), the AdamW neural network optimizer and mean squared error loss function are selected for training.

[0031] Preferably, in step (5), the programming language used for training includes Python, and the environment used includes PyTorch.

[0032] Compared with the prior art, the beneficial technical effects of the present invention are as follows:

[0033] 1. This invention utilizes the publicly available CIPIC database and trains a deep neural network for personalized HRTF reconstruction using a spatial orientation and frequency information fusion MLP network with residual connections. By inputting partial spatial orientation HRTF data from a subject, personalized HRTFs for all spatial orientations of that subject can be obtained. This invention can obtain a comprehensive personalized HRTF for a user using only a small amount of user HRTF data. Compared to measuring HRTF in a semi-abstract or fully absorptive laboratory, this method reduces costs and saves experimental time. Compared to other methods such as physiological parameter methods and deep learning methods for obtaining HRTF, the model proposed in this invention has fewer parameters, shorter training time, and a smaller LSD of the obtained HRTF.

[0034] 2. The model of this invention has low complexity. The proposed model fully considers the nonlinear relationship between different orientations and frequencies of HRTF data, and shows good performance in terms of average logarithmic spectral distortion and root mean square error, and has a short training time. Attached Figure Description

[0035] Figure 1 This is a flowchart of the method described in Embodiment 1 of the present invention.

[0036] Figure 2 This is a spatial orientation fusion MLP structure diagram of the method described in Embodiment 1 of the present invention.

[0037] Figure 3 This is a frequency channel fusion MLP structure diagram of the method described in Embodiment 1 of the present invention.

[0038] Figure 4 This is a schematic diagram of the information fusion MLP structure of the method described in Embodiment 1 of the present invention.

[0039] Figure 5 This is a diagram of the deep neural network structure of the method described in Embodiment 1 of the present invention.

[0040] Figure 6 This is a schematic diagram of the interauricular polar coordinate system in Embodiment 1 of the present invention. Detailed Implementation

[0041] The specific embodiments of the present invention will be described below with reference to the accompanying drawings and examples. However, the following examples are only used to illustrate the present invention in detail and do not limit the scope of the present invention in any way.

[0042] The programs involved or relied upon in the following embodiments are all conventional or simple programs in this technical field. Those skilled in the art can make conventional choices or adaptive adjustments according to specific application scenarios.

[0043] Example 1: A personalized reconstruction method for the head-related transfer function based on spatial orientation fusion and frequency channel fusion, see [link to example]. Figure 1 This includes the following steps:

[0044] (1) Preprocess the HRTF data in the publicly available dataset CIPIC database.

[0045] The CIPIC database used is a publicly available dataset. The CIPIC dataset includes measurements of 45 individuals, including 27 men, 16 women, and 2 foremen. Each individual's data contains HRIR and ITD data in 1250 directions. The sound source location is determined by the azimuth angle in interaural polar coordinates. and elevation angle It indicates that participants Figure 6 The azimuth angles are -80°, -65°, -55°, from -45° to 45° in increments of 5°, and 55°, 65°, 80°, and the elevation angles are from -45° to 230.625° in increments of 5.625°.

[0046] The HRIR data for each pitch and azimuth angle in the CIPIC library is extended to 256 points with zero padding, and then transformed to the frequency domain using Fast Fourier Transform (FFTs) to obtain the frequency domain HRTF. The spectrum of the first 129 points (frequency range from 0 to 22 kHz) is selected to obtain the HRTF data of the three-dimensional amplitude spectrum of pitch-azimuth-frequency.

[0047] (2) The three-dimensional amplitude spectrum of all azimuths under all pitch angles after preprocessing is rearranged in a specific way to obtain the two-dimensional amplitude spectrum of spatial azimuth-frequency.

[0048] The horizontal axis of the two-dimensional amplitude spectrum of spatial azimuth-frequency is the number of frequency boxes, and the vertical axis is the number of spatial azimuths. The three-dimensional amplitude spectrum of all azimuths is rearranged in a specific way. Specifically, rows 0 to 25 of the two-dimensional amplitude spectrum are the 25 azimuths corresponding to an elevation angle of -45° (the azimuths are ordered from -80° to 80°), rows 25 to 50 are the 25 azimuths corresponding to an elevation angle of 39.375° (the azimuths are ordered from -80° to 80°), and so on.

[0049] (3) Retain the amplitude values ​​of all frequencies in the spatial orientation of the two-dimensional amplitude spectrum, and set the amplitude values ​​of the other orientations to 0 to obtain the input dataset.

[0050] The selected area includes the amplitude values ​​of all frequencies in three spatial orientations: -45° and 45° azimuth angles corresponding to an elevation angle of 0°, and 0° azimuth angle corresponding to an elevation angle of 180°.

[0051] (4) See Figure 2-5A deep neural network structure for personalized HRTF reconstruction was established, with the network mainly consisting of an information fusion MLP layer. The deep neural network for personalized HRTF includes an input layer, an information fusion MLP layer, and an output layer.

[0052] The information fusion MLP layer consists of multiple sub-layers with the same structure. Each sub-layer includes two types of MLP layers and an activation function layer. The two types of MLP layers are frequency channel fusion MLP layer and spatial orientation fusion MLP layer.

[0053] The frequency channel fusion MLP layer allows information fusion between different channels, while the spatial orientation fusion MLP layer allows information fusion between different spatial locations. Residual connections exist between the two MLP layers, and these two types of MLP layers are interleaved to achieve information fusion across two input dimensions. The formula for calculating the information fusion MLP is:

[0054]

[0055]

[0056] Where X is the input data, A is the learnable weight matrix of the spatial orientation fusion MLP layer, Z is the output result after the spatial orientation fusion MLP layer, B and C are the learnable weight matrices of the frequency channel fusion MLP layer, Y is the output result of the frequency channel fusion MLP layer, GELU is the activation function, and F is calculated using the following formula:

[0057]

[0058] Where X represents the input data. , It is a learnable weight vector.

[0059] (5) The preprocessed data is fed into a deep neural network model for personalized HRTF reconstruction.

[0060] The experimental data consisted of data from 43 individuals (26 males and 17 females) in the CIPIC database. The dataset was divided into two groups: one individual was used for testing and the remaining 42 individuals were used for training.

[0061] The training set data is input into the neural network used for personalized HRTF reconstruction. The AdamW neural network optimizer is selected, with a learning rate set to 0.006 and a loss function set to mean squared error loss. Training of the personalized HRTF neural network is then completed. Simultaneously, the model is tested using the test set, forming the final neural network model for personalized HRTF reconstruction. The programming language used for building and training the deep neural network for personalized HRTF reconstruction includes Python, and the environment used includes PyTorch.

[0062] The evaluation metrics for the model are log-spectral distortion (LSD) and root-mean-square error (RMSE), and the formulas are as follows:

[0063]

[0064] Where K represents the number of frequency boxes, The number representing spatial locations, Represents azimuth. Represents the angle of elevation. and These represent the azimuth angles respectively. and elevation angle In the direction, the linear scale of the true HRTF and the reconstructed HRTF of the k-th frequency box.

[0065]

[0066] Where S represents the number of subjects, t represents time, m represents spatial orientation, and N represents the total length of the HRIR. Representing the true HRIR, This represents the predicted HRIR.

[0067] (6) Process the HRTF of some spatial orientations of the user to be tested according to the above preprocessing method, and input it into the deep neural network model of the trained personalized HRTF reconstruction to obtain the HRTF data of 1250 orientations of the user.

[0068] This embodiment verifies through experiments that the neural network model for personalized HRTF reconstruction determined in this invention has relatively small mean log-spectral distortion and root mean square error on the test set, and can quickly and with low error reconstruct the HRTF values ​​of an individual subject in all directions. The following table shows the comparison results between this method and Unet's LSD when recovering 1250 directions using 3 orientations:

[0069] method U-net This method LSD (dB) 5.60 3.03

[0070] The present invention has been described in detail above with reference to the accompanying drawings and embodiments; however, those skilled in the art will understand that, without departing from the concept of the present invention, various specific parameters in the above embodiments can be changed, or related programs can be equivalently substituted, thereby forming multiple specific embodiments, all of which are common variations of the present invention, and will not be described in detail one by one.

Claims

1. A personalized reconstruction method for the head-related transfer function based on spatial orientation fusion and frequency channel fusion, characterized in that, Includes the following steps: (1) Preprocess the HRTF data in the publicly available dataset CIPIC database; (2) The three-dimensional amplitude spectrum of all azimuths under all pitch angles after preprocessing is rearranged to obtain the two-dimensional amplitude spectrum of the spatial azimuth-frequency channel; (3) Retain the amplitude values ​​of all frequencies in the spatial orientation of the two-dimensional amplitude spectrum, and set the amplitude values ​​of the other orientations to 0 to obtain the input dataset; (4) Establish a neural network structure for personalized HRTF reconstruction, including an input layer, an information fusion MLP layer, and an output layer; the information fusion MLP layer includes multiple sub-layers with the same structure, each sub-layer including a frequency channel fusion MLP layer, a spatial orientation fusion MLP layer, and an activation function layer; the calculation formula in the information fusion MLP layer is: ； ； Where X is the input data, A is the learnable weight matrix of the spatial orientation fusion MLP layer, Z is the output result after the spatial orientation fusion MLP layer, B and C are the learnable weight matrices of the frequency channel fusion MLP layer, Y is the output result of the frequency channel fusion MLP layer, GELU is the activation function, and F is calculated using the following formula: ； Where X represents the input data. , It is a learnable weight vector; (5) Input the preprocessed data in step (1) into the neural network structure described in step (4) for training to form a neural network model for personalized HRTF reconstruction.

2. The personalized reconstruction method of the head-related transfer function based on spatial orientation fusion and frequency channel fusion according to claim 1, characterized in that, In step (1), the HRIR data of each pitch angle and each azimuth angle in the CIPIC library are transformed to the frequency domain by fast Fourier transform to obtain the frequency domain HRTF. The spectrum of the first 129 points in the frequency range of 0~22 kHz is selected to obtain the HRTF data of the three-dimensional amplitude spectrum of pitch angle-azimuth angle-frequency.

3. The personalized reconstruction method of the head-related transfer function based on spatial orientation fusion and frequency channel fusion according to claim 1, characterized in that, In step (2), the horizontal axis of the two-dimensional amplitude spectrum of the spatial azimuth-frequency channel is the number of frequency boxes, and the vertical axis is the number of spatial azimuths. The 0th to 25th rows of the two-dimensional amplitude spectrum are the 25 azimuth angles corresponding to an elevation angle of -45°, and the 25th to 50th rows are the 25 azimuth angles corresponding to an elevation angle of 39.375°. The azimuth angles are ordered from -80° to 80°.

4. The personalized reconstruction method of the head-related transfer function based on spatial orientation fusion and frequency channel fusion according to claim 1, characterized in that, In step (3), the partial spatial orientation includes -45° and 45° azimuth angles corresponding to an elevation angle of 0°, and 0° azimuth angle corresponding to an elevation angle of 180°.

5. The personalized reconstruction method of the head-related transfer function based on spatial orientation fusion and frequency channel fusion according to claim 1, characterized in that, In step (4), the frequency channel fusion MLP layer allows information fusion between different frequencies, the spatial orientation fusion MLP layer allows information fusion between different spatial locations, and there is a residual connection between the frequency channel fusion MLP layer and the spatial orientation fusion MLP layer, which are interleaved.

6. The personalized reconstruction method of the head-related transfer function based on spatial orientation fusion and frequency channel fusion according to claim 1, characterized in that, In step (5), the evaluation metrics of the model are logarithmic spectral distortion (LSD) and root mean square error (RMSE), and the formulas are as follows: ； Where K represents the number of frequency boxes, The number representing spatial locations, Represents azimuth. Represents the angle of elevation. and These represent the azimuth angles respectively. and elevation angle In the direction, the linear scale of the true HRTF and the reconstructed HRTF of the k-th frequency box; ； Where S represents the number of subjects, t represents time, m represents spatial orientation, and N represents the total length of the HRIR. Representing the true HRIR, This represents the predicted HRIR.

7. The personalized reconstruction method of the head-related transfer function based on spatial orientation fusion and frequency channel fusion according to claim 1, characterized in that, In step (5), the AdamW neural network optimizer and mean squared error loss function are selected for training.

8. The personalized reconstruction method of the head-related transfer function based on spatial orientation fusion and frequency channel fusion according to claim 1, characterized in that, In step (5), the programming language used for training includes Python, and the environment used includes PyTorch.

Images