A head-related transfer function determination method, apparatus, device and medium

Abstract

The application relates to the technical field of spatial audio, and discloses a head-related transfer function determination method, device, equipment and medium, the method comprising: obtaining a target image containing an ear of a target object, identifying an ear frame of the target image and determining ear key points in the ear frame; modeling based on the ear key points to obtain a target three-dimensional ear model corresponding to the target image; comparing the target three-dimensional ear model with sample three-dimensional ear models in a preset database to determine a sample three-dimensional ear model closest to the target three-dimensional ear model, the preset database comprising: a plurality of different sample three-dimensional ear models and corresponding head-related transfer functions; and determining the head-related transfer function corresponding to the closest sample three-dimensional ear model as a target head-related transfer function of the target object, the application determines the target head-related transfer function by three-dimensional modeling of the image and comparison with the sample model, thereby reducing the acquisition cost of the personalized head-related transfer function.

Description

Technical Field

[0001] This invention relates to the field of spatial audio technology, and specifically to a method, apparatus, device, and medium for determining head-related transfer functions. Background Technology

[0002] Spatial audio is widely used in virtual / augmented reality, gaming, and film production. Currently, the primary method for implementing binaural spatial audio is based on Head-Related Transfer Function (HRTF) rendering. According to the principles of digital signal processing, HRTF can be viewed as a system function representing sound transmission, describing the physical process of sound transmission from the sound source to the ear in a free-field environment. This process is closely related to the shape of the listener's auricle, head, and torso. Considering the individual differences in head shape and auricle parameters, these differences lead to different positions of the zeros and poles of the transfer function, causing the sound waves emitted from the source to undergo different filtering processes as they enter different individuals' ears.

[0003] The "universal HRTF" currently used in game development and virtual reality cannot provide accurate spatial precision for everyone. This results in head-to-head effects and front-to-back confusion, limiting the externalization of sound and even introducing significant positioning errors, thus severely impacting the listener's immersive experience. Therefore, to achieve a better sense of immersion when rendering spatial audio, a personalized HRTF that incorporates the listener's physiological characteristics is needed.

[0004] In existing technologies, HRTF is often determined through acoustic measurements. However, this method requires specific equipment and an anechoic environment, and the measurement cost is relatively high, which cannot meet users' needs for obtaining personalized HRTF at low cost. Summary of the Invention

[0005] In view of this, the present invention provides a method, apparatus, device and medium for determining header-related transfer functions to solve the problem of high cost of obtaining personalized HRTFs for different users.

[0006] In a first aspect, the present invention provides a method for determining a header-related transfer function, the method comprising:

[0007] Obtain a target image containing the ear of the target object, identify the ear border of the target image, and determine the key points of the ear within the ear border;

[0008] Based on the key points of the ear, a target three-dimensional ear model corresponding to the target image is obtained;

[0009] The target 3D ear model is compared with sample 3D ear models in a preset database to determine the sample 3D ear model that is closest to the target 3D ear model. The preset database includes multiple different sample 3D ear models and corresponding head-related transfer functions.

[0010] The head-related transfer function corresponding to the closest sample 3D ear model is determined as the target head-related transfer function of the target object.

[0011] By performing ear recognition on images containing the ear of the target object and identifying key points within the ear, a corresponding 3D model is created. This model is then compared with sample 3D models in a database to determine the closest 3D model to the target object, and its corresponding head-related transfer function (HRTF) is identified as the target object's HRTF. This process only requires a series of processing calculations on the image containing the target object's ear to obtain a personalized HRTF, significantly reducing the cost of acquiring such a model.

[0012] In one optional implementation, the step of modeling based on the ear key points to obtain a target 3D ear model corresponding to the target image includes:

[0013] Determine the location information corresponding to the key points of the ear, and determine the auricle information in the target image based on the location information;

[0014] Based on the auricle information, a three-dimensional reconstruction of the ear is performed to obtain the target three-dimensional ear model corresponding to the target image.

[0015] By determining the auricle information in the target image through the location information of key points, a more accurate three-dimensional ear model of the target can be obtained through three-dimensional reconstruction, which further ensures the matching degree of the target head-related transfer function determined subsequently.

[0016] In one optional implementation, the three-dimensional ear model is a three-dimensional point cloud model;

[0017] The step of comparing the target 3D ear model with sample 3D ear models in a preset database to determine the sample 3D ear model that is closest to the target 3D ear model includes:

[0018] Similarity calculation is performed on the 3D point clouds corresponding to the target 3D ear model and the sample 3D ear models in the preset database.

[0019] Based on the similarity between each sample 3D ear model and the target 3D ear model, the sample 3D ear model that is closest to the target 3D ear model is determined.

[0020] By calculating the similarity between the target 3D ear model and the sample 3D ear model, the sample 3D ear model that is closest to the target 3D ear model is determined, thereby ensuring that the head-related transfer function determined subsequently can meet the personalized needs of the target object.

[0021] In one optional implementation, the similarity calculation of the 3D point clouds corresponding to the target 3D ear model and the sample 3D ear models in the preset database includes:

[0022] The distance between the target 3D ear model and the sample 3D ear model in the preset database is calculated by using the point cloud iterative nearest neighbor algorithm to obtain the average nearest neighbor distance for each sample 3D ear model.

[0023] The similarity between each sample 3D ear model and the target 3D ear model is determined based on the average nearest neighbor distance. The smaller the average nearest neighbor distance of the sample 3D ear model, the higher its similarity to the target 3D ear model.

[0024] By using the point cloud iterative nearest neighbor algorithm to calculate the nearest neighbor distance, the accuracy of the similarity calculation between the target 3D ear model and the sample 3D ear model can be effectively guaranteed.

[0025] In an optional implementation, before calculating the similarity between the 3D point clouds corresponding to the target 3D ear model and the sample 3D ear models in the preset database, the method further includes:

[0026] Preprocessing is performed on the target 3D ear model and the sample 3D ear models in the preset database to unify the number of point clouds, size, and model orientation of the target 3D ear model and the sample 3D ear models.

[0027] By preprocessing the target 3D ear model and the sample 3D ear models in the preset database, the error during calculation is reduced and the accuracy of similarity calculation is improved.

[0028] In one optional implementation, the preprocessing of the target 3D ear model and sample 3D ear models in a preset database includes:

[0029] The number of point clouds in the target 3D ear model and the sample 3D ear model is unified by resampling;

[0030] The size of the target 3D ear model and the sample 3D ear model is unified by a scaling factor;

[0031] The model orientation of the target 3D ear model and the sample 3D ear model is unified by the average transformation matrix.

[0032] Preprocessing the target 3D ear model and the sample 3D ear model by resampling, scaling factor, and average transformation matrix can further reduce the error during calculation and improve the accuracy of similarity calculation.

[0033] In one optional implementation, identifying the ear border of the target image and determining ear key points within the ear border includes:

[0034] The target image is identified using an ear target detection model to obtain the corresponding ear bounding box;

[0035] The ear border is detected using an ear key point detection model to determine the ear key points within the ear border.

[0036] By using an ear target detection model to identify the ear outline in an image, and by using an ear key point detection model to further determine the ear key points, the ear recognition effect and the accuracy of key point determination can be further guaranteed, thereby ensuring the accuracy of the target 3D ear model obtained through modeling.

[0037] In a second aspect, the present invention provides a head-related transfer function determination apparatus, the apparatus comprising:

[0038] An ear recognition module is used to acquire a target image containing the ear of a target object, identify the ear border of the target image, and determine the ear key points within the ear border.

[0039] A 3D modeling module is used to model based on the key points of the ear to obtain a target 3D ear model corresponding to the target image;

[0040] The model comparison module is used to compare the target 3D ear model with sample 3D ear models in a preset database to determine the sample 3D ear model that is closest to the target 3D ear model. The preset database includes: multiple different sample 3D ear models and corresponding head-related transfer functions.

[0041] The head-related transfer function determination module is used to determine the head-related transfer function corresponding to the closest sample three-dimensional ear model as the target head-related transfer function of the target object.

[0042] Thirdly, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to perform the head-related transfer function determination method of the first aspect or any corresponding embodiment thereof.

[0043] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to perform the header-related transfer function determination method of the first aspect or any corresponding embodiment thereof. Attached Figure Description

[0044] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0045] Figure 1 This is a flowchart illustrating a method for determining a header-related transfer function according to an embodiment of the present invention;

[0046] Figure 2 This is a flowchart illustrating another method for determining the relevant transfer function according to an embodiment of the present invention;

[0047] Figure 3 This is an example diagram of the detection process of the ear target detection model according to an embodiment of the present invention;

[0048] Figure 4 This is an example diagram of the detection process of the ear key point detection model according to an embodiment of the present invention;

[0049] Figure 5 This is an example diagram of key ear points according to an embodiment of the present invention;

[0050] Figure 6 This is an example diagram of an ear model comparison process according to an embodiment of the present invention;

[0051] Figure 7 This is an example diagram of the overall process of ear model matching according to an embodiment of the present invention;

[0052] Figure 8 This is a structural block diagram of a head-related transfer function determination device according to an embodiment of the present invention;

[0053] Figure 9 A schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0055] Spatial audio is widely used in virtual / augmented reality, gaming, and film production. Currently, the primary method for implementing binaural spatial audio is based on Head-Related Transfer Function (HRTF) rendering. According to the principles of digital signal processing, HRTF can be viewed as a system function representing sound transmission, describing the physical process of sound transmission from the sound source to the ear in a free-field environment. This process is closely related to the shape of the listener's auricle, head, and torso. Considering the individual differences in head shape and auricle parameters, these differences lead to different positions of the zeros and poles of the transfer function, causing the sound waves emitted from the source to undergo different filtering processes as they enter different individuals' ears.

[0056] The "universal HRTF" currently used in game development and virtual reality cannot provide accurate spatial precision for everyone. This results in head-to-head effects and front-to-back confusion, limiting the externalization of sound and even introducing significant positioning errors, thus severely impacting the listener's immersive experience. Therefore, to achieve a better sense of immersion when rendering spatial audio, a personalized HRTF that incorporates the listener's physiological characteristics is needed.

[0057] In existing technologies, HRTF is often determined through acoustic measurements. However, this method requires specific equipment and an anechoic environment, and the measurement cost is relatively high, which cannot meet users' needs for obtaining personalized HRTF at low cost.

[0058] To address this, this invention provides a method for determining a head-related transfer function (HRT). This method involves performing ear recognition on an image containing the ear of a target object and identifying key points within the ear to create a corresponding 3D model. This model is then compared with sample 3D models in a database to determine the sample 3D model that best matches the target object, and its corresponding HRT is identified as the target object's HRT. By performing only a series of processing calculations on the image containing the target object's ear, a personalized HRT matching the target object can be obtained, significantly reducing the cost of acquiring a personalized HRT.

[0059] According to an embodiment of the present invention, a method for determining a header-related transfer function is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0060] This invention provides a method for determining a header-related transfer function, which can be used for the aforementioned personalized HRTF determination. Figure 1 This is a flowchart of a header-related transfer function determination method according to an embodiment of the present invention, such as... Figure 1 As shown, the process includes the following steps:

[0061] Step S101: Obtain a target image containing the ear of the target object, identify the ear border of the target image, and determine the key points of the ear within the ear border.

[0062] To determine a personalized HRTF for different objects, it is first necessary to acquire an image containing the ear of the target object. Then, the ear in the image is identified to obtain the ear outline, and key points are detected within the ear within the outline to determine the key points within the ear. These key points can represent the overall outline of the ear.

[0063] Step S102: Model the ear based on key points to obtain the target three-dimensional ear model corresponding to the target image.

[0064] After identifying the key points of the ear in the target object's ear image, these key points can represent the overall outline of the ear. The identified key points can be modeled based on existing 3D modeling networks to obtain the target 3D ear model corresponding to the ear in the target image.

[0065] Step S103: Compare the target 3D ear model with the sample 3D ear models in the preset database to determine the sample 3D ear model that is closest to the target 3D ear model. The preset database includes: multiple different sample 3D ear models and corresponding head-related transfer functions.

[0066] This can be understood as follows: different sample 3D ear models and corresponding head-related transfer functions (HRTFs) that can meet the personalized needs of these different sample 3D ear models are pre-stored in the database. The target 3D ear model is compared with each sample 3D ear model in the database to determine the sample 3D ear model in the database that is closest to the target 3D ear model. Since the closest sample 3D ear model is most similar to the physiological characteristics of the target object's ear, the HRTF corresponding to the closest sample 3D ear model can meet the personalized needs of the target object.

[0067] Step S104: Determine the head-related transfer function corresponding to the closest sample 3D ear model as the target head-related transfer function of the target object.

[0068] Based on the above, it can be concluded that the target 3D ear model constructed from the ear image of the target object and the closest sample 3D ear model are similar in physiological characteristics. Therefore, the HRTF corresponding to the closest sample 3D ear model can also meet the personalized needs of the target object.

[0069] By performing ear recognition on an image containing the ear of a target object and determining key points within the ear, a corresponding 3D model is obtained. This model is then compared with sample 3D models in a database to determine the sample 3D model that best matches the target object, and its corresponding head-related transfer function (HRTF) is identified as the target object's HRTF. This invention only requires a series of processing calculations on the image containing the target object's ear to obtain a personalized HRTF that matches the target object, significantly reducing the cost of acquiring a personalized HRTF.

[0070] According to an embodiment of the present invention, another embodiment of the header-related transfer function determination method is provided, which can be used for the above-described personalized HRTF determination. Figure 2 This is a flowchart of another method for determining the relevant transfer function according to an embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps:

[0071] Step S201: Obtain a target image containing the ear of the target object, identify the ear border of the target image, and determine the key points of the ear within the ear border.

[0072] Specifically, in step S201, identifying the ear border of the target image and determining the ear key points within the ear border includes:

[0073] The target image is identified using an ear target detection model, and the corresponding ear bounding box is obtained.

[0074] The ear border is detected using an ear key point detection model to determine the ear key points within the ear border.

[0075] This can be understood as follows: a pre-trained ear target detection model and an ear key point detection model are used to identify the target image and obtain the corresponding ear bounding box; the pre-trained ear key point detection model is used to detect the ear bounding box and determine the ear key points within the ear bounding box.

[0076] Specifically, an ear target detection model can be built based on the RTMDet model, using CSPDarknet as the backbone architecture of the detection model, while introducing larger convolutional kernels to increase the receptive field. Adding a 5×5 depthwise separable convolutional kernel to the basic unit achieves the best balance between computational efficiency and accuracy. For the neck module used for multi-scale feature fusion, the model improves inference speed without significantly increasing the number of parameters by simply increasing the computational cost of the neck module.

[0077] For example, such as Figure 3 The diagram shown is an example of the detection process of an ear target detection model according to an embodiment of the present invention. This model is built based on the RTMDet model, where CSPNexT is based on CSPDarknet. Exemplarily, the overall detection process is as follows:

[0078] The ear image to be predicted is input into the model. Assuming the input image size is 640x640 and the number of channels is 3, after downsampling by 8 times, 16 times, and 32 times in the CSPNEXt and CSPNEXtPAFPN layers of the model, the head layer obtains feature maps of three sizes: 80x80, 40x40, and 20x20. At this point, the number of channels in all three layers is 256.

[0079] Following the head layer, there are two prediction branches: ① the rtm_cls class prediction branch, which reduces the number of channels from 256 to 80, with 80 corresponding to the total number of classes; and ② the rtm_reg bounding box regression branch, which reduces the number of channels from 256 to 4, where 4 represents the coordinates of the bounding box. Specifically, within these two branches, the _predict_by_feat_single function converts the features of a single image extracted from the head layer into bounding box results, resulting in three lists: cls_score_list, bbox_pred_list, and mlvl_priors, with detailed sizes shown in the figure. Then, the three feature layers are traversed, processing the class prediction branch and the bounding box regression branch respectively. Taking the first layer as an example, the dimensions of the bounding box prediction branch [4, 80, 80] are transformed to [6400, 4], and the class prediction branch [80, 80, 80] is transformed to [6400, 80], and normalized to ensure that the class confidence is between 0 and 1.

[0080] Following the head layer, there is a grid initialization layer that initializes three grids based on the feature map size, with sizes of 6400 (80x80), 1600 (40x40), and 400 (20x20). For example, the shape of the first layer is torch.Size([6400,2]), and the last dimension is 2, which represents the horizontal and vertical coordinates of the grid points. 6400 indicates the number of grid points in the current feature layer.

[0081] Finally, threshold filtering is performed. First, an `nms_pre` operation is used to filter out most predictions with low confidence (e.g., if the `score_thr` threshold is set to 0.05, results with a current prediction confidence below 0.05 are removed). Then, the bounding box coordinates, the coordinates of its corresponding grid cell, the confidence score, and the label information are obtained, resulting in the final detection result. Specifically, after threshold filtering, a process of restoring to the original image scale is performed, mapping the network's prediction results onto the entire image to obtain the bounding box's coordinates. Finally, an `nms` operation is performed, and the final prediction return value is the post-processed detection result for each ear image, including the classification confidence score, the bounding box's labels, and the four coordinates of the bounding box.

[0082] For ear keypoint detection models, the RTMPose model can be built, with CSPNEXt as the backbone network, achieving a good balance between prediction speed and accuracy, and is easy to deploy. Furthermore, RTMPose uses an algorithm based on SimCC to transform keypoint localization into a classification task, eliminating the reliance on high-resolution heatmaps and achieving a very compact architecture that avoids the loss of spatial information, offering significant advantages for building lightweight pose estimation models.

[0083] like Figure 4 The diagram shown is an example of the detection process of the ear keypoint detection model according to an embodiment of the present invention. An image with the ear outline already labeled is input. A 7×7 fully connected layer is used to extend the 1D keypoint representation to the required dimension controlled by hyperparameters. Simultaneously, a self-attention module, i.e., an FC layer, is used to refine the keypoint representation. A transformer variant, the Gated Attention Unit (GAU), is employed. GAU improves upon the transformer layer feedforward network (FFN) with gated linear units (GLU) and integrates the attention mechanism.

[0084] Coordinate classifiers, specifically X-axis and Y-axis classifiers, treat keypoint localization as a classification problem. The core idea is to divide the horizontal and vertical axes into equally wide bins and discretize continuous coordinates into complete bin labels. The model is then trained to predict the bin in which a keypoint belongs. The coordinate classifier uses a 1×1 convolutional layer to convert the features extracted from the backbone into vectorized keypoint representations, and two fully connected (FC) layers perform classification separately, thus obtaining the keypoint detection coordinates and their classification confidence scores.

[0085] To ensure the detection and recognition performance of the model, a top-down approach can be used to train the aforementioned ear target detection model and ear keypoint detection model in supervised learning to automatically label the auricle keypoints in photos. The top-down keypoint estimation algorithm first detects the target bounding box, and then predicts the target keypoints within each bounding box. To obtain a high-performance ear keypoint detection model, AudioEar2D can be used as the auricle keypoint annotation dataset. This dataset was collected outdoors and includes 2000 high-resolution ear images. Each image is annotated with 55 keypoints according to the IBug Ear standard, and the image is noted as clean, whether it is obscured by hair, and whether earrings are worn. To avoid earrings and hair obscuring keypoints in the images, all data with earrings and significant hair obscuring the auricle were removed, resulting in a total of 1593 human ear images.

[0086] The key points of the ear are marked as follows: Figure 5 The image shown is an example of ear key points. Ear key point detection models trained using such datasets can accurately identify key points such as… Figure 5 The key points shown are crucial for effectively representing the overall outline and related parameters, thus ensuring the modeling effect of the subsequent 3D ear model of the target.

[0087] By using an ear target detection model to identify the ear outline in an image, and by using an ear key point detection model to further determine the ear key points, the ear recognition effect and the accuracy of key point determination can be further guaranteed, thereby ensuring the accuracy of the target 3D ear model obtained through modeling.

[0088] Step S202: Model the ear based on key points to obtain the target three-dimensional ear model corresponding to the target image.

[0089] Specifically, in step S202, modeling is performed based on key points of the ear to obtain a target 3D ear model corresponding to the target image, including:

[0090] Determine the location information corresponding to key points of the ear, and determine the auricle information in the target image based on the location information;

[0091] Based on auricular information, a three-dimensional reconstruction of the ear is performed to obtain a target three-dimensional ear model corresponding to the target image.

[0092] This can be understood as follows: after determining the key points of the ear in the target image, based on the location of these key points, we determine the key points belonging to the auricle region, thereby determining the auricle information of the target object, such as the specific physiological parameters of the auricle. Figure 5 For example, the key points of the auricle are those numbered 0 to 19. In one example, after identifying the key points of the ear in the target image, these key points can be automatically labeled using an auricle mask.

[0093] After determining the relevant information about the auricle, the auricle information can be processed using a preset 3D reconstruction model to perform 3D reconstruction of the ear and obtain the target 3D ear model corresponding to the ear in the target image.

[0094] By determining the auricle information in the target image through the location information of key points, a more accurate three-dimensional ear model of the target can be obtained through three-dimensional reconstruction, which further ensures the matching degree of the target head-related transfer function determined subsequently.

[0095] Step S203: Compare the target 3D ear model with the sample 3D ear models in the preset database to determine the sample 3D ear model that is closest to the target 3D ear model. The preset database includes: multiple different sample 3D ear models and corresponding head-related transfer functions.

[0096] The three-dimensional ear model is a three-dimensional point cloud model. Specifically, step S203 includes:

[0097] Step S2031: Preprocess the target 3D ear model and the sample 3D ear models in the preset database to unify the number of point clouds, size and orientation of the target 3D ear model and the sample 3D ear models.

[0098] This can be understood as follows: both the sample 3D ear model and the target 3D ear model in the preset database are 3D point cloud models, but they may differ in format. For example, the number of 3D point clouds, model size, and model orientation may differ between the target and sample 3D ear models. The difference in model orientation can be understood through an example: the target 3D ear model may be oriented forward, while the sample 3D ear model may be oriented at an angle, leading to a difference in model orientation. Therefore, preprocessing is necessary before subsequent comparison calculations to reduce errors and improve the accuracy of similarity calculations.

[0099] Specifically, in step S2031, the target 3D ear model and the sample 3D ear models in the preset database are preprocessed, including:

[0100] The number of point clouds in the target 3D ear model and the sample 3D ear model is unified by resampling;

[0101] The size of the target 3D ear model and the sample 3D ear model is unified by a scaling factor;

[0102] The model orientation of the target 3D ear model and the sample 3D ear model is unified by the average transformation matrix.

[0103] This can be understood as follows: First, resample the target 3D ear model and the sample 3D ear model to ensure a uniform number of 3D point clouds in each model. Second, scale the model using a scaling factor to ensure consistent size of the corresponding 3D point clouds. Third, adjust the model's orientation using an average transformation matrix to achieve consistent orientation; specifically, the orientation can be adjusted based on the plane connecting the ear and face. It's important to note that these three preprocessing steps are not fixed. In some cases, the 3D point cloud formats of the sample and target 3D ear models are similar or identical. In such cases, the preprocessing steps can be reduced or omitted.

[0104] Preprocessing the target 3D ear model and the sample 3D ear model by resampling, scaling factor, and average transformation matrix can further reduce the error during calculation and improve the accuracy of similarity calculation.

[0105] Step S2032: Calculate the similarity between the target 3D ear model and the sample 3D ear models in the preset database and their corresponding 3D point clouds.

[0106] Based on the similarity between each sample 3D ear model and the target 3D ear model, the sample 3D ear model that is closest to the target 3D ear model is determined.

[0107] This can be understood as follows: after preprocessing, the format of the target 3D ear model and the sample 3D ear model is unified. At this time, the similarity between the target 3D ear model and each sample 3D ear model in the database is calculated. Based on the similarity between each sample 3D ear model and the target 3D ear model, the sample 3D ear model that is closest to the target 3D ear model is determined.

[0108] By calculating the similarity between the target 3D ear model and the sample 3D ear model, the sample 3D ear model that is closest to the target 3D ear model is determined, thereby ensuring that the head-related transfer function determined subsequently can meet the personalized needs of the target object.

[0109] Specifically, similarity calculations are performed on the 3D point clouds corresponding to the target 3D ear model and the sample 3D ear models in the preset database, including:

[0110] The distance between the target 3D ear model and the sample 3D ear model in the preset database is calculated by using the point cloud iterative nearest neighbor algorithm to obtain the average nearest neighbor distance for each sample 3D ear model.

[0111] The similarity between each sample 3D ear model and the target 3D ear model is determined based on the average nearest neighbor distance. The smaller the average nearest neighbor distance of the sample 3D ear model, the higher its similarity to the target 3D ear model.

[0112] This can be understood as using the point cloud iterative nearest neighbor algorithm to calculate the average nearest neighbor distance of each sample 3D ear model relative to the target 3D ear model.

[0113] The specific calculation formula is as follows:

[0114]

[0115] Among them, P j This represents the 3D point cloud data corresponding to the sample 3D ear model in the preset database, where N is the number of sampling points and p i p represents the points after resampling of the target 3D ear model. j,k This represents the points after resampling of the 3D ear model of the sample.

[0116] After calculating the average nearest neighbor distance of each sample 3D ear model relative to the target 3D ear model using the above formula, the average nearest neighbor distance can be used to reflect the similarity between 3D models. The smaller the average nearest neighbor distance, the higher the similarity between the sample and the target 3D ear model.

[0117] By using the point cloud iterative nearest neighbor algorithm to calculate the nearest neighbor distance, the accuracy of the similarity calculation between the target 3D ear model and the sample 3D ear model can be effectively guaranteed.

[0118] To facilitate understanding of the overall process of step S203 above, an example diagram of the ear model comparison process is provided, such as... Figure 6 As shown,

[0119] First, the reconstructed ear model of the subject and the sample 3D ear model (i.e., the target 3D ear model and the sample 3D ear model mentioned above) are resampled to unify the number of 3D point clouds corresponding to the models. Next, the model size is adjusted using a scaling factor until the two models are equal in size. Then, an average transformation matrix is ​​used to coarsely register the two ear models based on the plane connecting the ear and face, i.e., orientation adjustment, to unify the orientation of the models. Finally, the Iterative Closest Point (ICP) algorithm is used to calculate the average nearest neighbor distance after the ear model registration as a similarity measure. This determines the similarity between each sample 3D ear model and the target 3D ear model.

[0120] In one example, the HUTUBS database, publicly available from the Technical University of Berlin, provides both human physiological parameters and head scan models. The ear portion was manually separated from the head scan model to calculate the matching error between the reconstructed ear and the database ear. During the separation and calibration process, the contact area between the ear and the head was minimized as much as possible, and the edges were smoothed to avoid matching errors introduced by non-auricular areas.

[0121] Step S204: The head-related transfer function corresponding to the closest sample 3D ear model is determined as the target head-related transfer function of the target object. See details in [link to documentation]. Figure 1 Step S104 of the illustrated embodiment will not be described again here.

[0122] This invention processes images containing the ears of a target object using a trained ear target detection model and an ear keypoint detection model to obtain the corresponding ear keypoint location information. This information is then used for 3D modeling to obtain a target 3D ear model. The model is then compared with sample 3D ear models in a pre-set database to determine the closest sample 3D ear model. The corresponding head-related transfer function (HRT) of this sample 3D ear model is then determined as the HRT corresponding to the target object's ear. This significantly reduces the cost of obtaining personalized HRT functions. Furthermore, through precise similarity calculation, it ensures that the matched sample 3D ear model and the target object's ear physiological characteristics are extremely close, guaranteeing that the HRT corresponding to the matched sample 3D ear model meets the target object's personalized needs.

[0123] To facilitate understanding of the overall content of the above embodiments, an example diagram of the overall process of ear model matching is provided, such as... Figure 7As shown, firstly, an image of the target user's ear is acquired and preprocessed. This preprocessing refers to removing occlusions that may be present in some images. Next, a pre-trained ear target detection model and ear keypoint detection model are used to process the input image, extracting keypoint location information. Specifically, the ear target detection model and ear keypoint detection model mentioned above are used to process the image and obtain the ear's keypoint location information. Then, a 3D ear reconstruction module performs 3D reconstruction based on the keypoint location information. Specifically, an existing ear reconstruction network can be used to reconstruct the ear model. Finally, a three-stage registration scheme is used for matching, corresponding to step S203 above. This three-stage registration scheme can be understood as resampling and adjusting the size, adjusting the model orientation using an average transformation matrix, and calculating similarity using a point cloud iterative nearest-neighbor algorithm. Finally, the nearest neighbor ear is matched, resulting in the closest sample 3D ear model.

[0124] This invention uses readily available photographs of the subject's ears as input, automatically extracting parameters and features and reconstructing the ear model. A three-stage registration scheme is used to further optimize the loss function, resulting in the best judgment. Furthermore, a lightweight deep neural network-based algorithm for ear target detection and ear keypoint detection is designed and built. This algorithm can automatically detect key points of the auricle in a single image using a pre-trained model, and annotate the relative positions of various anatomical structures of the ear. The total number of parameters and computational complexity of the training model are low, achieving an optimal balance between real-time computation and detection accuracy.

[0125] This invention utilizes pose detection and 3D reconstruction technology. By employing a strategy of nearest neighbor matching in 3D using a reconstructed ear model, it avoids the perspective bias and other problems inherent in traditional methods that use a single view to calibrate auricular parameters in 2D. From a portability and practicality perspective, this system only requires a single ear photograph from the subject as input to automatically reconstruct the ear model, rather than a difficult-to-obtain 3D scan model, thus reducing the system's computational requirements and deployment complexity.

[0126] This invention provides a modular processing method for quickly and conveniently obtaining a user-customized HRTF from a single ear photograph. The method automatically detects the ear target and extracts key ear point information from the photograph, estimates human physiological parameters, and performs ear reconstruction. By performing nearest neighbor matching with a database, an HRTF that meets the target's personalized needs can be obtained. The accuracy of human measurement parameter estimation and the personalization performance of HRTF achieve levels comparable to or even better than existing technologies, and its low cost and portability are unmatched by traditional measurement methods. The innovative application of 3D reconstruction technology to solve the HRTF personalization problem effectively meets the low spectral difference requirements of personalization algorithms and can be applied to related fields such as binaural spatial audio, sound source localization, in-vehicle spatial audio, and VR / AR audio.

[0127] In one example, for the ear target detection model in the above embodiment, its detection performance can be evaluated using the mean Average Precision (mAP). The overlap ratio between the ear bounding box identified by the ear target detection model and the actual bounding box is calculated, i.e., the Intersection over Union (IoU). The IoU is used as the threshold for distinguishing between positive and negative samples in target detection. The calculation of Average Precision (AP) is essentially equivalent to calculating the area under the Precision-Recall curve, but with slight differences. The PR curve needs to be smoothed first. Specifically, when the same Recall value appears, only the point with the highest Precision for that Recall value is retained.

[0128] In one example, the performance of a keypoint detector model can be evaluated based on the mean average recall (mAR) and average average return (mAP), but the calculation methods are slightly different. Object keypoint similarity (OKS) is the similarity between a predicted keypoint and a keypoint in its ground truth. Similar to IoU, it is used to set a threshold for filtering.

[0129] In a specific example, the core network parameters of the ear target detection model trained in this patent are 4.87M, and the number of floating-point operations per second is 8.02G. The mAP is 0.968 at IoU=0.5; the core network parameters of the ear keypoint detection model trained in this patent are 6.39M. The mAP is 0.638 at OKS=0.5. Therefore, it is evident that the ear target detection model and the ear keypoint detection model in this patent have extremely high recognition accuracy.

[0130] In one example, the error between the matched HRTF and the true HRTF can be measured using the log-spectral distance (LSD). At a given elevation angle... With azimuth angle θ, and The LSD is defined as:

[0131]

[0132] Among them, f h and f l These are the upper and lower limits of the frequency band, respectively.

[0133] In a specific example, the HRTF determined by the above evaluation method in the embodiments of the present invention is calculated, and the corresponding LSD is shown in the table below:

[0134] Method (dB) 1.16 1.85 2.04 1.93 3.59 5.12 6.51

[0135] It can be seen that the HRTF matched by the embodiments of the present invention has a high degree of matching, which can meet the personalized needs of the target object.

[0136] In this embodiment, a header-related transfer function determination apparatus is also provided. This apparatus is used to implement the above embodiments and preferred embodiments, and details already described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0137] This embodiment provides a head-related transfer function determination device, such as... Figure 8 As shown, it includes:

[0138] The ear recognition module 401 is used to acquire a target image containing the ear of the target object, recognize the ear border of the target image, and determine the ear key points within the ear border.

[0139] The 3D modeling module 402 is used to model based on key points of the ear to obtain a target 3D ear model corresponding to the target image;

[0140] The model comparison module 403 is used to compare the target 3D ear model with the sample 3D ear models in the preset database to determine the sample 3D ear model that is closest to the target 3D ear model. The preset database includes: multiple different sample 3D ear models and corresponding head-related transfer functions.

[0141] The head-related transfer function determination module 404 is used to determine the head-related transfer function corresponding to the closest sample three-dimensional ear model as the target head-related transfer function of the target object.

[0142] In some optional implementations, modeling is performed based on ear key points to obtain a target 3D ear model corresponding to the target image, including:

[0143] Determine the location information corresponding to key points of the ear, and determine the auricle information in the target image based on the location information;

[0144] Based on auricular information, a three-dimensional reconstruction of the ear is performed to obtain a target three-dimensional ear model corresponding to the target image.

[0145] In some alternative implementations, the three-dimensional ear model is a three-dimensional point cloud model;

[0146] The target 3D ear model is compared with sample 3D ear models in a pre-set database to determine the sample 3D ear model that is closest to the target 3D ear model, including:

[0147] Similarity calculation is performed on the 3D point clouds corresponding to the target 3D ear model and the sample 3D ear models in the preset database.

[0148] Based on the similarity between each sample 3D ear model and the target 3D ear model, the sample 3D ear model that is closest to the target 3D ear model is determined.

[0149] In some optional implementations, similarity calculations are performed on the 3D point clouds corresponding to the target 3D ear model and the sample 3D ear models in a preset database, including:

[0150] The distance between the target 3D ear model and the sample 3D ear model in the preset database is calculated by using the point cloud iterative nearest neighbor algorithm to obtain the average nearest neighbor distance for each sample 3D ear model.

[0151] The similarity between each sample 3D ear model and the target 3D ear model is determined based on the average nearest neighbor distance. The smaller the average nearest neighbor distance of the sample 3D ear model, the higher its similarity to the target 3D ear model.

[0152] In some optional implementations, before calculating the similarity between the 3D point clouds corresponding to the target 3D ear model and the sample 3D ear models in a preset database, the method further includes:

[0153] Preprocessing is performed on the target 3D ear model and the sample 3D ear models in the preset database to unify the number of point clouds, size, and model orientation of the target 3D ear model and the sample 3D ear models.

[0154] In some optional implementations, preprocessing is performed on the target 3D ear model and sample 3D ear models in a pre-defined database, including:

[0155] The number of point clouds in the target 3D ear model and the sample 3D ear model is unified by resampling;

[0156] The size of the target 3D ear model and the sample 3D ear model is unified by a scaling factor;

[0157] The model orientation of the target 3D ear model and the sample 3D ear model is unified by the average transformation matrix.

[0158] In some alternative implementations, identifying the ear border of the target image and determining ear key points within the ear border includes:

[0159] The target image is identified using an ear target detection model, and the corresponding ear bounding box is obtained.

[0160] The ear border is detected using an ear key point detection model to determine the ear key points within the ear border.

[0161] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0162] In this embodiment, the head-related transfer function determination device is presented in the form of a functional unit. Here, a unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above-mentioned functions.

[0163] This invention also provides a computer device having the above-described features. Figure 8 The head-related transfer function determination device is shown.

[0164] Please see Figure 9 , Figure 9 This is a schematic diagram of the structure of a computer device provided in an optional embodiment of the present invention, such as... Figure 9As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 9 Take a processor 10 as an example.

[0165] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0166] The memory 20 stores instructions executable by at least one processor 10 to cause at least one processor 10 to perform the method shown in the above embodiments.

[0167] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0168] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0169] The computer device also includes an input device 30 and an output device 40. The processor 10, memory 20, input device 30, and output device 40 can be connected via a bus or other means. Figure 9 Taking the example of a connection between China and Israel via a bus.

[0170] Input device 30 can receive input numerical or character information, and generate key signal inputs related to user settings and function control of the computer device, such as a touchscreen, keypad, mouse, trackpad, touchpad, joystick, one or more mouse buttons, trackball, joystick, etc. Output device 40 may include display devices, auxiliary lighting devices (e.g., LEDs), and haptic feedback devices (e.g., vibration motors). The aforementioned display devices include, but are not limited to, liquid crystal displays, light-emitting diodes, displays, and plasma displays. In some alternative embodiments, the display device may be a touchscreen.

[0171] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0172] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A method for determining a head-related transfer function, characterized in that, The method includes: Obtain a target image containing the ear of the target object, identify the ear border of the target image, and determine the key points of the ear within the ear border; Based on the key points of the ear, a target three-dimensional ear model corresponding to the target image is obtained; Similarity calculation is performed on the 3D point clouds corresponding to the target 3D ear model and the sample 3D ear models in the preset database. The preset database includes: multiple different sample 3D ear models and corresponding head-related transfer functions. Based on the similarity between each sample 3D ear model and the target 3D ear model, the sample 3D ear model that is closest to the target 3D ear model is determined. The head-related transfer function corresponding to the closest sample 3D ear model is determined as the target head-related transfer function of the target object; The step of calculating the similarity between the target 3D ear model and the sample 3D ear models in the preset database, including: The distance between the target 3D ear model and the sample 3D ear model in the preset database is calculated by using the point cloud iterative nearest neighbor algorithm to obtain the average nearest neighbor distance for each sample 3D ear model. The similarity between each sample 3D ear model and the target 3D ear model is determined based on the average nearest neighbor distance. The smaller the average nearest neighbor distance of the sample 3D ear model, the higher its similarity to the target 3D ear model.

2. The method according to claim 1, characterized in that, The process of modeling based on the ear key points to obtain the target 3D ear model corresponding to the target image includes: Determine the location information corresponding to the key points of the ear, and determine the auricle information in the target image based on the location information; Based on the auricle information, a three-dimensional reconstruction of the ear is performed to obtain the target three-dimensional ear model corresponding to the target image.

3. The method according to claim 1, characterized in that, Before calculating the similarity between the target 3D ear model and the corresponding 3D point clouds of sample 3D ear models in the preset database, the method further includes: Preprocessing is performed on the target 3D ear model and the sample 3D ear models in the preset database to unify the number of point clouds, size, and model orientation of the target 3D ear model and the sample 3D ear models.

4. The method according to claim 1, characterized in that, The preprocessing of the target 3D ear model and the sample 3D ear models in the preset database includes: The number of point clouds in the target 3D ear model and the sample 3D ear model is unified by resampling; The size of the target 3D ear model and the sample 3D ear model is unified by a scaling factor; The model orientation of the target 3D ear model and the sample 3D ear model is unified by the average transformation matrix.

5. The method according to claim 1, characterized in that, The step of identifying the ear border of the target image and determining the ear key points within the ear border includes: The target image is identified using an ear target detection model to obtain the corresponding ear bounding box; The ear border is detected using an ear key point detection model to determine the ear key points within the ear border.

6. A head-related transfer function determination device, characterized in that, The device includes: An ear recognition module is used to acquire a target image containing the ear of a target object, identify the ear border of the target image, and determine the ear key points within the ear border. A 3D modeling module is used to model based on the key points of the ear to obtain a target 3D ear model corresponding to the target image; The model comparison module is used to compare the target 3D ear model with sample 3D ear models in a preset database to determine the sample 3D ear model that is closest to the target 3D ear model. The preset database includes: multiple different sample 3D ear models and corresponding head-related transfer functions. The head-related transfer function determination module is used to determine the head-related transfer function corresponding to the closest sample three-dimensional ear model as the target head-related transfer function of the target object; The model comparison module is also used to calculate the similarity between the target 3D ear model and the sample 3D ear models in the preset database and their corresponding 3D point clouds; and to determine the sample 3D ear model that is closest to the target 3D ear model based on the similarity between each sample 3D ear model and the target 3D ear model. The step of calculating the similarity between the target 3D ear model and the sample 3D ear models in the preset database includes: calculating the distance between the target 3D ear model and the sample 3D ear models in the preset database using a point cloud iterative nearest neighbor algorithm to obtain the average nearest neighbor distance for each sample 3D ear model; determining the similarity between each sample 3D ear model and the target 3D ear model based on the average nearest neighbor distance, wherein the smaller the average nearest neighbor distance of the sample 3D ear model, the higher its similarity to the target 3D ear model.

7. A computer device, characterized in that, include: A memory and a processor are communicatively connected, the memory storing computer instructions, and the processor executing the computer instructions to perform the head-related transfer function determination method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a computer to perform the header-related transfer function determination method according to any one of claims 1 to 5.

Images