Audio processing method and apparatus, device and storage medium

Abstract

An audio processing method and apparatus, a device and a storage medium. The method comprises: acquiring at least one ear image of a target user and an audio to be processed; determining depth information of the ear image on the basis of the at least one ear image; on the basis of the depth information, determining target acoustic transmission features corresponding to the target user, the target acoustic transmission features being at least used for indicating acoustic features exhibited during the process in which sound propagates to the ears of the target user; and, on the basis of the target acoustic transmission features, processing the audio to be processed, so as to generate target audio content corresponding to the audio to be processed. In this way, the depth information of the ear image of the target user is used to generate the personalized acoustic transmission features for the target user, and the audio to be processed is processed on the basis of the personalized acoustic transmission features to generate the target audio content, such that the sound source localization ability of the target user can be effectively improved, thus improving the immersive experience and preference level of the target user.

Description

Audio processing method, apparatus, device, and storage medium

[0001] This application claims priority to the Chinese patent application No. 202411267190.3, filed on September 10, 2024, entitled “Audio processing method, apparatus, device, and storage medium”, the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0002] Example embodiments of the present disclosure generally relate to the field of computers, and in particular, to an audio processing method, apparatus, device, and computer-readable storage medium. BACKGROUND

[0003] With the continuous progress of audio processing technology, it brings users a more rich and immersive auditory experience, especially in the fields of multimedia and virtual reality, etc. play an increasingly important role, such as rendering audio to generate target audio that can allow users to improve the sense of immersion. How to process audio is the focus of attention. SUMMARY

[0004] In a first aspect of the present disclosure, an audio processing method is provided. The method comprises: obtaining at least one ear image of a target user and audio to be processed; determining depth information of the ear image based on the at least one ear image; determining a target acoustic transmission characteristic corresponding to the target user based on the depth information, the target acoustic transmission characteristic being at least used to indicate an acoustic characteristic in a process of sound propagating to the ear of the target user; and processing the audio to be processed based on the target acoustic transmission characteristic to generate target audio content corresponding to the audio to be processed.

[0005] In a second aspect of the present disclosure, an apparatus for audio processing is provided. The apparatus comprises: an obtaining module configured to obtain at least one ear image of a target user and audio to be processed; a first determining module configured to determine depth information of the ear image based on the at least one ear image; a second determining module configured to determine a target acoustic transmission characteristic corresponding to the target user based on the depth information, the target acoustic transmission characteristic being at least used to indicate an acoustic characteristic in a process of sound propagating to the ear of the target user; and a generating module configured to process the audio to be processed based on the target acoustic transmission characteristic to generate target audio content corresponding to the audio to be processed.

[0006] In a third aspect of the present disclosure, an electronic device is provided. The device comprises at least one processor; and at least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor. The instructions, when executed by the at least one processor, cause the device to perform the method of the first aspect.

[0007] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided. The computer-readable storage medium has stored thereon computer-executable instructions that, when executed by a processor, implement the method of the first aspect.

[0008] In a fifth aspect of the present disclosure, a computer program product is provided. The computer program product includes computer-executable instructions that, when executed by a processor, implement the method of the first aspect.

[0009] It is to be understood that the details set forth herein are by way of example and not intended to limit the scope of the subject disclosure, but rather to provide further appreciation of the subject disclosure. Other features and advantages of the subject disclosure will become apparent from the following detailed description taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0010] The above and other features, aspects and advantages of embodiments of the present disclosure will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which like reference characters designate like elements in which:

[0011] FIG. 1 shows a schematic diagram of an example environment in which embodiments of the present disclosure can be implemented;

[0012] FIG. 2 shows a flowchart of a process of audio processing according to some embodiments of the present disclosure;

[0013] FIG. 3 shows a schematic diagram of a target model structure according to some embodiments of the present disclosure;

[0014] FIG. 4 shows a schematic diagram of structures of a first model and a second model according to some embodiments of the present disclosure;

[0015] FIG. 5 shows an example flowchart of audio processing according to some embodiments of the present disclosure;

[0016] FIG. 6 shows a schematic structural block diagram of an apparatus for audio processing according to certain embodiments of the present disclosure;

[0017] FIG. 7 shows a block diagram of an electronic device capable of implementing various embodiments of the present disclosure. DETAILED DESCRIPTION

[0018] Embodiments of the present disclosure will be described herein below with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be embodied in various forms and should not be construed as being limited to the embodiments set forth herein; rather, these embodiments are provided so as to more completely and thoroughly understand the present disclosure. It is to be understood that the drawings and embodiments of the present disclosure are merely for illustrative purposes and should not be construed as limiting the scope of protection of the present disclosure.

[0019] It should be noted that the headings provided herein are for convenience only and are not to be construed as limiting. Various embodiments are described herein, and any type of embodiment can be included under any section. Further, embodiments described in any section can be combined with any other embodiment described in the same section and / or a different section in any manner.

[0020] In the description of embodiments of the disclosure, the term "includes" and its synonyms shall be understood as open-ended, i.e., "including but not limited to". The term "based on" shall be understood as "based at least in part on". The term "one embodiment" or "the embodiment" shall be understood as "at least one embodiment". The term "some embodiments" shall be understood as "at least some embodiments". Other explicit and implicit definitions can also be included below. The terms "first", "second", etc. can refer to different or the same objects. Other explicit and implicit definitions can also be included below.

[0021] Data of users, acquisition and / or use of data, etc. can be involved in embodiments of the disclosure. These aspects all comply with corresponding laws and regulations and relevant provisions. In embodiments of the disclosure, all data collection, acquisition, processing, processing, forwarding, use, etc. are performed on the premise that the user is aware of and confirms. Accordingly, when implementing embodiments of the disclosure, the type of data or information that can be involved, the use range, the use scenario, etc. should be notified to the user and authorized by the user according to relevant laws and regulations through appropriate means. The specific notification and / or authorization manner can vary according to the actual situation and application scenario, and the scope of the disclosure is not limited in this respect.

[0022] In the description and embodiments of the present specification, if personal information processing is involved, it will be processed on the premise of legality (for example, obtaining the consent of the subject of personal information, or being necessary for the performance of a contract, etc.), and only within the prescribed or agreed range. Users refuse to process personal information other than the necessary information required for basic functions, which will not affect the user's use of basic functions.

[0023] Conventionally, in a spatial audio playback system, to virtually reproduce a sound from a certain position, a common practice is to filter a signal via an acoustic transfer characteristic, which can be a head-related transfer function (HRTF), to obtain ear audio for playback. The acoustic transfer characteristic is used to describe the characteristics of sound waves propagating from a specific position to the eardrum of a human ear in a free field, which includes many physical phenomena such as reflection, scattering, and diffraction of sound waves on the human torso, head, and pinna. A user can locate a sound based on the frequency and phase differences of the sound received by the eardrum. Since different cues for sound localization have certain commonalities, in spatial audio playback, a common acoustic transfer characteristic is used for audio processing in the related art. Since the human body characteristics of each person are different, the common acoustic transfer characteristic can impair the user's ability to locate a sound source, especially in the vertical direction, and also cannot bring an immersive experience to the user.

[0024] Embodiments of the present disclosure provide an audio processing scheme. According to the scheme, at least one ear image of a target user and to-be-processed audio are obtained; depth information of the ear image is determined based on the at least one ear image; a target acoustic transfer characteristic corresponding to the target user is determined based on the depth information, the target acoustic transfer characteristic being used to at least indicate an acoustic characteristic in a process of sound propagating to the ear of the target user; and the to-be-processed audio is processed based on the target acoustic transfer characteristic to generate target audio content corresponding to the to-be-processed audio.

[0025] Based on such a manner, embodiments of the present disclosure can generate a personalized acoustic transfer characteristic for a target user by using depth information of an ear image of the target user, and process to-be-processed audio based on the personalized acoustic transfer characteristic to generate target audio content, which can effectively improve the sound source localization ability of the target user, improve the immersive experience and preference of the target user.

[0026] Example environment

[0027] FIG. 1 shows a schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented. As shown in FIG. 1, the example environment 100 can include an electronic device 110.

[0028] In the example environment 100, the electronic device 110 can obtain at least one ear image of a target user and to-be-processed audio. The electronic device 110 can process the to-be-processed audio based on the at least one ear image to generate target audio content corresponding to the to-be-processed audio. The target audio content has the characteristics of enhanced spatial sense, improved positioning accuracy, and improved immersion, and the like, relative to the to-be-processed audio, which will not be described here.

[0029] The target audio content can be played back based on any suitable speaker-equipped device, such as a head-wearable device.

[0030] The electronic device 110 can be a head-wearable device, and can also be any type of mobile terminal, fixed terminal, or portable terminal including a mobile telephone, a tablet computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a media computer, a multimedia tablet, a palmtop computer, a portable gaming terminal, a VR / AR device, a Personal Communication System (PCS) device, a personal navigation device, a Personal Digital Assistant (PDA), an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio broadcast receiver, an electronic book device, a game device, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some embodiments, the electronic device 110 can also support any type of interface to the target user (such as “wearable” circuitry, etc.).

[0031] The electronic device 110 can also be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, content distribution network, and big data and artificial intelligence platform. The electronic device 110 may, for example, include a computing system / server, such as a mainframe, an edge computing node, a computing device in a cloud environment, etc.

[0032] It should be understood that the structure and function of the various elements in the environment 100 are described for illustrative purposes only, without implying any limitation on the scope of the present disclosure.

[0033] Some example embodiments of the present disclosure will be described below with continued reference to the accompanying drawings.

[0034] Example process

[0035] FIG. 2 shows a flowchart of a process 200 of audio processing according to some embodiments of the present disclosure. The process 200 can be implemented at the electronic device 110. The process 200 is described below with reference to FIG. 1.

[0036] At block 210, the electronic device 110 obtains at least one ear image of the target user and audio to be processed.

[0037] In some embodiments, the at least one ear image can be obtained by an image acquisition unit of a head-wearable device.

[0038] In some embodiments, the head-mounted device can be any suitable device deployed with an image capturing unit, such as a Head-Mounted Display (HMD), an Extended Reality (XR) device.

[0039] In some embodiments, the image capturing unit can be any suitable image capturing unit with image capturing function, which can include one or more cameras.

[0040] In some embodiments, the ear image can be an image including an ear of the target user. Further, the ear image can be an image including an auricle region of the target user.

[0041] As an example, the electronic device 110 can obtain two ear images including the ear of the target user by using the binocular camera of the HMD.

[0042] At block 220, the electronic device 110 determines depth information of the ear image based on the at least one ear image.

[0043] In some embodiments, the depth information can be any suitable data about the three-dimensional geometry of the ear, which is used to describe the distance of each point on the ear surface relative to the image capturing unit. The depth information can indicate the three-dimensional shape, size, position, etc. of the ear of the target user. The depth information can be information of any suitable dimension, such as the dimension can be 224*224, etc.

[0044] In some embodiments, in response to the image capturing unit being a binocular camera, the electronic device 110 can obtain two ear images including the target user based on the binocular camera. The electronic device 110 can determine the depth information corresponding to the ear of the target user based on the two ear images including the target user and a predetermined first algorithm. The first algorithm can be any suitable depth information determination algorithm, such as a CREStereo algorithm.

[0045] In some embodiments, the third image type of image corresponding to the ear can be obtained based on a third image type of image including the head. Specifically, the electronic device 110 can obtain the third image type of image including the head. The electronic device 110 can detect the position information of the ear based on the third image type of image including the head. The electronic device 110 can obtain the bounding box corresponding to the ear based on the position information of the ear. The electronic device 110 can obtain the third image type of image corresponding to the ear based on the bounding box corresponding to the ear.

[0046] In some embodiments, the third image type of image corresponding to the ear can be obtained based on a third image type of image including the head. Specifically, the electronic device 110 can obtain the third image type of image including the head. The electronic device 110 can detect the position information of the ear based on the third image type of image including the head. The electronic device 110 can obtain the bounding box corresponding to the ear based on the position information of the ear. The electronic device 110 can obtain the third image type of image corresponding to the ear based on the bounding box corresponding to the ear.

[0047] In block 230, the electronic device 110 determines the target acoustic transmission characteristic corresponding to the target user based on the depth information, the target acoustic transmission characteristic being used to at least indicate the acoustic characteristic in the process of sound propagating to the ear of the target user.

[0048] In some embodiments, the acoustic transmission characteristic can be any appropriate acoustic characteristic that can indicate the process of sound propagating to the ear of the target user. As an example, the acoustic transmission characteristic can be a head-related transfer function characteristic. Of course, the acoustic transmission characteristic can also be a sound source positioning characteristic, a parallax of sound, an intensity difference of sound, etc. In some embodiments, the acoustic transmission characteristic can be a characteristic in the form of wherein represents a spatial position, and f represents an audio frequency. In some embodiments, the acoustic transmission characteristic can be a characteristic of any appropriate dimension, such as the corresponding dimension of which can be 1850*256.

[0049] In some embodiments, the electronic device 110 can determine the acoustic transmission feature corresponding to the target user based on the depth information and a target model. The target model can be any appropriate deep learning network for determining the acoustic transmission feature based on the depth information. The target model can be deployed on the electronic device 110, or on other devices other than the electronic device 110, which will not be described herein.

[0050] The process of determining the acoustic transmission feature based on the target model will be described below.

[0051] In some embodiments, the electronic device 110 can determine the first feature based on the depth information by using a target editor of the target model. The first feature represents the depth feature of the ear image. The target editor is used to reduce the dimension of the depth information to obtain the first feature with a lower dimension than the depth information, i.e., the feature dimension of the first feature is lower than the feature dimension of the depth information. As an example, the electronic device 110 can determine the first feature with a dimension of 256*1 based on the depth information with a dimension of 256*256 by using the target editor.

[0052] The electronic device 110 can determine the second feature based on the first feature by using a target feature matching layer of the target model. The target feature matching layer is mainly used to determine the second feature (acoustic transmission feature) based on the mapping relationship between the depth feature and the acoustic transmission feature and the first feature (depth feature). As an example, the target feature matching layer can be a fully connected layer. The second feature represents the acoustic feature in the process of sound transmission to the ear of the target user, and the feature dimension of the second feature is lower than the feature dimension of the target acoustic transmission feature.

[0053] The electronic device 110 can determine the acoustic transmission feature based on the second feature by using a target decoder of the target model. The target decoder is used to increase the dimension of the second feature output by the target feature matching layer to obtain the acoustic transmission feature with a higher dimension than the second feature. As an example, the electronic device 110 can determine the acoustic transmission feature with a dimension of 1850*256 based on the second feature with a dimension of 256*1 output by the target feature matching by using the target decoder.

[0054] FIG. 3 shows a schematic diagram of a target model structure according to some embodiments of the present disclosure, which will be described with reference to FIG. 3.

[0055] The target model can include a target encoder 310, a target feature matching layer 320, and a target decoder 330.

[0056] The electronic device 110 can obtain, by using the target encoder 310, a feature 302 output by the target encoder 310 based on the depth information 301 of the ear image of the target user. The electronic device 110 can obtain, by using the target feature matching layer 320, a feature 303 output by the target feature matching layer 320 based on the feature 302. The electronic device 110 can obtain, by using the target decoder 330, an acoustic transmission feature 304 output by the target decoder 330 based on the feature 303.

[0057] To obtain a target model with high precision to improve the accuracy of determining the personalized acoustic transmission feature of the target user, taking the training of the target model by the electronic device 110 as an example, the electronic device 110 can train the target model to be trained based on a training sample set to obtain this trained target model. The training process of the target model mainly includes a training sample set acquisition stage and a target model training stage based on the training sample set, which will be further described below.

[0058] The training sample set can include a plurality of training samples, and for each training sample, the training sample can include a sample acoustic transmission feature corresponding to a sample user and sample depth information corresponding to the sample user. The sample acoustic transmission feature at least indicates the acoustic characteristics in the process of sound propagating to the ear of the sample user.

[0059] The acquisition process of the training sample will be described below.

[0060] In some embodiments, the electronic device 110 can determine, based on a pair of first image acquisition devices, a pair of images of a first image type corresponding to the head of the sample user. The first image acquisition device can be any appropriate image acquisition device that can at least support the acquisition of images of the first image type. The first image type can also be any appropriate image type. As an example, the first image acquisition device can be an infrared image acquisition device, and the pair of images of the first image type can be infrared images.

[0061] The electronic device 110 can determine, based on a second image acquisition device, an image of a second image type corresponding to the head of the sample user. The second image acquisition device can be any appropriate image acquisition device that can at least support the acquisition of images of the second image type. The second image type can also be any appropriate image type. As an example, the second image acquisition device can be a color image acquisition device, and the image of the second image type can be a color image. The color image is an image that can display color information, which includes red, green, and blue color channels that can be combined to create a variety of colors. As an example, the image of the second image type can be an RGB image.

[0062] In some embodiments, the electronic device 110 can determine the target depth information based on a pair of images of the first image type. Further, the electronic device 110 can determine the target depth information by stereo matching the pair of images of the first image type based on a predetermined third algorithm. The predetermined third algorithm can be any suitable algorithm for determining depth information, such as a Semi-Global Matching (SGM) algorithm.

[0063] The electronic device 110 can determine the pose information of the camera relative to the human body based on the images of the second image type. The pose information of the camera relative to the human body is used to describe the precise spatial relationship of the camera relative to the human body (or human body part). Further, the electronic device 110 can determine the pose information of the camera relative to the human body based on a predetermined fourth algorithm and the images of the second image type. The predetermined fourth algorithm can be any suitable algorithm for determining the pose information of the camera relative to the human body, such as a Columbia Object Model and Rendering Pipeline (COLMAP) algorithm, etc.

[0064] In some embodiments, the electronic device 110 can determine the head mesh based on the target depth information and the pose information. The head mesh is a mesh model used to represent the geometry of the user's head, which provides a three-dimensional surface representation of the human head and can accurately simulate the shape and features of the head. Further, the electronic device 110 can determine the head mesh based on the target depth information, the pose information, and a predetermined fifth algorithm. The predetermined fifth algorithm can be any suitable algorithm for determining the head mesh, such as a kinect fusion algorithm, etc.

[0065] In some embodiments, the head mesh determined based on the target depth information and the pose information can be a coarse mesh. Further, the electronic device 110 can determine a watertight mesh based on the coarse mesh and a predetermined mesh optimization algorithm. The watertight mesh has more details and higher resolution than the coarse mesh. The predetermined mesh optimization algorithm can be any suitable mesh optimization algorithm, such as a hole filling algorithm, a Poisson reconstruction algorithm, and a combination of these algorithms, etc., which will not be described here.

[0066] In some embodiments, the electronic device 110 can determine the training samples based on the head mesh. Further, the electronic device 110 can determine the sample acoustic transfer characteristics of the sample users in the training samples based on the head mesh and a predetermined algorithm for determining the acoustic transfer characteristics. The predetermined algorithm for determining the acoustic transfer characteristics can be any suitable algorithm, such as a Finite Element Method (FEM) or the like.

[0067] Further, the electronic device 110 can determine the sample depth information in the training samples based on a virtual image acquisition device located at a predetermined position on the head of the sample users and the head mesh. The virtual image acquisition device is used to simulate the function of a camera in the real world, for capturing images in a virtual environment or a three-dimensional scene. The predetermined position can be set according to requirements, for example, the predetermined position can be a predetermined distance outward from the eardrum of the sample user. The predetermined distance can be any suitable distance, such as 0.5 m or the like.

[0068] In order to improve the accuracy of modeling, when acquiring the pair of images of the first image type and the images of the second image type corresponding to the sample user, the sample user can wear a swimming cap so that the hair of the sample user can be covered by the cap. In addition, the sample user can be located on a rotating turntable which rotates at a predetermined speed, at this time, a pair of first image acquisition devices for acquiring the pair of images of the first image type and a second image acquisition device for acquiring the images of the second image type can be triggered at a predetermined frame rate to start the image acquisition operation. The predetermined frame rate can be any suitable frame rate, such as 5 Hz or the like. The resolution of the acquired images can also be any suitable resolution, such as 2048*1500. The predetermined speed can be any suitable speed, such as 1 min / revolution.

[0069] In some embodiments, the pair of first image acquisition devices for acquiring the pair of images of the first image type and the second image acquisition device for acquiring the images of the second image type can be set at any suitable acquisition position, such as adopting a default spatial sampling position, on a sphere 1.5 m away from the center of the human head, and a preset number of uniform sampling points can be set. The preset number can be any suitable number, which can be set according to requirements, such as the preset number can be 1850, at this time, the electronic device 110 can obtain sample acoustic transfer characteristics with a dimension of 1850*256, where 256 is the frequency domain sampling point.

[0070] To improve the accuracy of the training samples, for each sample user, a predetermined number of times of acquisition can be set, and a data verification operation can be performed based on the predetermined number of times of acquisition to verify the accuracy of the data acquisition. To avoid dizziness of the sample user, the rotation direction of the turntable can not be used when the predetermined number of times of acquisition is performed. The predetermined number can be any appropriate number, such as 2 times, and the like.

[0071] In the embodiments of the present disclosure, the head mesh acquisition device is composed of a pair of first image acquisition devices for acquiring a pair of first images of the same type and a second image acquisition device for acquiring a second image of a different type, and the process of calculating the acoustic transmission characteristics through FEM simulation, which realizes efficient acquisition of acoustic transmission characteristics, that is, efficient acquisition of training samples. Compared with the way of scanning the head and torso based on a handheld scanner in the related art, the embodiments of the present disclosure effectively improve the acquisition efficiency and can quickly construct an acoustic transmission characteristic database of an order of magnitude sufficient to provide sufficient training data for target model training.

[0072] In some embodiments, the target decoder of the target model is trained based on the following process:

[0073] The electronic device 110 can obtain a training sample set, and a training sample in the training sample set includes a sample acoustic transmission characteristic corresponding to a sample user, and the sample acoustic transmission characteristic at least indicates a process of sound propagation to an ear of the sample user. The electronic device 110 can determine, by using a first encoder of a first model, a first acoustic transmission characteristic based on the sample acoustic transmission characteristic. The first acoustic transmission characteristic characterizes an acoustic feature in a process of sound transmission to the ear of the sample user. The first encoder is used to learn deep information corresponding to the ear of the target user, and is used to generate an acoustic transmission characteristic of a lower dimension based on the sample acoustic transmission characteristic. Specifically, the first acoustic transmission characteristic corresponds to a first feature dimension, the sample acoustic transmission characteristic corresponds to a second feature dimension, and the first feature dimension is lower than the second feature dimension. The electronic device 110 can determine, by using a first decoder of the first model, a second acoustic transmission characteristic based on the first acoustic transmission characteristic. The second acoustic transmission characteristic characterizes an acoustic feature in a process of sound transmission to the ear of the sample user, the second acoustic transmission characteristic corresponds to a third feature dimension, and the first acoustic transmission characteristic corresponds to a fourth feature dimension, the third feature dimension is higher than the fourth feature dimension. The electronic device 110 can train the first model based on the difference between the first acoustic transmission characteristic and the second acoustic transmission characteristic, and the first decoder in the trained first model is the target decoder.

[0074] Further, the electronic device 110 can determine a first loss value based on the first acoustic transmission feature, the second acoustic transmission feature, and a predetermined first loss function. The first loss function can be any suitable loss function, which is not described herein. The electronic device 110 can train the first model, i.e., adjust the parameters of the first decoder and the first editor, based on the first loss value and a gradient descent method. The electronic device 110 can determine that the trained first model is obtained when a predetermined training condition is met at the current time. The predetermined training condition can be that a training number reaches a predetermined training number, or a training duration reaches a predetermined training duration, or the first loss value reaches a minimum value, and the like, which is not described herein.

[0075] In some embodiments, the first decoder in the trained first model is also a target decoder in the target model.

[0076] In some embodiments, the target encoder and the target feature matching layer of the target model are trained based on the following process:

[0077] The electronic device 110 can determine a first sample depth feature based on the sample depth information using the second encoder in the second model. The first sample depth feature represents a depth feature corresponding to the ear of the sample user. The first sample depth feature corresponds to a fifth feature dimension, and the sample depth information corresponds to a sixth feature dimension. The fifth feature dimension is lower than the sixth feature dimension. The electronic device 110 can determine a second sample depth feature based on the first sample depth feature using the second decoder in the second model. The second sample depth feature corresponds to a seventh feature dimension, and the first sample depth feature corresponds to an eighth feature dimension. The seventh feature dimension is higher than the eighth feature dimension. The electronic device 110 can determine a third acoustic transmission feature based on the first sample depth feature using the feature matching layer in the second model. The third acoustic transmission feature represents an acoustic feature in the process of sound transmission to the ear of the sample user. The third acoustic transmission feature corresponds to a ninth feature dimension, and the ninth feature dimension is lower than the third feature dimension. As an example, the feature matching layer can be a fully connected layer.

[0078] In some embodiments, the electronic device 110 can train the second model based on the difference between the third acoustic transmission feature and the first acoustic transmission feature, the sample depth information, and the difference between the second sample depth feature.

[0079] Further, the electronic device 110 can determine a first sub-loss value based on the third acoustic transmission feature and the first acoustic transmission feature and a predetermined second loss function. The second loss function can be any appropriate loss function. The electronic device 110 can determine a second sub-loss value based on the sample depth information and the second sample depth feature and a predetermined third loss function. The electronic device 110 can determine the second loss value based on the first sub-loss value and the second sub-loss value. Further, the electronic device 110 can determine the sum of the first sub-loss value and the second sub-loss value as the second loss value.

[0080] The electronic device 110 can train the second model based on the second loss value and a gradient descent method, i.e., adjust the parameters of the second decoder, the second editor, and the feature matching layer. The electronic device 110 can determine that the current time has reached a predetermined training condition, and obtain the trained second model. The predetermined training condition can be that the number of training times reaches a predetermined number of training times, or the training time length reaches a predetermined training time length, or the second loss value reaches a minimum value, and the like, which will not be described here.

[0081] In other embodiments, the electronic device 110 can determine a target divergence based on the first sample depth feature and a predetermined divergence algorithm. The electronic device can train the second model based on the difference between the third acoustic transmission feature and the first acoustic transmission feature, the difference between the sample depth information and the second sample depth feature, and the target divergence.

[0082] Further, the electronic device 110 can determine a first sub-loss value based on the third acoustic transmission feature and the first acoustic transmission feature and a predetermined second loss function. The electronic device 110 can determine a second sub-loss value based on the sample depth information and the second sample depth feature and a predetermined third loss function. The electronic device 110 can determine a third loss value based on the first sub-loss value, the second sub-loss value, and the target divergence. Further, the electronic device 110 can determine the sum of the first sub-loss value and the second sub-loss value. The electronic device 110 can determine the difference between this sum and the target divergence, and determine this difference as the third loss value.

[0083] The electronic device 110 can train the second model based on the third loss value and the gradient descent method, i.e., adjust the parameters of the second decoder, the second editor, and the feature matching layer. The electronic device 110 can determine that the current time reaches the predetermined training condition, and obtain the trained second model. The predetermined training condition can be that the number of training times reaches a predetermined number of training times, or the training time length reaches a predetermined training time length, or the second loss value reaches a minimum value, and the like, which will not be described here. In some embodiments, the feature matching layer in the trained second model is the target feature matching layer in the target model, and the second encoder in the trained second model is the target encoder in the target model.

[0084] FIG. 4 shows a structural schematic diagram of the first model and the second model according to some embodiments of the present disclosure. Now, FIG. 4 will be described.

[0085] The first model 430 can include an encoder 410 and a decoder 420. The second model 440 can include an encoder 430, a decoder 440, and a feature matching layer 450.

[0086] The electronic device 110 can obtain, by using the encoder 410, a feature 402 output by the encoder 410 based on a sample acoustic transmission feature 401. The electronic device 110 can obtain, by using the decoder 420, a feature 403 output by the decoder 420 based on the feature 402. The electronic device 110 can determine a first loss value based on a difference between the feature 402 and the feature 403. The electronic device 110 can adjust the parameters of the encoder 410 and the decoder 420 based on the first loss value, and obtain a trained decoder 420 (target decoder).

[0087] The electronic device 110 can obtain, by using the encoder 430, a feature 412 output by the encoder 430 based on a sample depth information 411. The electronic device 110 can obtain, by using the feature matching layer 450, a feature 413 output by the feature matching layer 450 based on the feature 412. The electronic device 110 can obtain, by using the decoder 440, a feature 414 output by the decoder 440 based on the feature 412. The electronic device 110 can determine a first sub-loss value based on a difference between the depth information 411 and the feature 414. The electronic device 110 can determine a second sub-loss value based on a difference between the feature 402 and the feature 413. The electronic device 110 can determine a target divergence based on the feature 412 and a predetermined divergence function. The electronic device 110 can determine a third loss value based on the first sub-loss value, the second sub-loss value, and the target divergence. The electronic device 110 can adjust the parameters of the encoder 430, the decoder 440, and the feature matching layer 450 based on the third loss value, and obtain a trained encoder 430 (target encoder) and a trained feature matching layer 450 (target feature matching layer).

[0088] At block 240, the electronic device 110 processes the to-be-processed audio based on the target acoustic transfer characteristic, to generate target audio content corresponding to the to-be-processed audio.

[0089] In some embodiments, the processing of the to-be-processed audio can be processed by the electronic device 110, or by other devices, such as a spatial audio rendering system.

[0090] Taking the electronic device 110 processing the to-be-processed audio as an example, in order to improve flexibility, in some embodiments, the electronic device 110 can process the to-be-processed audio based on the processing engine and the personalized acoustic transfer characteristic created by the target user, to generate target audio content corresponding to the to-be-processed audio.

[0091] In some embodiments, the electronic device can perform convolution operation based on the to-be-processed audio and the target acoustic transfer characteristic, to generate target audio content corresponding to the to-be-processed audio. Specifically, for the to-be-processed audio x(t), the electronic device 110 can select an acoustic transfer characteristic filter h(t) based on the position of the audio. The electronic device 110 can perform convolution operation on x(t) and h(t) to obtain the rendered audio content y(t).

[0092] In other embodiments, the electronic device 110 can process the to-be-processed audio based on the processing engine and a general acoustic transfer characteristic, to generate audio content corresponding to the to-be-processed audio, in response to the target user not creating a task of personalized acoustic transfer characteristic. The general acoustic transfer characteristic can be determined based on common characteristics of human ears, which can be set according to requirements.

[0093] In some embodiments, the electronic device 110 can output the target audio content to any appropriate playback device, such as earphones or surround sound speaker systems, to provide the target user with the final auditory experience.

[0094] FIG. 5 illustrates an example flowchart of audio rendering according to some embodiments of the present disclosure, which will now be described with reference to FIG. 5.

[0095] The electronic device 110 can obtain binocular images of the target user by using an XR device 501 with a binocular camera. The electronic device 110 can determine depth information corresponding to the ear of the target user based on the binocular images. The electronic device 110 can determine the acoustic transfer characteristic corresponding to the target user based on the depth information and a target model.

[0096] The spatial audio rendering system 502 can process the to-be-processed audio based on the to-be-processed audio and a filter of the acoustic transmission characteristic corresponding to the target user, to generate spatial audio content (target audio content) played by the XR device 501. The electronic device 110 can play back the spatial audio content by using the loudspeakers of the XR device 501.

[0097] It should be noted that there are other appropriate manners for playing back the spatial audio content, for example, the spatial audio content can be played back by using an Ambisonic technology, and the like, which will not be described herein.

[0098] Based on such a manner, embodiments of the present disclosure can generate a personalized acoustic transmission characteristic for the target user by using the depth information of the ear image of the target user, and process the to-be-processed audio based on the personalized acoustic transmission characteristic, to generate the target audio content, which can effectively improve the sound source positioning ability of the target user and improve the immersive experience and preference of the target user.

[0099] Example apparatuses and devices

[0100] Embodiments of the present disclosure further provide a corresponding apparatus for implementing the above method or process. FIG. 6 shows a schematic structural block diagram of an apparatus 600 for audio rendering according to certain embodiments of the present disclosure. The apparatus 600 can be implemented as or included in the electronic device 110 as discussed above. Various modules / components in the apparatus 600 can be implemented by hardware, software, firmware, or any combination thereof.

[0101] As shown in FIG. 6, the apparatus 600 includes an acquisition module 610 configured to acquire at least one ear image of a target user and to-be-processed audio; a first determination module 620 configured to determine depth information of the ear image based on the at least one ear image; a second determination module 630 configured to determine a target acoustic transmission characteristic corresponding to the target user based on the depth information, the target acoustic transmission characteristic being used at least to indicate an acoustic characteristic in a process of sound propagation to the ear of the target user; and a generation module 640 configured to process the to-be-processed audio based on the target acoustic transmission characteristic, to generate target audio content corresponding to the to-be-processed audio.

[0102] In some embodiments, the second determination module 630 is further configured to determine the target acoustic transmission characteristic corresponding to the target user based on the depth information and a target model.

[0103] In some embodiments, the second determining module 630 is further configured to: determine, by using a target editor of the target model, based on the depth information, the first feature, the first feature representing a depth feature of the ear image, and the first feature corresponding to a feature dimension lower than a feature dimension corresponding to the depth information; determine, by using a target feature matching layer of the target model, based on the first feature, the second feature, the second feature representing an acoustic feature in a process of sound transmission to the ear of the target user, and the second feature corresponding to a feature dimension lower than a feature dimension corresponding to the target acoustic transmission feature; and determine, by using a target decoder of the target model, based on the second feature, the target acoustic transmission feature.

[0104] In some embodiments, the target decoder of the target model is trained based on the following process: obtaining a training sample set, a training sample in the training sample set including a sample acoustic transmission feature corresponding to a sample user, the sample acoustic transmission feature indicating at least an acoustic feature in a process of sound propagation to the ear of the sample user; determining, by using the first encoder of the first model, based on the sample acoustic transmission feature, the first acoustic transmission feature, the first acoustic transmission feature representing an acoustic feature in a process of sound transmission to the ear of the sample user, the first acoustic transmission feature corresponding to a first feature dimension, and the sample acoustic transmission feature corresponding to a second feature dimension, the first feature dimension being lower than the second feature dimension; determining, by using the first decoder of the first model, based on the first acoustic transmission feature, the second acoustic transmission feature, the second acoustic transmission feature representing an acoustic feature in a process of sound transmission to the ear of the sample user, the second acoustic transmission feature corresponding to a third feature dimension, and the first acoustic transmission feature corresponding to a fourth feature dimension, the third feature dimension being higher than the fourth feature dimension; and training the first model based on a difference between the first acoustic transmission feature and the second acoustic transmission feature, the first decoder in the trained first model being the target decoder.

[0105] In some embodiments, the training sample includes sample depth information corresponding to the ear of the sample user, and the target encoder of the target model and the target feature matching layer are trained based on the following process: determining, by using a second encoder in the second model, a first sample depth feature based on the sample depth information, the first sample depth feature representing depth features corresponding to the ear of the sample user, the first sample depth feature corresponding to a fifth feature dimension, the sample depth information corresponding to a sixth feature dimension, the fifth feature dimension being lower than the sixth feature dimension; determining, by using a second decoder in the second model, a second sample depth feature based on the first sample depth feature, the second sample depth feature representing depth features corresponding to the ear of the sample user, the second sample depth feature corresponding to a seventh feature dimension, the first sample depth feature corresponding to an eighth feature dimension, the seventh feature dimension being higher than the eighth feature dimension; determining, by using a feature matching layer in the second model, a third acoustic transmission feature based on the first depth sample feature, the third acoustic transmission feature representing acoustic features in the process of sound transmission to the ear of the sample user, the third acoustic transmission feature corresponding to a ninth feature dimension, the ninth feature dimension being lower than the third feature dimension; training the second model based on the difference between the third acoustic transmission feature and the first acoustic transmission feature, the difference between the sample depth information and the second sample depth feature, the feature matching layer in the trained second model being the target feature matching layer, and the second encoder in the trained second model being the target encoder.

[0106] In some embodiments, training the second model based on the difference between the third acoustic transmission feature and the first acoustic transmission feature, the difference between the sample depth information and the second sample depth feature includes: determining a target divergence based on the first sample depth feature and a predetermined divergence algorithm; and training the second model based on the difference between the third acoustic transmission feature and the first acoustic transmission feature, the difference between the sample depth information and the second sample depth feature, and the target divergence.

[0107] In some embodiments, the training sample is determined based on the following manner: determining a pair of images of a first image type corresponding to the head of the sample user based on a pair of first image acquisition devices; determining an image of a second image type corresponding to the head of the sample user based on a second image acquisition device; determining target depth information based on the pair of images of the first image type; determining pose information of a camera relative to a human body based on the image of the second image type; determining a head mesh based on the target depth information and the pose information; and determining sample depth information corresponding to the ear of the sample user in the training sample based on a virtual image acquisition device located at a predetermined position of the head of the sample user and the head mesh; and determining sample acoustic transmission features of the sample user in the training sample based on the head mesh and a predetermined acoustic transmission feature determination algorithm.

[0108] In some embodiments, the apparatus 600 further includes an obtaining module configured to obtain an image of a third image type including the ear of the target user; and a third determining module configured to determine the target depth information based on the image of the third image type.

[0109] In some embodiments, the target acoustic transfer feature includes a head-related transfer function (HRFT) feature and / or the at least one ear image is acquired by an image acquisition unit of a head-mounted device.

[0110] In some embodiments, the generating module 640 is further configured to generate the target audio content corresponding to the audio to be processed by performing a convolution operation on the audio to be processed and the target acoustic transfer feature. The units included in the apparatus 600 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units can be implemented using software and / or firmware, e.g., machine-executable instructions stored on a storage medium. In addition to or alternatively, some or all of the units in the apparatus 600 can be implemented at least partially by one or more hardware logic components. As an example and not by way of limitation, example types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SOCs), complex programmable logic devices (CPLDs), etc.

[0111] FIG. 7 shows a block diagram of an electronic device 700 in which one or more embodiments of the disclosure can be implemented. It should be understood that the electronic device 700 illustrated in FIG. 7 is merely an example and should not be construed to limit the functionality and scope of the embodiments described herein. The electronic device 700 illustrated in FIG. 7 can be used to implement the electronic device 110 illustrated in FIG. 1.

[0112] As shown in FIG. 7, the electronic device 700 is in the form of a general electronic device. The components of the electronic device 700 can include, but are not limited to, one or more processing units or processors 710, a memory 720, a storage device 730, one or more communication units 740, one or more input devices 750, and one or more output devices 760. The processor 710 can be a real or virtual processor and is capable of performing various processes according to programs stored in the memory 720. In a multi-processor system, multiple processors perform computer-executable instructions in parallel to improve the parallel processing capability of the electronic device 700.

[0113] The electronic device 700 typically includes a plurality of computer storage media. Such media can be volatile and / or nonvolatile, removable and / or non-removable, and can be implemented in any method or technology for storage of information and / or data. For example, memory 720 can be implemented using a volatile and / or nonvolatile memory such as random access memory (RAM), cache memory, and / or flash memory. Storage 730 can be implemented using a non-transitory computer-readable medium, such as a solid state memory or a magnetic hard disk, to name a few. The computer-readable medium can be used to store data and / or instructions for use by and / or in connection with one or more computer programs and / or devices. For example, the computer-readable medium can be used for storing training data for training, and / or instructions for implementing the various embodiments of the present disclosure.

[0114] The electronic device 700 can further include additional removable / non-removable, volatile / nonvolatile computer storage media. Although not shown in FIG. 7, a disk drive can be provided for reading from and writing to a removable, non-removable, volatile, and / or non-volatile memory (e.g., a "floppy disk", a "flexible disk", a "hard disk", a "solid state drive", a "flash drive", a "USB flash drive", a "memory stick", a "read only memory" (ROM), an "electrically erasable programmable read only memory" (EEPROM), and / or a "Compact Disk Read Only Memory" (CD-ROM)). In these instances, each drive can be connected to the bus by one or more data media interfaces. The memory 720 can include a computer program product 725 having one or more program modules configured to carry out the various methods or actions of the various embodiments of the present disclosure.

[0115] The communication unit 740 enables communications with other electronic devices over a communication medium. Additionally, the functionality of the components of the electronic device 700 can be implemented in a single computing cluster or a plurality of computer machines that are capable of communicating over a communication connection. As such, the electronic device 700 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network nodes in the networking environment.

[0116] The input device 750 can be one or more input devices such as a mouse, a keyboard, a trackball, etc. The output device 760 can be one or more output devices such as a display, a speaker, a printer, etc. The electronic device 700 can further communicate with one or more external devices (not shown) such as a storage device, a display device, etc. through the communication unit 740, with one or more devices that enable a user to interact with the electronic device 700, or with any devices (e.g., a network card, a modem, etc.) that enable the electronic device 700 to communicate with one or more other electronic devices. Such communication can be enabled by an input / output (I / O) interface (not shown).

[0117] According to an example implementation of the present disclosure, a computer readable storage medium is provided having computer executable instructions stored thereon, where the computer executable instructions are executed by a processor to implement the method described above. According to an example implementation of the present disclosure, a computer program product is also provided that is tangibly stored on a non-transitory computer readable medium and includes computer executable instructions, where the computer executable instructions are executed by a processor to implement the method described above.

[0118] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0119] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0120] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0121] The computer program product of the present disclosure can have a signal including said computer program. This signal can be electronic, electromagnetic, optical, or any other suitable type of signal. Such a signal can be provided through a communication connection, such as electrical wiring, optical fiber, wireless interface, etc. Examples of computer program products include computer program implemented on a personal computer, server, or other networked device. A non-transitory computer readable medium, such as a floppy disk, CD-ROM, DVD-ROM, Blu-ray Disc, hard disk, or memory stick, can also be used to implement the present disclosure. The computer program product of the present disclosure can also be provided as a service to download and use the computer program over a network, such as the Internet.

[0122] Having described several implementations of the present disclosure, it will be clear to those skilled in the art that many modifications, additions, and substitutions are possible without departing from the scope and spirit of the described implementations. Many modifications and variations of the present disclosure are possible in light of the above teachings. It is, therefore, to be understood that within the scope of the appended claims, the present disclosure can be practiced otherwise than as specifically described. While the present disclosure has been described with reference to the implementation figures, it will be understood by those skilled in the art that various changes can be made and equivalents can be substituted for elements thereof without departing from the scope of the present disclosure. In addition, many modifications can be made to adapt a particular situation or material to the teachings of the disclosure without departing from its scope. Therefore, it is contemplated to cover any and all adaptations and modifications falling within the scope of the appended claims. It should also be noted that, in this document, the terms "can", "can not", "may", "may not", "might", "might not", "must", "must not", "need", "need not", "should", "should not", "will", "will not", "would", and "would not" are used in their permissive sense (they mean that there are situations in which the statement can or might be true).

Claims

1. An audio processing method, comprising: obtaining at least one ear image of a target user and audio to be processed; determining, based on the at least one ear image, depth information of the ear image; determining, based on the depth information, a target acoustic transmission characteristic corresponding to the target user, the target acoustic transmission characteristic being indicative of at least an acoustic characteristic in a process of sound propagating to an ear of the target user; and processing the audio to be processed based on the target acoustic transmission characteristic to generate target audio content corresponding to the audio to be processed.

2. The method of claim 1, wherein determining, based on the depth information, the target acoustic transmission characteristic corresponding to the target user comprises: determining, based on the depth information and a target model, the target acoustic transmission characteristic corresponding to the target user.

3. The method of claim 2, wherein determining, based on the depth information and the trained target model, the target acoustic transmission characteristic corresponding to the target user comprises: determining, based on the depth information, a first characteristic using a target editor of the target model, the first characteristic characterizing a depth characteristic of the ear image, a feature dimension corresponding to the first characteristic being lower than a feature dimension corresponding to the depth information; determining, based on the first characteristic, a second characteristic using a target feature matching layer of the target model, the second characteristic characterizing an acoustic characteristic in a process of sound propagating to the ear of the target user, a feature dimension corresponding to the second characteristic being lower than a feature dimension corresponding to the target acoustic transmission characteristic; and determining, based on the second characteristic, the target acoustic transmission characteristic using a target decoder of the target model.

4. The method of claim 3, wherein the target decoder of the target model is trained based on a process comprising: obtaining a training sample set, a training sample in the training sample set comprising a sample acoustic transmission characteristic corresponding to a sample user, the sample acoustic transmission characteristic being indicative of at least an acoustic characteristic in a process of sound propagating to an ear of the sample user; determining, based on the sample acoustic transmission characteristic, a first acoustic transmission characteristic using a first encoder of a first model, the first acoustic transmission characteristic characterizing an acoustic characteristic in a process of sound propagating to the ear of the sample user, a first feature dimension corresponding to the first acoustic transmission characteristic being lower than a second feature dimension corresponding to the sample acoustic transmission characteristic; determining, based on the first acoustic transmission characteristic, a second acoustic transmission characteristic using a first decoder of the first model, the second acoustic transmission characteristic characterizing an acoustic characteristic in a process of sound propagating to the ear of the sample user, a third feature dimension corresponding to the second acoustic transmission characteristic being higher than a fourth feature dimension corresponding to the first acoustic transmission characteristic; and training the first model based on a difference between the first acoustic transmission characteristic and the second acoustic transmission characteristic, the first decoder in the trained first model being the target decoder. ​ ​ ​ 5. The method of claim 4, wherein the training sample comprises sample depth information corresponding to the ear of the sample user, and the target encoder of the target model and the target feature matching layer are trained based on the following process: determining, by a second encoder in the second model, a first sample depth feature based on the sample depth information, the first sample depth feature representing depth features corresponding to the ear of the sample user, the first sample depth feature corresponding to a fifth feature dimension, the sample depth information corresponding to a sixth feature dimension, the fifth feature dimension being lower than the sixth feature dimension; determining, by a second decoder in the second model, a second sample depth feature based on the first sample depth feature, the second sample depth feature representing depth features corresponding to the ear of the sample user, the second sample depth feature corresponding to a seventh feature dimension, the first sample depth feature corresponding to an eighth feature dimension, the seventh feature dimension being higher than the eighth feature dimension; determining, by a feature matching layer in the second model, a third acoustic transmission feature based on the first depth sample feature, the third acoustic transmission feature representing acoustic features in a process of sound transmission to the ear of the sample user, the third acoustic transmission feature corresponding to a ninth feature dimension, the ninth feature dimension being lower than the third feature dimension; training the second model based on a difference between the third acoustic transmission feature and the first acoustic transmission feature, a difference between the sample depth information and the second sample depth feature, the feature matching layer in the trained second model being the target feature matching layer, the second encoder in the trained second model being the target encoder.

6. The method of claim 5, wherein training the second model based on a difference between the third acoustic transmission feature and the first acoustic transmission feature, a difference between the sample depth information and the second sample depth feature comprises: determining a target divergence based on the first sample depth feature and a predetermined divergence algorithm; training the second model based on the difference between the third acoustic transmission feature and the first acoustic transmission feature, the difference between the sample depth information and the second sample depth feature, and the target divergence.

7. The method of claim 4, wherein the training sample is determined based on the following manner: determining a pair of first image type images corresponding to the head of the sample user based on a pair of first image capturing devices; determining a second image type image corresponding to the head of the sample user based on a second image capturing device; determining target depth information based on the pair of first image type images; determining camera pose information relative to the human body based on the second image type image; determining a head mesh based on the target depth information and the pose information; and determining the sample depth information corresponding to the ear of the sample user in the training sample based on a virtual image capturing device located at a predetermined position on the head of the sample user and the head mesh. ​ determine the sample acoustic transfer feature of the sample user in the training sample based on the head mesh and a predetermined algorithm of acoustic transfer feature determination.

8. The method of claim 1, wherein, Before determining the target acoustic transfer feature corresponding to the target user based on the depth information, the method further comprises: obtaining a third image type of image including an ear of the target user; and determining the target depth information based on the third image type of image. 9.The method of claim 1, wherein the target acoustic transfer feature comprises a head related transfer function (HRFT) feature and / or the at least one ear image is acquired by an image acquisition unit of a head-mounted device. 10.The method of claim 1, wherein processing the processing-to-be-processed audio based on the target acoustic transfer feature to generate the target audio content corresponding to the processing-to-be-processed audio comprises: performing a convolution operation based on the processing-to-be-processed audio and the target acoustic transfer feature to generate the target audio content corresponding to the processing-to-be-processed audio. 11.An apparatus for audio rendering, comprising: an obtaining module configured to obtain at least one ear image of a target user and processing-to-be-processed audio; a first determining module configured to determine depth information of the ear image based on the at least one ear image; a second determining module configured to determine a target acoustic transfer feature corresponding to the target user based on the depth information, the target acoustic transfer feature being used to at least indicate acoustic features in a process of sound propagation to an ear of the target user; and a generating module configured to process the processing-to-be-processed audio based on the target acoustic transfer feature to generate target audio content corresponding to the processing-to-be-processed audio. 12.An electronic device, comprising: at least one processor; and at least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor, the instructions, when executed by the at least one processor, causing the electronic device to perform the method according to any one of claims 1 to 10. 13.A computer-readable storage medium having computer-executable instructions stored thereon, the computer-executable instructions, when executed by a processor, implementing the method according to any one of claims 1 to 10. 14.A computer program product comprising computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the method according to any one of claims 1 to 10.

Images