Audio processing method, device and earphone

Abstract

The present application relates to an audio processing method, device, headset and computer-readable storage medium, which first obtains a multi-channel sound signal recorded by a microphone array of the headset, then performs sound source separation on the multi-channel sound signal based on a preset category, determines the target sound source of the preset category, determines different categories based on the multi-channel sound signal based on the orientation information of the wearer, and obtains spatial audio according to the orientation information of the target sound source and the target sound source corresponding to the preset category. Thus, the user can hear the real sound transmitted from different directions by different categories of sound sources through the headset, with a high degree of restoration of the real world and high quality of spatial audio, which is conducive to users to better avoid risks based on spatial audio and improves the reliability of the headset.

Description

Technical Field

[0001] The present application relates to the field of earphone technology, and in particular to an audio processing method, device, earphone and computer-readable storage medium. Background Art

[0002] Headphones can be divided into non-hearing aid headphones and hearing aid headphones. Non-hearing aid headphones do not have hearing aid functions. They receive electrical signals from media players or receivers and use the speakers of the headphones to convert them into audible sound waves. Hearing aid headphones have hearing aid functions and are electronic devices that help hearing-impaired people improve their hearing. It can amplify sound, remove noise, and improve speech clarity, so that hearing-impaired people can better hear the surrounding sounds and language, and improve daily communication and quality of life. When using hearing aids, users can feel the amplification of external sounds, so that they can better hear the surrounding sounds.

[0003] However, in traditional technologies, although users can feel the amplification of external sounds, they cannot distinguish the direction of useful sound sources. Therefore, users still need vision to prevent risks outdoors. For example, when a user uses headphones, there is a car honking on his right, a dog walking on his lower left, and a friend calling him directly behind him. After the user puts on the headphones, although all these sounds are amplified, it is unclear where the sound is coming from, resulting in poor sound recognition. The direction of the sound needs to be identified by vision, which also brings certain safety risks. Therefore, traditional headphones are unreliable to use. Summary of the invention

[0004] Based on this, it is necessary to provide an audio processing method, device, headset and computer-readable storage medium that can improve the reliability of headset use in response to the above technical problems.

[0005] In a first aspect, the present application provides an audio processing method, the method comprising:

[0006] Acquiring a multi-channel sound signal; the sound signal is recorded through a microphone array of a headset;

[0007] Based on the preset categories, the sound sources of the multi-channel sound signals are separated to determine the target sound sources of the preset categories;

[0008] Determine different categories of wearer-based position information based on multi-channel sound signals;

[0009] Spatial audio is obtained according to the orientation information of the target sound source and the target sound source corresponding to the preset category.

[0010] In one embodiment, the earphone is a non-hearing aid earphone, and the step of obtaining a multi-channel sound signal includes:

[0011] Get multi-channel sound signals in Transparency mode.

[0012] In one embodiment, the method further comprises:

[0013] If a transparency mode on instruction is received, the working mode of the headset is controlled to be the transparency mode.

[0014] In one embodiment, the method further comprises:

[0015] If a sound belonging to a preset warning category is obtained, the working mode of the headset is controlled to be a transparent mode.

[0016] In one embodiment, obtaining spatial audio according to the position information of the target sound source and the target sound source corresponding to the preset category includes:

[0017] Inputting the position information of the target sound source into the head transfer function as a key parameter to obtain a head-related impulse response;

[0018] Spatial audio is obtained according to the head-related impulse response and the target sound source corresponding to the preset category.

[0019] In one embodiment, the head-related impulse response includes a head-related impulse response to a left ear and a head-related impulse response to a right ear, the spatial audio includes an output audio signal of the left ear and an output audio signal of the right ear, and obtaining the spatial audio according to the head-related impulse response and the target sound source corresponding to the preset category includes:

[0020] Convolving the target sound source corresponding to the preset category with the head-related impulse response to the left ear, and then performing a downmixing algorithm to obtain an output audio signal for the left ear;

[0021] The target sound source corresponding to the preset category and the head-related impulse response to the right ear are convolved, and then a downmixing algorithm is performed to obtain an output audio signal for the right ear.

[0022] In one embodiment, determining different categories of wearer-based position information based on multi-channel sound signals includes:

[0023] The sound signal is localized based on the sound arrival time difference technology to determine the different categories based on the wearer's position information.

[0024] In one embodiment, the sound source positioning of the sound signal based on the sound arrival time difference technology and determining different categories of wearer-based position information include:

[0025] Obtaining the sound arrival time difference between relative array elements in the microphone array;

[0026] Different categories are determined based on the orientation information of the wearer according to the sound arrival time difference between the relative array elements and the spatial geometric relationship of the microphone array.

[0027] In one embodiment, performing sound source separation on multi-channel sound signals based on preset categories and determining target sound sources of preset categories includes:

[0028] The sound signal is input into an audio recognition model, and a target sound source of a preset category is determined according to the preset categories corresponding to each sound source type in the sound signal; the audio recognition model is trained based on audio samples of various types.

[0029] In one embodiment, after acquiring the multi-channel sound signal, before performing sound source separation on the multi-channel sound signal based on the preset category and determining the target sound source of the preset category, the method further includes:

[0030] The sound signal is subjected to acoustic echo cancellation, noise reduction and automatic gain processing.

[0031] In one embodiment, obtaining spatial audio according to the position information of the target sound source and the target sound source corresponding to the preset category includes:

[0032] Obtaining intermediate spatial audio according to the orientation information of the target sound source and the target sound source corresponding to the preset category;

[0033] Background noise is added to the intermediate audio data to obtain spatial audio.

[0034] In a second aspect, the present application further provides an audio processing device, which is applied to headphones. The device comprises:

[0035] A sound recording module, used to obtain multi-channel sound signals; the sound signals are recorded through the microphone array of the headset;

[0036] An audio separation module is used to separate the sound sources of multi-channel sound signals based on preset categories and determine the target sound sources of the preset categories;

[0037] A sound source localization module, used to determine different types of wearer-based position information based on multi-channel sound signals;

[0038] The spatial audio reconstruction module is used to obtain spatial audio according to the orientation information of the target sound source and the target sound source corresponding to the preset category.

[0039] In a third aspect, the present application further provides a headset, comprising a microphone array and a processor, wherein the microphone array is connected to the processor, the microphone array is used to record multi-channel sound signals and send them to the processor, and the processor is used to implement the following steps:

[0040] Acquiring a multi-channel sound signal; the sound signal is recorded through a microphone array of a headset;

[0041] Based on the preset categories, the sound sources of the multi-channel sound signals are separated to determine the target sound sources of the preset categories;

[0042] Determine different categories of wearer-based position information based on multi-channel sound signals;

[0043] Spatial audio is obtained according to the orientation information of the target sound source and the target sound source corresponding to the preset category.

[0044] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0045] Acquiring a multi-channel sound signal; the sound signal is recorded through a microphone array of a headset;

[0046] Based on the preset categories, the sound sources of the multi-channel sound signals are separated to determine the target sound sources of the preset categories;

[0047] Determine different categories of wearer-based position information based on multi-channel sound signals;

[0048] Spatial audio is obtained according to the orientation information of the target sound source and the target sound source corresponding to the preset category.

[0049] The above-mentioned audio processing method, device, headset and computer-readable storage medium first obtain the multi-channel sound signal recorded by the microphone array of the headset, and then perform sound source separation on the multi-channel sound signal based on the preset category, determine the target sound source of the preset category, determine the different categories based on the wearer's orientation information based on the multi-channel sound signal, and obtain spatial audio according to the orientation information of the target sound source and the target sound source corresponding to the preset category. Therefore, the user can hear the real sound transmitted from different directions by different categories of sound sources through the headset, with a high degree of restoration of the real world and high quality of spatial audio, which is conducive to users to better avoid risks based on spatial audio and improves the reliability of the headset. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 is a schematic flow chart of an audio processing method in one embodiment;

[0051] Figure 2 is a schematic diagram of the azimuth information of a sound source in one embodiment;

[0052] Figure 3 is a flowchart of an audio processing method in yet another embodiment;

[0053] Figure 4 is a schematic flow chart of a step of obtaining spatial audio in one embodiment;

[0054] Figure 5 A schematic diagram of a flow chart of a step of obtaining spatial audio in another embodiment;

[0055] Figure 6 A schematic diagram of a process for constructing a header transmission function in one embodiment;

[0056] Figure 7 is a flowchart of an audio processing method in another embodiment;

[0057] Figure 8 A flowchart of the steps of determining different types of wearer-based position information in one embodiment;

[0058] Fig. 9 is a flowchart of an audio processing method in yet another embodiment;

[0059] Fig.10 A schematic diagram of a flow chart of an audio processing method in yet another embodiment;

[0060] Fig.11 is a detailed flowchart of an audio processing method in an embodiment;

[0061] Fig.12 is a schematic diagram of the structure of a microphone array in one embodiment;

[0062] Fig.13 A schematic diagram of an application scenario of a header transfer function in an embodiment;

[0063] Fig.14 FIG. 4 is a structural block diagram of an audio processing device in an embodiment. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0065] The audio processing method provided in the embodiment of the present application is applied to headphones, and is used to process multi-channel sound signals recorded by the microphone array of the headphones, and after generating spatial audio, it is played by the headphones so that the user can hear the generated spatial audio through the headphones. The headphones can be hearing-aid headphones, which generally refers to hearing-aid devices. The audio processing method can be executed by a controller built into the headphones, or by a server that communicates with the headphones.

[0066] In one embodiment, Figure 1 As shown, an audio processing method is provided, which is described by taking the method applied to a controller in an earphone as an example, and includes the following steps:

[0067] Step 102: Acquire multi-channel sound signals.

[0068] The sound signal is recorded by the microphone array of the headset. The sound signal refers to the sound in the environment where the headset is located. After the microphone array records the sound signal, it sends the sound signal to the controller for processing.

[0069] The sound signal can be recorded based on stereo technology. That is, when the earphone is a hearing aid earphone, a microphone is set on each earphone, and when the hearing aid earphone is recording, a two-channel recording can be recorded, and then the horizontal 360° sound source positioning can be achieved based on the two-channel recording.

[0070] Furthermore, sound signals can also be recorded based on microphone arrays. Microphone arrays use microphones to simulate human ears. Theoretically, sound localization can be achieved by constructing an array including two or more microphones. However, since the human positioning mechanism is more complex, two microphones may not be enough to achieve sound localization. In this case, the number of microphones can be increased.

[0071] A microphone array is an array of two or more microphones arranged in a certain geometric structure. According to different topological structures, microphone arrays can be uniform linear arrays, non-uniform linear arrays, non-linear arrays, circular arrays, planar arrays or stereo arrays. A uniform linear array is a sound collection device composed of microphones arranged at equal distances. This array can maximize signal enhancement and noise reduction in a specific direction. A non-uniform linear array is a sound collection device composed of microphones arranged at unequal distances. This array can improve signal resolution and optimize the capture of sounds of different frequencies. A non-linear array is a device that achieves sound collection by performing nonlinear processing on the microphone signal. This array can improve the spatial quality and fidelity of the sound. A circular array is a circular array composed of multiple microphones, which can achieve signal enhancement and noise reduction in all directions. A planar array is a planar array composed of multiple microphones, which can achieve sound collection and signal enhancement on a two-dimensional plane. A stereo array is a three-dimensional array composed of multiple microphones, which can achieve sound collection and signal enhancement in three-dimensional space.

[0072] Step 104 : Based on the preset categories, perform sound source separation on the multi-channel sound signal to determine the target sound source of the preset category.

[0073] Among them, sound source separation is a method of decomposing multiple mixed sound data into single audio data. The preset category refers to the category of the object that makes the sound. After the controller obtains the sound signal, it combines the sound source and category in the sound signal to perform audio separation on the sound data, that is, separate the sound corresponding to each category and obtain the target sound source corresponding to each preset category.

[0074] The controller stores a plurality of preset categories and corresponding sound features. After acquiring the multi-channel sound signal, the controller performs sound source separation on the sound signal, separates the sound corresponding to each preset category, and obtains the target sound source corresponding to each preset category.

[0075] In an extensible manner, the preset categories can also be classified according to the usage scenarios. For example, in the subway scenario, the preset categories include broadcasting sound, subway running sound, subway door opening and closing sound, and alarm sound. The controller can obtain the scene information of the current environment input by the user, or the controller can determine the scene information of the current environment based on the multi-channel sound signal obtained. After determining the scene information, the preset category corresponding to each target sound source can be identified in the preset category pre-stored corresponding to the scene information.

[0076] For example, if in the following scenario, the user is wearing headphones and has turned on the hearing aid function. There is a honking car on his right, a walking dog on his lower left, and a friend calling him directly behind him. Without the function of this application, it is difficult for the user to distinguish the direction of these sounds. These sounds are all amplified, but it is unclear where the direction of the sound is. Take the microphone array including 4 microphones as an example. The data recorded by the 4 microphones is the data of 4 channels. The data of these 4 channels are repeated. For example, if there is a honking sound in the environment, there will be a honking sound in all 4 channels, but the volume and delay are different. Therefore, the audio processing method of this application is used to separate the sounds in the 4 channels through the sound source separation technology to separate the car horn, female voice, dog barking and other sounds, and they are represented by x1, x2, x3, x4, x5...xn multiple channels. The number of channels is related to the useful elements in the sound, and the useful elements are the elements that the user is interested in. Users can select elements of interest in advance, such as dog barking, car horns, baby crying, human voices, etc. These useful elements are classified as categories during audio separation.

[0077] Step 106 : determining different categories of wearer-based position information based on the multi-channel sound signal.

[0078] After the controller obtains the multi-channel sound signal, it uses the sound source localization technology to process the sound signal and determine the different types of sound sources in the sound signal based on the wearer's position information. The type of sound source is determined based on the obtained sound signal. The number and type of sound sources are generally multiple, or one. For example, when a user is walking on the road with hearing aids, if there is a honking car on the right side of the user, a walking dog on the lower left, and a friend calling him directly behind, the headset can obtain a sound signal including these three sounds, and the sound signal includes three sound sources.

[0079] The direction information of the sound source can be expressed as (L, α, β), where L is the distance between each sound source and the user, α is the horizontal angle between each sound source and the user, and β is the vertical angle between each sound source and the user. The schematic diagram of the spatial direction information can be found in Figure 2 .

[0080] There is not only one way to locate the sound source of a sound signal. For example, controllable beamforming technology based on maximum output power, high-resolution spectrum estimation technology, or positioning technology based on sound pressure amplitude ratio can be used to locate the sound source and determine the azimuth information of the sound source in the sound signal.

[0081] Specifically, the controllable beamforming technology based on maximum output power utilizes the signals of all microphones in the array and maximizes the output power in a certain direction by selecting a set of specific weight coefficients. In this way, the azimuth information of the sound source can be determined, that is, in which direction the output power is the largest. High-resolution spectrum estimation technology can estimate the power spectrum density of the sound signal in different directions by performing frequency domain analysis on the sound signal, thereby determining the azimuth information of the sound source. The positioning technology based on the sound pressure amplitude ratio can utilize the arrival time difference and amplitude difference of the sound at different positions to determine the azimuth information of the sound source. According to the relationship between the arrival time difference and the amplitude difference, the distance and direction from the sound source to each microphone can be calculated to determine the azimuth information of the sound source. It can be understood that in other embodiments, other methods can also be used to locate the sound source of the sound signal, which is not limited here, as long as those skilled in the art believe that it can be achieved.

[0082] Step 108: obtaining spatial audio according to the orientation information of the target sound source and the target sound source corresponding to the preset category.

[0083] Among them, the azimuth information of the target sound source can be obtained based on the sound source localization. The azimuth information of the target sound source can be expressed by L, α, β, where L is the distance of each target sound source from the user, α is the horizontal angle of each target sound source from the user, and β is the vertical angle of each target sound source from the user. After obtaining the azimuth information of the target sound source and the target sound sources of each preset category, the spatial audio of each target sound source can be obtained, thereby obtaining all the spatial audio.

[0084] After obtaining the spatial audio, the spatial audio is sent to headphones, such as hearing aid headphones, to better restore the sound.

[0085] In this embodiment, the multi-channel sound signal recorded by the microphone of the headset is first obtained, and then the sound source of the multi-channel sound signal is separated based on the preset category, the target sound source of the preset category is determined, and the different categories based on the wearer's position information are determined based on the multi-channel sound signal. According to the position information of the target sound source and the target sound source corresponding to the preset category, the spatial audio is obtained. Therefore, the user can hear the real sound transmitted from different directions by different categories of sound sources through the headset, with a high degree of restoration of the real world and high quality of spatial audio, which is conducive to the user to better avoid risks based on spatial audio and improves the reliability of the headset.

[0086] In one embodiment, the earphone is a non-hearing aid earphone, such as Figure 3 As shown, step 102 includes step 302 .

[0087] Step 302: Acquire a multi-channel sound signal in a transparent mode.

[0088] The transparent mode refers to the non-hearing aid earphones working in the transparent mode. When the non-hearing aid earphones work in the transparent mode, the earphones can record more sound signals, and the user can hear the outside world through the earphones.

[0089] In this embodiment, when the non-hearing-aided earphones are in the transparent mode, they obtain multi-channel sound signals, then perform sound source separation, determine the orientation information, and obtain spatial audio according to the orientation information of the target sound source and the target sound source corresponding to the preset category. Therefore, when the user hears the sound of the outside world through the non-hearing-aided earphones, the real sound transmitted from different directions by different types of sound sources can be restored, with a high degree of restoration of the real world and high quality of spatial audio.

[0090] It can be understood that when the earphone is a hearing aid earphone, the method of the present application can be directly executed during operation without being limited to a specific working mode.

[0091] Furthermore, in one embodiment, the audio processing method further includes step 304 .

[0092] Step 304: If a transparent mode on instruction is received, the working mode of the headset is controlled to be the transparent mode.

[0093] The user can send a transparency mode on command through the headset, or send a transparency on command through other devices connected to the headset. After the controller receives the transparency mode on command, considering that the user needs the headset to work in the transparency mode at this time, the controller controls the headset to work in the transparency mode.

[0094] Alternatively, in one embodiment, the audio processing method further includes step 306 .

[0095] Step 306: If a sound belonging to a preset warning category is obtained, the working mode of the earphone is controlled to be a transparent mode.

[0096] The preset warning sounds are pre-stored in the controller, including whistle sounds and broadcast sounds, etc. The specific types are not limited. The controller analyzes the sound signals recorded by the headset. If the analysis shows that there are sounds belonging to the preset warning categories, considering that the user needs to pay attention to the environment at this time, the headset's working mode is controlled to switch or remain in the transparent mode, so that the user can better receive external sounds and avoid risks in time.

[0097] In one embodiment, Figure 4 As shown, step 108 includes step 402 and step 404 .

[0098] Step 402: Input the position information of the target sound source as a key parameter into the head transfer function to obtain a head-related impulse response.

[0099] Among them, the head transmission function (HRTF) is a signal processing function used to simulate the listening space. It is determined by the influence of factors such as the head, ears and body on the propagation and reception of sound, including the influence of sound waves passing through the head, the front and back of the ears, and the reflection and refraction of body parts such as hair, shoulders and chest. HRTF can describe the way sound reaches the listener at different directions and distances, and can be used to calculate the filter response between the sound source and the listener's ear.

[0100] After the azimuth information of each target sound source is input as a key parameter into the head transfer function, the head transfer function can simulate the human ear receiving sound source signals from different directions and positions to obtain the head-related impulse response. The head-related impulse response is the representation of the head transfer function in the time domain. After the azimuth information of each target sound source is input as a key parameter into the head transfer function, the head transfer function will contain many angles (horizontal angle, vertical angle) and many distances (L). At this time, it may be a three-dimensional data. After determining the azimuth information of the target sound source, two angles (horizontal angle, vertical angle) and a distance (the distance of the image category from the user) corresponding to the target sound source can be obtained. In this way, a curve corresponding to the output can be obtained as the head-related impulse response.

[0101] Step 404: obtaining spatial audio according to the head-related impulse response and the target sound source corresponding to the preset category.

[0102] Furthermore, spatial audio can be obtained according to each head-related impulse response including the position information of the target sound source and the target sound source corresponding to each preset category. The processing method can be: convolving the head-related impulse response with the target sound source corresponding to each preset category to obtain spatial audio, simulate the sound source position in the three-dimensional space, generate stereo or surround sound effects, and thus provide a realistic stereo effect for the listener.

[0103] In this embodiment, the position of the sound source in three-dimensional space can be simulated by using the azimuth information of the target sound source, the head transfer function and the target sound source corresponding to the preset category, thereby providing a realistic stereo effect for the listener.

[0104] In one embodiment, the head-related impulse response includes a head-related impulse response to a left ear and a head-related impulse response to a right ear, and the spatial audio includes an output audio signal for the left ear and an output audio signal for the right ear, such as Figure 5 As shown, step 404 includes step 502 and step 504 .

[0105] Step 502 , convolve the target sound source corresponding to the preset category with the head-related impulse response to the left ear, and then perform a downmixing algorithm to obtain an output audio signal for the left ear.

[0106] The head-related impulse response to the left ear refers to the transfer function of the sound of a certain category of target sound source to the left ear. For example, if a person or an animal makes a sound, the head-related impulse response to the left ear includes the transfer function of the person's sound to the left ear and the transfer function of the animal's sound to the left ear.

[0107] Step 504 , convolve the target sound source corresponding to the preset category with the head-related impulse response to the right ear, and then perform a downmixing algorithm to obtain an output audio signal for the right ear.

[0108] The head-related impulse response to the right ear refers to the transfer function of the sound of a certain category of target sound source to the right ear. For example, if a person or an animal makes a sound, the head-related impulse response to the right ear includes the transfer function of the person's sound to the right ear and the transfer function of the animal's sound to the right ear.

[0109] Since some audio devices have only two channels, for example, headphones have only left and right channels, therefore, when forming spatial audio, the output audio signal of the left ear is obtained based on the head-related impulse response to the left ear and the target sound source corresponding to the preset category, and played at the left ear channel, and the output audio signal of the right ear is obtained based on the head-related impulse response to the right ear and the target sound source corresponding to the preset category, and played at the right ear channel, so as to restore the spatial reality of the sound.

[0110] For example, if the sounds of a car horn, a girl, and a dog are x1, x2, and x3 respectively, and the channels of the remaining elements are x4-xn, the left and right ear sound signals heard by the human ear (the output audio signal of the left ear and the output audio signal of the right ear) are yl and yr respectively:

[0111] yl=x1*h1L+x2*h2L+x3*h3L+...+xn*hnL;

[0112] yr=x1*h1R+x2*h2R+x3*h3R+...+xn*hnR;

[0113] Among them, h1L represents the head-related impulse response of the sound of x1 to the left ear, h1R represents the head-related impulse response of the sound of x1 to the right ear, hnL represents the head-related impulse response of the sound of xn to the left ear, and hnR represents the head-related impulse response of the sound of xn to the right ear. After the above processing, users can judge the direction of the sound of useful information such as human voice, which can not only help them remind risks, but also achieve better and more realistic listening effects.

[0114] Furthermore, if Figure 6 As shown, the process of constructing the header transfer function includes step 602 and step 604.

[0115] Step 602: Under preset environmental conditions, measure the designated user to obtain measurement data.

[0116] The preset environmental conditions refer to environmental conditions with known environmental parameters, such as in an anechoic room or in a concert hall.

[0117] Designated users refer to users determined according to needs. Measuring designated users means measuring the influence of various positions of designated users on sound and obtaining measurement data. For example, the reflection and refraction of sound waves passing through the head, front and back of the ears, and hair, shoulders, chest and other body parts to obtain measurement data.

[0118] Step 604: Obtain a head transmission function according to the measurement data.

[0119] After obtaining the measurement data, the measurement data is analyzed and processed to obtain the head transmission function.

[0120] Specifically, data acquisition devices are used for measurement and recording. These devices usually include an artificial head model and an array of multiple microphones or microphones. Under preset environmental conditions, sound is transmitted to the array from different directions and distances, and the signal captured by each microphone is recorded as measurement data. With these data, the head transfer function in each direction can be calculated.

[0121] In this embodiment, by constructing a head transmission function based on measurement data obtained by measuring a designated user under preset environmental conditions, the head transmission function can be made to better fit the characteristics of the designated user.

[0122] After the head transfer function is constructed, when the head transfer function is needed, the user's auricle and other parts can be feature recognized, and according to the recognized feature parameters, the head transfer function corresponding to the recognized feature parameters can be matched in the stored head transfer functions. Alternatively, a head transfer function can be assigned to the user according to the default settings.

[0123] It is understandable that in other embodiments, other methods may be used to determine the head function. For example, public data may be used to match the head function of a certain person's head. Alternatively, the HRTFs in the database are first clustered, and then the user's personalized auricle features are used to select the most matching HRTF from the public database as the final head function. The above methods can be selected according to actual needs and are not limited here.

[0124] In one embodiment, Figure 7 As shown, step 106 includes step 706 .

[0125] Step 706: localize the sound source of the sound signal based on the sound arrival time difference technology, and determine the different categories based on the wearer's position information.

[0126] Specifically, different categories of orientation information based on the wearer can be understood as the orientation information of different categories of target sound sources relative to the wearer. The orientation information of the target sound source can be expressed as (L, α, β), where L is the distance of each target sound source from the wearer, α is the horizontal angle of each target sound source from the wearer, and β is the vertical angle of each target sound source from the wearer. The wearer is the user.

[0127] Based on the sound arrival time difference technology, the azimuth information of the sound source can be determined by calculating the arrival time difference of the sound in different microphones in the microphone array. The specific implementation steps include: first, the microphone array needs to be geometrically calibrated, that is, the distance from each microphone to the array reference point is measured, and the position coordinates of each microphone are calculated. Then the collected sound signal is preprocessed, including denoising, filtering and other operations. Then the moment when different microphones receive the sound signal is recorded at the same time, and then the arrival time difference between different microphones is calculated. If there is a time delay difference in the sound signals received by the two microphones, it means that the sound source is in the direction they determine. Using the arrival time difference and the position coordinates of each microphone, the distance and azimuth from the sound source to each microphone can be calculated, thereby determining the azimuth information of the sound source.

[0128] In this embodiment, the sound source of the sound signal is located based on the sound arrival time difference technology, and the position information of the different categories based on the wearer is determined, so that accurate position information of the target sound source can be obtained.

[0129] In one embodiment, Figure 8 As shown, step 706 includes step 802 and step 804 .

[0130] Step 802: Obtain the sound arrival time difference between relative array elements in the microphone array.

[0131] Among them, the sound signal is collected by microphones at different positions in the array, and the time difference between them is calculated using a signal processing algorithm. This time difference is called the time difference of arrival (TDOA), which can be used to determine the direction of the sound source. There are many methods to obtain TDOA, which can be roughly divided into cross-correlation method, generalized cross-correlation method, adaptive filter method, cross-power spectrum phase method and high-order statistics method.

[0132] Step 804: determine different categories of wearer-based orientation information according to the sound arrival time difference between relative array elements and the spatial geometric relationship of the microphone array.

[0133] Specifically, based on the sound arrival time difference between relative array elements and the spatial geometric relationship of the microphone array, the distance and direction between the sound source and the array can be calculated by using triangulation and back-calculation methods.

[0134] In this embodiment, accurate positioning of the sound source can be achieved by obtaining the sound arrival time difference between the relative array elements in the microphone array, and then determining different categories of wearer-based orientation information according to the sound arrival time difference between the relative array elements and the spatial geometric relationship of the microphone array.

[0135] In one embodiment, Fig. 9 As shown, step 104 includes step 904 .

[0136] Step 904: input the sound signal into the audio recognition model, obtain the preset category corresponding to each sound source type in the sound signal, and determine the target sound source of the preset category.

[0137] The audio recognition model is a pre-trained model, which is trained based on various types of audio samples and contains audio features of different categories of sound sources.

[0138] The training process of the audio recognition model includes: obtaining a database about audio separation, such as a database of sounds and labels of various categories of sound sources, such as children, boys, girls, dogs, horns, etc., and dividing the database into training, testing, and validation sets, and using deep learning and other methods to train the model. Then, the trained model can be deployed on hardware devices such as headphones. For example, the model can be trained using methods such as conv-tasNet, Demucs, or D3net. conv-tasNet, Demucs, or D3net are all based on deep neural networks.

[0139] conv-tasNet is an audio separation algorithm based on temporal convolutional neural networks (TCN) and recurrent neural networks (RNNs). It uses TCN to learn the temporal structural features of audio signals and uses RNNs to model the contextual information of audio signals. By passing the convolution output into the encoder and decoder, the original signal can be reconstructed and separated.

[0140] Demucs is an audio separation algorithm based on the U-Net structure. It uses independent convolution blocks between layers to achieve audio signal separation. Demucs enables the model to handle long-term dependency issues by performing convolution operations with jump connections in multiple layers. In addition, Demucs also uses adaptive batch normalization technology to improve the stability and performance of the model.

[0141] D3net is an audio separation algorithm based on a three-dimensional convolutional neural network, using an architecture similar to Unet. It can separate and reconstruct audio signals in different frequency ranges. D3net also uses deep separable convolution to reduce the number of parameters and computational complexity, thereby improving the efficiency and speed of the model.

[0142] After the sound signal is input into the audio recognition model, the sound features corresponding to different sound source types stored in the audio recognition model are matched based on the sound features in the sound signal to obtain the target sound source of the preset category.

[0143] In this embodiment, the sound signal is input into the audio recognition model, so that the preset category corresponding to each sound source type in the sound signal can be quickly and accurately obtained, and the target sound source of the preset category can be determined.

[0144] In one embodiment, after step 102 and before step 104 , the audio processing method further includes step 103 .

[0145] Step 103: Perform acoustic echo cancellation, noise reduction and automatic gain processing on the sound signal.

[0146] In this embodiment, after obtaining the multi-channel sound signal recorded by the headphones, before performing sound source separation on the multi-channel sound signal based on preset categories and determining the target sound source of the preset category, it is also necessary to perform acoustic echo cancellation, noise reduction and automatic gain processing on the sound signal.

[0147] Specifically, acoustic echo cancellation detects the sound of the response array, compares it with the output audio to calculate the echo path, and removes the echo signal from the output audio to eliminate the echo. Noise reduction estimates the energy of the ambient noise from the input sound, then calculates the noise model, and finally eliminates the noise signal from the original signal to improve the voice quality. Automatic gain is a technology that automatically controls the gain. It can adjust the gain of the microphone according to the voice intensity of the user's speech, increase the volume of the voice while ensuring the clarity of the voice, so that the listener can hear the other party better.

[0148] Furthermore, the sound signal can also be processed for anti-howling. Anti-howling processing is implemented based on the anti-howling algorithm, which includes an adaptive filtering algorithm, a frequency elimination algorithm, and a comb filtering algorithm. The adaptive filtering algorithm processes the input signal and the output signal based on an adaptive digital filter, thereby suppressing the howling noise and improving the speech clarity. The frequency elimination algorithm uses differential equations or convolution methods to achieve elimination processing based on the frequency characteristics of the howling noise. The comb filtering algorithm is based on the principle of comb filters. Through an array of filters, the input signal is filtered to achieve the purpose of eliminating howling. Combining the above algorithms, the howling noise in the hearing aid can be effectively suppressed and the hearing effect can be improved.

[0149] In this embodiment, after acquiring the sound signal recorded by the earphone, before locating the sound source of the sound signal and determining the azimuth information of the sound source in the sound signal, it is also necessary to perform acoustic echo cancellation, noise reduction and automatic gain processing on the sound signal to improve the audio quality.

[0150] In one embodiment, Fig.10 As shown, step 108 includes step 1008 and step 1010 .

[0151] Step 1008: Obtain intermediate spatial audio according to the orientation information of the target sound source and the target sound source corresponding to the preset category.

[0152] After obtaining the orientation information of each target sound source and the target sound source corresponding to the preset category, the spatial audio of each target sound source can be obtained as intermediate audio data.

[0153] Step 1010: Add background noise to the intermediate audio data to obtain spatial audio.

[0154] If the active noise reduction function is activated in the headset, the recorded sound signal may not include noise. When the sound signal is subsequently processed to obtain spatial audio, adding background noise to the intermediate audio data can make the obtained spatial audio more realistic and improve the listening performance of the headset.

[0155] In this embodiment, background noise is added to the intermediate audio data obtained according to the azimuth information of the target sound source and the target sound source corresponding to each preset category to obtain spatial audio, which can simulate the real listening effect under specific environmental conditions, so that the obtained spatial audio is closer to reality.

[0156] In order to better understand the above-mentioned embodiments, a detailed explanation is given below in conjunction with a specific embodiment. In one embodiment, when implementing the audio processing method, the user uses headphones to turn on the hearing aid mode. The headphones can be hearing aid headphones or Bluetooth headphones. The link is that the external microphone records the sound → the speaker in the headphones plays this link. The user can choose to turn on the hearing aid function in the active noise reduction mode, or can choose to turn on the hearing aid function in the active noise reduction mode. Generally, active noise reduction is a hardware channel, and the delay can usually reach a subtle level. The audio processing method can be placed in the hardware link or in the software link. The delay is lower when it is placed in the hardware link. Before performing the hearing aid function, some pre-processing algorithms such as: 3A algorithm, namely acoustic echo cancellation, noise reduction and automatic gain processing, can be performed, and an anti-howling algorithm can also be added.

[0157] For example, Fig.11As shown in the figure, the specific process of the audio processing method includes: using the microphone of a device such as headphones to record the sound, and then using the multi-microphone beam positioning algorithm to identify the direction of the sound and obtain the sound source location information. Then, the recorded sound is separated into different types of sound elements through deep learning sound source separation technology, and the separated audio is processed using a spatial audio algorithm. The above algorithm is performed in the transparent mode and the final processed audio is output.

[0158] Specifically, a microphone array is used for sound recording. The microphone array algorithm requires at least two microphones. As the number of microphones increases, the positioning capability becomes better. For wireless Bluetooth headsets, the ffmic and talkmic on the left and right ears can be used, so there are four microphones that can be used for recording. For headphones, more microphones can be placed on each side for better results.

[0159] In sound source localization, microphones are used to simulate human ears. Theoretically, sound localization can be achieved by constructing an array of two or more microphones. However, the human localization mechanism is more complicated, and two microphones may not be enough. In this case, the number of microphones can be increased to achieve satisfactory localization accuracy.

[0160] A microphone array is an array composed of two or more microphones arranged in a certain geometric structure. According to different topological structures, microphone arrays can be divided into uniform linear arrays, non-uniform linear arrays, nonlinear arrays, circular arrays, planar arrays, and stereo arrays. Fig.12 As shown in the figure, the sound source localization technology based on microphone array can be basically divided into four categories: controllable beamforming technology based on maximum output power; high-resolution spectrum estimation technology; localization technology based on sound pressure amplitude ratio and localization technology based on time delay of arrival (TDOA).

[0161] The positioning method based on TDOA can be achieved in two steps. The first step is to carry out TDOA estimation to obtain the TDOA between the relative array elements in the microphone array. There are many methods to estimate TDOA, which can be roughly divided into cross-correlation method, generalized cross-correlation method, adaptive filter method, cross-power spectrum phase method and high-order statistics method. The second step is to use the estimated time difference between the relative array elements and the known spatial geometric relationship of the microphone array to determine the position and direction (L, α, β) of the sound source. L is the distance between the sound source and the user, α is the horizontal angle between the sound source and the user, and β is the vertical angle between the sound source and the user.

[0162] When separating sounds, suppose in the following scenario, the user is wearing headphones and has turned on the hearing aid function. To his right is a honking car, to his lower left is a walking dog, and directly behind him is a friend calling him. The data recorded by 4 microphones is 4 channels of data, and the data of these 4 channels is repeated. For example, if there is a honking sound in the environment, there will be honking sounds in all 4 channels, but the volume and delay will be different. Therefore, here we need to use audio separation technology to separate the sounds in the 4 channels from the car horn, female voice, dog barking and other sounds, and represent them with multiple channels x1, x2, x3, x4, x5...xn. The number of channels is related to the useful elements in the sound.

[0163] The steps to decompose the recorded data into multi-channel sounds containing specific sound elements using deep learning are as follows: obtain a database on audio separation, which includes sounds and labels of various categories such as children, boys, girls, dogs, horns, etc., and then divide them into training, testing, and validation sets, and then use deep learning and other methods to train the model, and deploy the trained model to hardware devices such as headphones. The audio separation algorithm can be conv-tasNet, Demucs, or D3net algorithms. Users can select what sounds they want to pay more attention to in advance through the app, such as dog barking, car horns, baby crying, human voices, etc. These useful elements are classified as categories when separating audio.

[0164] Spatial audio processing uses the head transfer function HRTF to obtain spatial positioning capabilities (including direction and distance) in the horizontal plane of the space or even in the entire sphere centered on the person. It can also be combined with the spatial sound field and some indoor acoustics or spatial acoustic reverberation algorithms to construct a more realistic sense of space, including:

[0165] Use the audio files separated by the above steps as multi-channel sound sources, and then perform binauralization. Binauralization is the core of spatial audio. After getting the multi-channel sound source, it can only be played in stereo on headphones. In binauralization technology, HRTF technology can be used to listen to fixed sound sources, simulating wearing headphones like not wearing headphones (if it is a stereo sound source, it simulates the feeling of two speakers placed in front of you, and if it is a multi-channel sound source, it simulates the feeling of multiple speakers placed around you).

[0166] There are many ways to determine the head-transmitted HRTF function, such as using public data to match the HRTF of a certain person's head; or, first clustering the head-transmitted HRTFs in the database, and then using the user's personalized auricle features to select the most matching HRTF from the public database as the final function; or, measuring the HRTF functions of the user's various positions in an anechoic room; or, measuring the HRTF functions of the user's various positions in a concert hall.

[0167] Then the reverberation algorithm is used. The reverberation algorithm is an important part of increasing the spatial sense of the sound effect. Generally speaking, the head transmission function adopts the HRTF function of the public laboratory. However, since most of these functions are measured in an anechoic chamber, a lot of spatial information will be lost. The reverberation algorithm can increase the early reflection and late reflection of the room through the algorithm, so as to achieve the real listening effect of simulating the room. Common ways to add reverberation include room impulse response HRIR, artificial reverberation, etc.

[0168] When performing spatial audio processing, the sound effect function that can represent the sense of sound orientation is HRTF (L, α, β), where L is the distance between the sound source and the user, α is the horizontal angle between the sound source and the user, and β is the vertical angle between the sound source and the user. After obtaining the orientation information, you can select the corresponding HRTF in the HRTF library (for example, a car horn as X1 has h1L and h1R representing the transfer function of its sound to the left ear and to the right ear respectively), and then convolve with the audio signal to obtain spatial audio. Users can also add different reverberations to enhance the sense of space.

[0169] like Fig.13 There are multiple target sound sources. When playing back with headphones, since headphones have only two channels, a down-mixing algorithm is needed here. Suppose the sounds of car horns, girls, and dogs are x1, x2, and x3 respectively, and the channels of the remaining elements are x4-xn, h1L represents the head transfer function from the sound of x1 to the left ear, h1R represents the head transfer function from the sound of x1 to the right ear, h2L represents the head transfer function from the sound of x2 to the left ear, h2R represents the head transfer function from the sound of x2 to the right ear, h3L represents the head transfer function from the sound of x3 to the left ear, and h3R represents the head transfer function from the sound of x3 to the right ear. The left and right ear sound signals yl and yr heard by the human ear are:

[0170] yl=x1*h1L+x2*h2L+x3*h3L+...+xn*hnL;

[0171] yr=x1*h1R+x2*h2R+x3*h3R+...+xn*hnR;

[0172] In addition to the channels with specific elements above, there is also a lot of background noise N, which can also be added to the left and right channels such as:

[0173] yl=yl*a1+N*a2;

[0174] yr=yr*a1+N*a2;

[0175] Among them, a1 and a2 are weight coefficients. After the above processing, even if the user is wearing headphones and has turned on the active noise reduction function, when the hearing aid mode is turned on, he can not only perceive the increase of useful audio around him, such as the increase of human voices, but also judge the direction of the sound of useful information such as human voices, which can not only help them remind themselves of risks, but also achieve a better and more realistic listening effect.

[0176] The audio processing method provided in this embodiment can construct a multi-track target sound source through audio decomposition technology and sound source localization technology in the hearing-aided mode of hearing-aided headphones or the transparency mode of non-hearing-aided headphones, and construct spatial audio that can restore the scene and reality, help users to be prompted with risky sounds, and help users experience more realistic spatial audio with a sense of direction.

[0177] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0178] Based on the same inventive concept, the embodiment of the present application also provides an audio processing device for implementing the audio processing method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more audio processing device embodiments provided below can refer to the limitations on the audio processing method above, and will not be repeated here.

[0179] In one embodiment, Fig.14 As shown, an audio processing device is provided, which is applied to headphones, including a sound recording module 1402, an audio separation module 1404, a sound source localization module 1406, and a spatial audio reconstruction module 1408, wherein:

[0180] The sound recording module 1402 is used to obtain multi-channel sound signals, and the sound signals are recorded through the microphone array of the headset.

[0181] The audio separation module 1404 is used to perform sound source separation on multi-channel sound signals based on preset categories, and determine target sound sources of preset categories.

[0182] The sound source localization module 1406 is used to determine different types of wearer-based position information based on the multi-channel sound signal.

[0183] The spatial audio reconstruction module 1408 is used to obtain spatial audio according to the orientation information of the target sound source and the target sound source corresponding to the preset category.

[0184] In one embodiment, the sound recording module is further used to obtain multi-channel sound signals in the transparency mode.

[0185] In one embodiment, the audio processing device further includes a switching module, and the switching module is used to control the working mode to the transparent mode if a transparent mode activation instruction is received.

[0186] In one embodiment, the switching module is further used to control the working mode to the transparent mode if a sound belonging to a preset warning category is obtained.

[0187] In one embodiment, the spatial audio reconstruction module is further used to input the orientation information of the target sound source as a key parameter into the head transfer function to obtain a head-related impulse response; and obtain spatial audio according to the head-related impulse response and the target sound source corresponding to a preset category.

[0188] In one embodiment, the head-related impulse response includes a head-related impulse response to the left ear and a head-related impulse response to the right ear, the spatial audio includes an output audio signal for the left ear and an output audio signal for the right ear, and the spatial audio reconstruction module is further used to convolve the target sound source corresponding to the preset category with the head-related impulse response to the left ear, and then perform a downmixing algorithm to obtain the output audio signal for the left ear; and convolve the target sound source corresponding to the preset category with the head-related impulse response to the right ear, and then perform a downmixing algorithm to obtain the output audio signal for the right ear.

[0189] In one embodiment, the audio processing device further includes a function construction module, which is used to construct a head transmission function, specifically including: measuring a designated user under preset environmental conditions to obtain measurement data, and obtaining the head transmission function according to the measurement data.

[0190] In one embodiment, the sound source localization module is further used to localize the sound source of the sound signal based on the sound arrival time difference technology, and determine the different categories based on the position information of the wearer.

[0191] In one embodiment, the sound source localization module is also used to obtain the sound arrival time difference between relative array elements in the microphone array, and determine different categories of wearer-based orientation information according to the sound arrival time difference between relative array elements and the spatial geometric relationship of the microphone array.

[0192] In one embodiment, the audio separation module is also used to input the sound signal into an audio recognition model, and determine the target sound source of the preset category according to the preset category corresponding to each sound source type in the sound signal; the audio recognition model is trained based on audio samples of various types.

[0193] In one embodiment, the audio processing device also includes a preprocessing module, which is used to perform acoustic echo elimination, noise reduction and automatic gain processing on the sound signal after the sound recording module obtains the sound signal recorded by the earphone and before the audio separation module separates the sound sources of the multi-channel sound signal based on preset categories and determines the target sound source of the preset category.

[0194] In one embodiment, the spatial audio reconstruction module is further used to obtain intermediate spatial audio according to the orientation information of the target sound source and the target sound source corresponding to the preset category, and add reverberation parameter background noise to the intermediate audio data to obtain spatial audio.

[0195] Each module in the above audio processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0196] In one embodiment, a headset is provided, including a microphone array and a processor, wherein the microphone array is connected to the processor, the microphone array is used to record sound signals and send them to the processor, and the processor is used to perform audio processing according to the method of any of the above embodiments.

[0197] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0198] It should be noted that the user information (including but not limited to recorded sound signals) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0199] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0200] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0201] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the present application. It should be noted that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An audio processing method, applied to headphones, characterized in that: The method comprises: Acquiring a multi-channel sound signal; the sound signal is recorded through a microphone array of a headset; Based on the preset categories, the sound sources of the multi-channel sound signals are separated to determine the target sound sources of the preset categories; Determine different categories of wearer-based position information based on multi-channel sound signals; Spatial audio is obtained according to the orientation information of the target sound source and the target sound source corresponding to the preset category.

2. The method according to claim 1, characterized in that The earphone is a non-hearing aid earphone, and the step of obtaining a multi-channel sound signal includes: Get multi-channel sound signals in Transparency mode.

3. The method according to claim 2, characterized in that The method further comprises: If a transparency mode on instruction is received, the working mode of the headset is controlled to be the transparency mode.

4. The method according to claim 2, characterized in that: The method further comprises: If a sound belonging to a preset warning category is obtained, the working mode of the headset is controlled to be a transparent mode.

5. The method according to claim 1, characterized in that The obtaining of spatial audio according to the orientation information of the target sound source and the target sound source corresponding to the preset category includes: Inputting the position information of the target sound source into the head transfer function as a key parameter to obtain a head-related impulse response; Spatial audio is obtained according to the head-related impulse response and the target sound source corresponding to the preset category.

6. The method according to claim 5, characterized in that The head-related impulse response includes a head-related impulse response to a left ear and a head-related impulse response to a right ear, the spatial audio includes an output audio signal for the left ear and an output audio signal for the right ear, and obtaining the spatial audio according to the head-related impulse response and the target sound source corresponding to the preset category includes: Convolving the target sound source corresponding to the preset category with the head-related impulse response to the left ear, and then performing a downmixing algorithm to obtain an output audio signal for the left ear; The target sound source corresponding to the preset category and the head-related impulse response to the right ear are convolved, and then a downmixing algorithm is performed to obtain an output audio signal for the right ear.

7. The method according to claim 1, characterized in that The method of determining different categories of wearer-based position information based on multi-channel sound signals includes: The sound source of the sound signal is localized based on the sound arrival time difference technology, and the different categories are determined based on the position information of the wearer.

8. The method according to claim 7, characterized in that The method of locating the sound source of the sound signal based on the sound arrival time difference technology and determining different types of position information based on the wearer includes: Obtaining the sound arrival time difference between relative array elements in the microphone array; Different categories are determined based on the orientation information of the wearer according to the sound arrival time difference between the relative array elements and the spatial geometric relationship of the microphone array.

9. The method according to claim 1, characterized in that: The method of performing sound source separation on the multi-channel sound signal based on the preset category and determining the target sound source of the preset category includes: The sound signal is input into an audio recognition model, and a target sound source of a preset category is determined according to the preset categories corresponding to each sound source type in the sound signal; the audio recognition model is trained based on audio samples of various types.

10. The method according to claim 1, characterized in that After acquiring the multi-channel sound signal, before performing sound source separation on the multi-channel sound signal based on the preset category and determining the target sound source of the preset category, the method further includes: The sound signal is subjected to acoustic echo cancellation, noise reduction and automatic gain processing.

11. The method according to claim 1, characterized in that: The obtaining of spatial audio according to the orientation information of the target sound source and the target sound source corresponding to the preset category includes: Obtaining intermediate spatial audio according to the orientation information of the target sound source and the target sound source corresponding to the preset category; Background noise is added to the intermediate audio data to obtain spatial audio.

12. An audio processing device, applied to headphones, characterized in that: The device comprises: A sound recording module, used to obtain multi-channel sound signals; the sound signals are recorded through the microphone array of the headset; An audio separation module is used to separate the sound sources of multi-channel sound signals based on preset categories and determine the target sound sources of the preset categories; A sound source localization module, used to determine different types of wearer-based position information based on multi-channel sound signals; The spatial audio reconstruction module is used to obtain spatial audio according to the orientation information of the target sound source and the target sound source corresponding to the preset category.

13. A headset, characterized in that: It comprises a microphone array and a processor, wherein the microphone array is connected to the processor, the microphone array is used to record multi-channel sound signals and send them to the processor, and the processor is used to perform audio processing according to the method described in any one of claims 1 to 11.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

Images