Audio processing method, hearing aid device and hearing aid system

Abstract

The invention relates to an audio processing method, a hearing aid device and a hearing aid system, and the method comprises the steps: obtaining audio frequency domain signals of each microphone in two microphone arrays, receiving pickup direction indication information, and determining a pickup suppression direction according to the pickup direction indication information, performing signal processing on each path of audio frequency domain signal according to each path of audio frequency domain signal and the pickup suppression direction to obtain a first directional enhanced audio signal and a second directional enhanced audio signal; the signal processing comprises performing signal amplitude adjustment according to a pickup suppression direction; the first directional enhanced audio signal is played through the first hearing aid unit, and the second directional enhanced audio signal is played through the second hearing aid unit; the two microphone arrays are respectively located on a first hearing aid unit and a second hearing aid unit of the hearing aid device. According to the method, the pickup direction of the equipment can be adjusted, the directionality and flexibility of audio processing are improved, and the quality of the audio signal obtained through processing is correspondingly improved.

Description

Technical Field

[0001] This application relates to the field of audio processing technology, and in particular to an audio processing method, a hearing aid device, and a hearing aid system. Background Technology

[0002] Hearing aids are electronic devices used to pick up audio signals from the environment and amplify them in a specific direction to obtain an enhanced audio signal. Common examples include headphones and hearing aids.

[0003] In related technologies, hearing aids are limited by the algorithms they employ, and can only achieve signal enhancement in a fixed direction, thus reducing the quality of the processed audio signal. Summary of the Invention

[0004] Therefore, it is necessary to provide an audio processing method, a hearing aid device, and a hearing aid system to address the aforementioned technical problems.

[0005] In a first aspect, this application provides an audio processing method applied to a hearing aid device, the hearing aid device including a first hearing aid unit and a second hearing aid unit, the method comprising:

[0006] Acquire the audio frequency domain signal of the sound collected by each microphone in the two microphone arrays; the two microphone arrays are located on the first hearing aid unit and the second hearing aid unit, respectively;

[0007] Receive the pickup direction indication information and determine the pickup suppression direction based on the pickup direction indication information;

[0008] The audio frequency domain signals are processed according to the pickup suppression direction to obtain a first directional enhanced audio signal and a second directional enhanced audio signal. The signal processing includes adjusting the signal amplitude according to the pickup suppression direction. The first directional enhanced audio signal is played through the first hearing aid unit, and the second directional enhanced audio signal is played through the second hearing aid unit.

[0009] In one embodiment, signal processing is performed on each audio frequency domain signal according to the audio frequency domain signal and the pickup suppression direction to obtain a first directional enhanced audio signal and a second directional enhanced audio signal, including:

[0010] Feature extraction is performed on each audio frequency domain signal to obtain feature information;

[0011] The feature information and the direction of sound suppression are input into the signal enhancement model. The signal enhancement model processes the frequency domain signals of each audio channel to obtain the first directional enhanced audio signal and the second directional enhanced audio signal. The signal enhancement model is a pre-trained neural network model.

[0012] In one embodiment, a first directional enhanced audio signal and a second directional enhanced audio signal are obtained by signal processing of each audio frequency domain signal, including:

[0013] The audio frequency domain signals of the first hearing aid unit are fused to obtain the fused first frequency domain signal;

[0014] The audio frequency domain signals of the second hearing aid unit are fused to obtain the fused second frequency domain signal;

[0015] The first frequency domain signal and the second frequency domain signal are subjected to signal amplitude suppression in the pickup suppression direction, respectively, to obtain the first directional enhanced audio signal and the second directional enhanced audio signal.

[0016] In one embodiment, the feature information includes amplitude difference features and phase difference features; feature extraction is performed on each audio frequency domain signal to obtain feature information, including:

[0017] Phase difference characteristics are obtained by extracting the phase information of each audio frequency domain signal from each microphone in the microphone array of the same hearing aid unit.

[0018] Amplitude difference characteristics are obtained by extracting the amplitude information of each audio frequency domain signal from each microphone in the microphone array of different hearing aid units.

[0019] In one embodiment, each microphone array includes two microphones; phase difference features are obtained by extracting the phase information of each audio frequency domain signal from each microphone in the microphone array of the same hearing aid unit, including:

[0020] The phase difference between the two audio frequency domain signals of the first hearing aid unit is obtained as the first feature;

[0021] The phase difference between the two audio frequency domain signals of the second hearing aid unit is obtained as a second feature;

[0022] The first and second features are used as phase difference features.

[0023] In one embodiment, the microphone array includes two microphones; by extracting the amplitude information of each audio frequency domain signal from each microphone in the microphone array of different hearing aid units, amplitude difference features are obtained, including:

[0024] The first amplitude average value between the two audio frequency domain signals of the first hearing aid unit at the same frequency point is obtained, and the second amplitude average value between the two audio frequency domain signals of the second hearing aid unit at the same frequency point is obtained.

[0025] The amplitude difference between the first and second amplitude averages at the same frequency points is obtained as the amplitude difference feature.

[0026] In one embodiment, signal amplitude suppression in the pickup suppression direction is performed on the first frequency domain signal and the second frequency domain signal respectively to obtain a first directional enhanced audio signal and a second directional enhanced audio signal, including:

[0027] Obtain the phase of each frequency point in the first frequency domain signal and the second frequency domain signal;

[0028] Based on the phase and pickup suppression direction of each frequency point in the first and second frequency domain signals, signal amplitude suppression is performed on the first and second frequency domain signals to obtain the first and second directional enhanced audio signals.

[0029] In one embodiment, signal amplitude suppression is performed on the first and second frequency domain signals based on the phase and pickup suppression direction of each frequency point in the first and second frequency domain signals to obtain a first directional enhanced audio signal and a second directional enhanced audio signal, including:

[0030] The phase amplitude of the frequency signal whose phase matches the pickup suppression direction in the first frequency domain signal and the second frequency domain signal is reduced, and / or the signal amplitude of the frequency signal whose phase does not match the pickup suppression direction in the first frequency domain signal and the second frequency domain signal is increased, to obtain the first directional enhancement frequency domain signal and the second directional enhancement frequency domain signal.

[0031] The first directional enhanced frequency domain signal and the second directional enhanced frequency domain signal are subjected to time-frequency transformation processing respectively to obtain the first directional enhanced audio signal and the second directional enhanced audio signal.

[0032] Secondly, this application also provides a hearing aid device, including a first hearing aid unit, a second hearing aid unit, and a control chip; both the first and second hearing aid units include a microphone array and a speaker; the control chip is used to implement the steps of any of the above-mentioned audio processing methods.

[0033] Thirdly, this application also provides a hearing aid system, including an input terminal and a hearing aid device that communicate with each other; the input terminal is used to send user-inputted pickup direction indication information to the hearing aid device, and the hearing aid device is used to implement the steps of any of the above-mentioned audio processing methods.

[0034] The aforementioned audio processing method, hearing aid device, and hearing aid system acquire the audio frequency domain signals from each microphone in two microphone arrays, receive pickup direction indication information, and determine the pickup suppression direction based on the pickup direction indication information. Signal processing is then performed on each audio frequency domain signal according to the pickup suppression direction to obtain a first directional enhanced audio signal and a second directional enhanced audio signal. Signal processing includes signal amplitude adjustment based on the pickup suppression direction. The two microphone arrays are located on the first and second hearing aid units of the hearing aid device, respectively. The first directional enhanced audio signal is played through the first hearing aid unit, and the second directional enhanced audio signal is played through the second hearing aid unit. In this method, the pickup suppression direction can be determined based on the received pickup direction indication information, and the signal amplitude can be adjusted accordingly, thereby enabling adjustment of the device's pickup direction. Users can flexibly adjust the pickup direction based on actual needs, thus improving the directionality and flexibility of audio processing and correspondingly improving the quality of the processed audio signal. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of the structure of a hearing aid device in one embodiment;

[0036] Figure 2 This is a flowchart illustrating an audio processing method in one embodiment;

[0037] Figure 3 This is a schematic diagram of the interface operation for inputting pickup indication information in one embodiment;

[0038] Figure 4(a) is a schematic diagram of the sound pickup direction distribution in one embodiment;

[0039] Figure 4(b) is a schematic diagram of the pickup direction distribution in another embodiment;

[0040] Figure 5 This is a schematic diagram of the process for obtaining the first directional enhanced audio signal and the second directional enhanced audio signal in one embodiment;

[0041] Figure 6 This is a schematic diagram of the process for obtaining the first directional enhanced audio signal and the second directional enhanced audio signal in another embodiment;

[0042] Figure 7 This is a flowchart illustrating the process of determining feature information in one embodiment;

[0043] Figure 8 This is a flowchart illustrating the process of determining phase difference characteristics in one embodiment;

[0044] Figure 9 This is a flowchart illustrating the process of determining amplitude difference characteristics in one embodiment;

[0045] Figure 10 This is a schematic diagram of the process for obtaining the first directional enhanced audio signal and the second directional enhanced audio signal in another embodiment;

[0046] Figure 11 This is a schematic diagram of the process for obtaining the first directional enhanced audio signal and the second directional enhanced audio signal in another embodiment;

[0047] Figure 12 This is a flowchart illustrating the audio processing method in another embodiment;

[0048] Figure 13 This is a schematic diagram illustrating an audio processing method in one embodiment;

[0049] Figure 14 This is a schematic diagram of the neural network structure in one embodiment;

[0050] Figure 15 This is a structural block diagram of an audio processing device in one embodiment. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0052] This application provides an audio processing method applied to a hearing aid device including a first hearing aid unit and a second hearing aid unit. Therefore, before describing the audio processing method in detail, the hearing aid device provided in this application is described first.

[0053] In one embodiment, such as Figure 1 As shown, a hearing aid device 100 is provided, comprising:

[0054] A first hearing aid unit 110, a second hearing aid unit 120, and a control chip 130; both the first hearing aid unit 110 and the second hearing aid unit 120 include a microphone array 10 and a speaker 20.

[0055] The control chip 130 communicates with the first hearing aid unit 110 and the second hearing aid unit 120.

[0056] Optionally, the hearing aid device 100 can be a hearing aid or headphones that support hearing aid functions. For example, the hearing aid device 100 can be a separate device, such as in-ear or on-ear headphones, or hearing aids worn in the left and right ears respectively, or it can be a one-piece device, such as headphones, or a headset that uses virtual reality (VR) technology.

[0057] When the hearing aid device 100 is a separate device, the control chip 130 can be located in either the first hearing aid unit 110 or the second hearing aid unit 120, such as by default in the hearing aid unit used on the right side. When the hearing aid device 100 is an integrated device, the control chip 130 can be located at any position on the integrated device, such as by default in the middle connecting area of ​​the headphones, or by default in the top area of ​​the VR headset.

[0058] Each microphone array 10 includes at least two microphones, which are independent of each other and positioned at different locations to collect sound signals from different directions. When a user wears the hearing aid device 100, the first hearing aid unit 110 and the second hearing aid unit 120 are located on the left and right sides of the ear, respectively. Correspondingly, the two microphone arrays 10 are located on the left and right sides of the ear, respectively, and can collect multiple sound signals from the left ear and multiple sound signals from the right ear, respectively.

[0059] The microphone is used to convert the acquired sound signals into electrical signals for signal processing by the control chip 130. A microphone array 10, which includes at least two microphones, can acquire at least two sound signals and send the electrical signals converted from the acquired sound signals to the control chip 130 for signal processing.

[0060] The control chip 130 is used to process the electrical signals of the various sound signals collected by the microphone array 10 and send the processed electrical signals to the speaker 20.

[0061] The sound signals collected by the microphone array 10 in the first hearing aid unit 110 correspond to the sound signals acquired by the first hearing aid unit 110, and the sound signals collected by the microphone array 10 in the second hearing aid unit 120 correspond to the sound signals acquired by the second hearing aid unit 120.

[0062] The control chip 130 is also used to receive pickup direction indication information sent by an external terminal, and determine the pickup suppression direction based on the pickup direction indication information. The signal processing accordingly includes adjusting the signal amplitude based on the pickup suppression direction to achieve directional audio enhancement. For example, the above signal processing may also include time-frequency transformation of the sound signal, filtering and noise reduction, or signal fusion.

[0063] The speaker 20 is used to convert electrical signals into sound signals and play them. The control chip 130 sends the processed electrical signals to the speaker 20, which then converts the processed electrical signals into sound signals and plays them.

[0064] After the user wears the hearing aid device 100, the two speakers 20 are located on the left and right sides of the ear, respectively. The speaker 20 in the first hearing aid unit 110 can play a first directional enhanced audio signal obtained based on the sound signal acquired by the first hearing aid unit 110; the speaker 20 in the second hearing aid unit 120 can play a second directional enhanced audio signal obtained based on the sound signal acquired by the second hearing aid unit 120.

[0065] The following section details the audio processing method provided in the embodiments of this application, applying it to... Figure 1 Taking a hearing aid as an example, this describes a control chip specifically used in a hearing aid. In one embodiment, such as... Figure 2 As shown, the provided audio processing method includes the following steps:

[0066] S210. Acquire the audio frequency domain signals of each microphone in the two microphone arrays; the two microphone arrays are located on the first hearing aid unit and the second hearing aid unit, respectively.

[0067] The microphone's audio frequency domain signal is the frequency domain signal obtained by converting the sound signal (i.e., audio signal) collected by the microphone from the time domain to the frequency domain.

[0068] Optionally, the control chip receives audio signals collected by each microphone in the microphone arrays of the first and second hearing aid units, and performs a Short-Time Fourier Transform (STFT) on each audio signal to convert the audio signal from the time domain to the frequency domain, obtaining each audio frequency domain signal. For example, when each microphone array includes two microphones, the control chip can convert the signals to obtain two audio frequency domain signals corresponding to the first hearing aid unit and two audio frequency domain signals corresponding to the second hearing aid unit.

[0069] S220: Receive the pickup direction indication information and determine the pickup suppression direction based on the pickup direction indication information.

[0070] The pickup direction indication information is determined based on the user's selection operation and can be either a pickup suppression direction or a pickup enhancement direction. For example, the pickup direction information can be represented by angles. The pickup direction that a hearing aid can cover is typically a 360° range centered on itself. The sum of the angle ranges of the pickup suppression direction and the pickup enhancement direction is 360°. Therefore, if one angle range is known, the other angle range can be derived. For example, the pickup suppression direction is 45°~135°, and the pickup enhancement direction is 135°~45°.

[0071] Hearing aids can communicate with an input terminal. Users can input control commands into the binaural hearing aids via the input terminal to control their operating status. The input control commands include pickup control commands to adjust the pickup direction of the hearing aid, which carry pickup direction indication information. The control chip in the hearing aid can receive the pickup control commands sent from the input terminal and parse them to obtain the pickup direction indication information carried within the commands.

[0072] Optionally, when the input terminal is a mobile terminal such as a mobile phone, the user can log in to the application (APP) associated with the hearing aid through the mobile phone and enter the sound pickup direction selection interface. Figure 3 As shown, users can select the pickup enhancement direction in the direction selection interface. Figure 3 The example demonstrates how a user selects a direction between 45° and 135° as the pickup enhancement direction. After the user makes the selection, the mobile phone uses the selected pickup enhancement direction as pickup direction indication information, generates a pickup control command carrying this pickup direction indication information, and sends it to the hearing aid.

[0073] It should be noted that the direction of sound pickup enhancement corresponds to the target sound pickup area of ​​the hearing aid, while the direction of sound pickup suppression corresponds to the non-target sound pickup area of ​​the hearing aid. The control chip needs to perform signal processing on each audio frequency domain signal to achieve the effect of enhancing the speech in the target sound pickup area while suppressing the speech in the non-target sound pickup area. For example, as shown in Figures 4(a) and 4(b), the shaded area is the non-target sound pickup area, where audio needs to be enhanced; the unshaded area is the target sound pickup area, where audio needs to be suppressed.

[0074] Optionally, if the pickup direction indication information is the pickup suppression direction selected by the user, the control chip can directly obtain the pickup suppression direction from the pickup direction indication information; conversely, if the pickup direction indication information is the pickup enhancement direction selected by the user, the control chip can deduce the pickup suppression direction based on the pickup enhancement direction in the pickup direction indication information.

[0075] S230. The audio frequency domain signals of each channel are processed according to the pickup suppression direction to obtain a first directional enhanced audio signal and a second directional enhanced audio signal; the signal processing includes adjusting the signal amplitude according to the pickup suppression direction; the first directional enhanced audio signal is played through the first hearing aid unit, and the second directional enhanced audio signal is played through the second hearing aid unit.

[0076] Optionally, after obtaining the audio frequency domain signals and pickup direction indication information, the control chip can adjust the signal amplitude of each audio frequency domain signal differently according to the pickup suppression direction. For example, the control chip can reduce the signal amplitude of each audio frequency domain signal corresponding to the pickup suppression direction, or it can increase the signal amplitude of each audio frequency domain signal corresponding to the non-pickup suppression direction.

[0077] Optionally, the control chip can further process the amplitude-adjusted directional enhanced audio frequency domain signal to obtain two directional enhanced audio signals, namely a first directional enhanced audio signal and a second directional enhanced audio signal. For example, the microphone arrays in the first and second hearing aid units include two microphones. The control chip can correspondingly obtain two amplitude-adjusted directional enhanced audio frequency domain signals corresponding to the first hearing aid unit and two amplitude-adjusted directional enhanced audio frequency domain signals corresponding to the second hearing aid unit. The control chip can perform filtering, noise reduction, and inverse short-time Fourier transform (ISTFT) on these four signals to convert them from the frequency domain to the time domain, obtaining four directional enhanced audio signals. These are then further processed to obtain the first and second directional enhanced audio signals, which are then played through the first and second hearing aid units respectively.

[0078] For example, continuing the above example, the control chip can select one of the two directional enhanced audio signals corresponding to each hearing aid unit as the output, or output the two signals after weighted fusion to obtain one directional enhanced audio signal. The two hearing aid units will obtain two directional enhanced audio signals respectively, namely the first directional enhanced audio signal and the second directional enhanced audio signal.

[0079] Optionally, after processing the first directional enhanced audio signal and the second directional enhanced audio signal, the control chip can play the first directional enhanced audio signal to the left ear through the speaker in the first hearing aid unit, and play the second directional enhanced audio signal to the right ear through the speaker in the second hearing aid unit.

[0080] In this embodiment, the audio frequency domain signals of each microphone in two microphone arrays located on the first and second hearing aid units are acquired. Pickup direction indication information is received, and a pickup suppression direction is determined based on the pickup direction indication information. Signal processing is then performed on each audio frequency domain signal according to the pickup suppression direction to obtain a first directional enhanced audio signal and a second directional enhanced audio signal. Signal processing includes signal amplitude adjustment based on the pickup suppression direction. The first directional enhanced audio signal is played through the first hearing aid unit, and the second directional enhanced audio signal is played through the second hearing aid unit. In the above method, the pickup suppression direction can be determined based on the received pickup direction indication information, and the signal amplitude can be adjusted accordingly, thereby enabling adjustment of the device's pickup direction. Users can flexibly adjust the pickup direction based on actual needs, thus improving the directionality and flexibility of audio processing and correspondingly improving the quality of the processed audio signal.

[0081] To obtain the first directional enhanced audio signal and the second directional enhanced audio signal, in one embodiment, such as Figure 5 As shown, the above-mentioned S220, which processes the audio frequency domain signals according to the frequency domain signals of each channel and the pickup suppression direction, to obtain the first directional enhanced audio signal and the second directional enhanced audio signal, includes:

[0082] S510: Extract features from each audio frequency domain signal to obtain feature information.

[0083] The feature information can be phase, amplitude, or frequency, or it can be information determined based on phase, amplitude, or frequency, such as phase difference, amplitude difference, or frequency difference.

[0084] Optionally, the control chip can extract features from each audio frequency domain signal to obtain feature information for each audio frequency domain signal, such as extracting the phase and amplitude of each audio frequency domain signal as feature information for the corresponding audio frequency domain signal. The control chip can also further process the extracted phase and amplitude of each audio frequency domain signal to obtain feature information. For example, the control chip can obtain the phase difference of the audio frequency domain signals of the same hearing aid unit and the amplitude difference between the audio frequency domain signals of different hearing aid units, and use the phase difference and amplitude difference as feature information.

[0085] S520. Input the feature information and the direction of sound pickup suppression into the signal enhancement model. Perform signal processing on each audio frequency domain signal through the signal enhancement model to obtain the first directional enhanced audio signal and the second directional enhanced audio signal. The signal enhancement model is a pre-trained neural network model.

[0086] For example, the signal enhancement model can be a convolutional neural network (CNN) model.

[0087] Optionally, after obtaining the feature information and the pickup suppression direction, the control chip can input the feature information and the pickup suppression direction into the signal enhancement model. The signal enhancement model then performs signal amplitude suppression, noise reduction, fusion, and time-frequency transformation on each audio frequency domain signal to obtain the first directional enhanced audio signal and the second directional enhanced audio signal.

[0088] It should be noted that this signal enhancement model is trained based on multiple noisy frequency signals collected by the hearing aid device. The model uses the feature information and pickup suppression direction of the multiple noisy frequency signals as training inputs to obtain two audio signals with suppressed pickup directions (i.e., directionally enhanced clean audio signals) as training targets. During training, the compressed complex spectral distance loss function can be used to constrain the model's output and target, and the Adam optimizer is used to update the model parameters using gradients.

[0089] In this embodiment, feature information is obtained by extracting features from each audio frequency domain signal. This feature information and the pickup suppression direction are then input into a signal enhancement model. The signal enhancement model processes each audio frequency domain signal to obtain a first directional enhanced audio signal and a second directional enhanced audio signal. In this method, the signal enhancement model performs signal processing on each audio frequency domain signal based on the feature information, including signal amplitude adjustment according to the pickup suppression direction, to obtain the first and second directional enhanced audio signals. The signal enhancement model is trained with a large number of samples, making it accurate and reliable, thus improving the reliability of signal processing and simultaneously enhancing the signal quality of the obtained first and second directional enhanced audio signals.

[0090] In an alternative embodiment, such as Figure 6 As shown, the above-mentioned S520 performs signal processing on each audio frequency domain signal to obtain a first directional enhanced audio signal and a second directional enhanced audio signal, including:

[0091] S610. The audio frequency domain signals of the first hearing aid unit are fused to obtain the fused first frequency domain signal.

[0092] The acquired audio frequency domain signals include audio frequency domain signals from the first hearing aid unit and audio frequency domain signals from the second hearing aid unit.

[0093] Optionally, the control chip can determine the corresponding audio frequency domain signal of the first hearing aid unit from each audio frequency domain signal, and fuse the audio frequency domain signals of the first hearing aid unit to obtain the fused first frequency domain signal.

[0094] S620: The audio frequency domain signals of the second hearing aid unit are fused to obtain the fused second frequency domain signal.

[0095] Optionally, similar to the first frequency domain signal acquisition method, the control chip can determine the corresponding audio frequency domain signal of the second hearing aid unit from each audio frequency domain signal, and fuse the audio frequency domain signals of the second hearing aid unit to obtain the fused second frequency domain signal.

[0096] S630. Perform signal amplitude suppression in the pickup suppression direction on the first frequency domain signal and the second frequency domain signal respectively to obtain the first directional enhanced audio signal and the second directional enhanced audio signal.

[0097] Optionally, for the fused first and second frequency domain signals, the control chip can determine the signal distribution direction based on the phase of the audio frequency domain signal to suppress the signal amplitude in the corresponding pickup suppression direction in the first and second frequency domain signals, thereby obtaining a first directional enhanced audio signal and a second directional enhanced audio signal, respectively. For example, the control chip can reduce the signal amplitude in the corresponding pickup suppression direction to 0 or to 50% of its original value to achieve signal amplitude suppression.

[0098] In this embodiment, the audio frequency domain signals of the first hearing aid unit are fused to obtain a fused first frequency domain signal. The audio frequency domain signals of the second hearing aid unit are also fused to obtain a fused second frequency domain signal. Signal amplitude suppression is then applied to the first and second frequency domain signals in the pickup suppression direction to obtain a first directional enhanced audio signal and a second directional enhanced audio signal. In this method, signal amplitude suppression is performed in the corresponding pickup suppression direction based on the pickup suppression direction, thereby achieving directional audio enhancement and improving audio quality.

[0099] The feature information includes amplitude difference features and phase difference features. Based on this, in one embodiment, such as Figure 7 As shown, S510 above extracts features from each audio frequency domain signal to obtain feature information, including:

[0100] S710. Phase difference characteristics are obtained by extracting the phase information of each audio frequency domain signal from each microphone in the microphone array of the same hearing aid unit.

[0101] In practical applications, when a user wears a hearing aid, the first and second hearing aid units are located on the user's left and right sides, respectively.

[0102] Optionally, the control chip can obtain the phase difference characteristics of the same hearing aid unit based on the phase information of each audio frequency domain signal of the first hearing aid unit and the phase information of each audio frequency domain signal of the second hearing aid unit.

[0103] S720: By extracting the amplitude information of each audio frequency domain signal from each microphone in the microphone array of different hearing aid units, amplitude difference characteristics are obtained.

[0104] Optionally, the control chip can obtain the amplitude difference characteristics between different hearing aids based on the amplitude information of each audio frequency domain signal of the first hearing aid unit and the amplitude information of each frequency domain signal of the second hearing aid unit.

[0105] In this embodiment, phase difference features are obtained by extracting the phase information of each audio frequency domain signal from each microphone in the microphone array of the same hearing aid unit, and amplitude difference features are obtained by extracting the amplitude information of each audio frequency domain signal from each microphone in the microphone array of different hearing aid units. The feature information obtained in the above method includes not only the phase difference features of the same unit but also the amplitude difference features between different units, thus improving the richness of feature information and consequently improving the accuracy of the final processed audio information.

[0106] In one embodiment, where each microphone array includes two microphones, such as Figure 8 As shown, in S710 above, phase difference characteristics are obtained by extracting the phase information of each audio frequency domain signal from each microphone in the microphone array of the same hearing aid unit, including:

[0107] S810: Obtain the phase difference between the two audio frequency domain signals of the first hearing aid unit as the first feature.

[0108] With each microphone array including two microphones, each hearing aid unit can obtain two audio frequency domain signals accordingly.

[0109] Optionally, the control chip can determine the two audio frequency domain signals of the first hearing aid unit from each audio frequency domain signal to obtain the phase difference between the two audio frequency domain signals as the first feature.

[0110] S820: Obtain the phase difference between the two audio frequency domain signals of the second hearing aid unit as the second feature.

[0111] Alternatively, similarly, the control chip can determine the two audio frequency domain signals of the second hearing aid unit from each audio frequency domain signal to obtain the phase difference between the two audio frequency domain signals as a second feature.

[0112] S830, take the first feature and the second feature as phase difference features.

[0113] Optionally, the control chip can combine the obtained first and second features as phase difference features.

[0114] In one embodiment, where each microphone array includes two microphones, such as Figure 9 As shown, S720 above obtains amplitude difference features by extracting the amplitude information of each audio frequency domain signal from each microphone in the microphone array of different hearing aid units, including:

[0115] S910. Obtain the first average amplitude of the two audio frequency domain signals of the first hearing aid unit at the same frequency point, and the second average amplitude of the two audio frequency domain signals of the second hearing aid unit at the same frequency point.

[0116] Optionally, the control chip can determine two audio frequency domain signals of the first hearing aid unit from each audio frequency domain signal to obtain the average amplitude of these two audio frequency domain signals at the same frequency point, thus obtaining the first average amplitude value at each frequency point. Similarly, the control chip can determine two audio frequency domain signals of the second hearing aid unit from each audio frequency domain signal to obtain the average amplitude of these two audio frequency domain signals at the same frequency point, thus obtaining the second average amplitude value at each frequency point.

[0117] S920. Obtain the amplitude difference between the first amplitude mean and the second amplitude mean at the same frequency point, and use it as the amplitude difference feature.

[0118] Optionally, after the control chip obtains the first amplitude mean and the second amplitude mean for each frequency point, it can obtain the amplitude difference between the first amplitude mean and the second amplitude mean for the same frequency point as the amplitude difference feature.

[0119] In this embodiment, each microphone array includes two microphones. The method acquires the phase difference between the two audio frequency domain signals of the first hearing aid unit as a first feature, and acquires the phase difference between the two audio frequency domain signals of the second hearing aid unit as a second feature, using both features as a phase difference feature. Additionally, it acquires the first average amplitude at the same frequency point between the two audio frequency domain signals of the first hearing aid unit, and the second average amplitude at the same frequency point between the two audio frequency domain signals of the second hearing aid unit, to obtain the amplitude difference between the first and second average amplitudes at the same frequency point, which is used as an amplitude difference feature. In the above method, the phase difference of the signals from the same hearing aid unit is acquired as a phase difference feature, and the amplitude difference between the signals from different hearing aid units is acquired as an amplitude difference feature, thus improving the richness of the feature information.

[0120] Signal phase determines signal direction. Based on this, in one embodiment, such as... Figure 10 As shown, S610 above performs signal amplitude suppression in the pickup suppression direction on the first frequency domain signal and the second frequency domain signal respectively to obtain a first directional enhanced audio signal and a second directional enhanced audio signal, including:

[0121] S1010: Obtain the phase of each frequency point in the first frequency domain signal and the second frequency domain signal.

[0122] Optionally, after obtaining the fused first frequency domain signal and second frequency domain signal, the control chip can obtain the phase at each frequency point in the first frequency domain signal and the phase at each frequency point in the second frequency domain signal, respectively.

[0123] S1020. Based on the phase and pickup suppression direction of each frequency point in the first frequency domain signal and the second frequency domain signal, the signal amplitude is suppressed to obtain the first directional enhanced audio signal and the second directional enhanced audio signal.

[0124] The phase of the signal and the direction of pickup suppression can both be characterized by angle.

[0125] Optionally, the control chip can match the phase of the first frequency domain signal and the second frequency domain signal with the pickup suppression direction, respectively, so as to suppress the signal amplitude of the first frequency domain signal according to the matching result to obtain a first directional enhanced audio signal, and suppress the amplitude of the second frequency domain signal to obtain a second directional enhanced audio signal.

[0126] In an alternative embodiment, such as Figure 11 As shown, the above-mentioned S1020, which involves suppressing the signal amplitude of the first and second frequency domain signals based on the phase and pickup suppression direction of each frequency point in the first and second frequency domain signals to obtain a first directional enhanced audio signal and a second directional enhanced audio signal, includes:

[0127] S1110, reduce the signal amplitude of the frequency point signal whose phase matches the pickup suppression direction in the first frequency domain signal and the second frequency domain signal, and / or increase the signal amplitude of the frequency point signal whose phase does not match the pickup suppression direction in the first frequency domain signal and the second frequency domain signal, to obtain the first directional enhancement frequency domain signal and the second directional enhancement frequency domain signal.

[0128] Optionally, for the first frequency domain signal, the control chip can reduce the signal amplitude of the frequency signal whose phase matches the pickup suppression direction in the first frequency domain signal according to a preset reduction degree to obtain the first directional enhanced frequency domain signal. Alternatively, it can increase the signal amplitude of the frequency signal whose phase does not match the pickup suppression direction in the first frequency domain signal according to a preset amplification degree to also obtain the first directional enhanced frequency domain signal. The control chip can also reduce the signal amplitude of the frequency signal whose phase matches the pickup suppression direction according to a preset reduction degree, and simultaneously increase the signal amplitude of the frequency signal whose phase matches the pickup suppression direction according to a preset amplification degree to obtain the first directional enhanced frequency domain signal.

[0129] Accordingly, for the second frequency domain signal, the control chip can reduce the signal amplitude of the frequency point signal whose phase matches the pickup suppression direction in the second frequency domain signal according to a preset reduction degree, to obtain the second directional enhanced frequency domain signal. Alternatively, it can increase the signal amplitude of the frequency point signal whose phase does not match the pickup suppression direction in the second frequency domain signal according to a preset amplification degree, also to obtain the second directional enhanced frequency domain signal. The control chip can also reduce the signal amplitude of the frequency point signal whose phase matches the pickup suppression direction according to a preset reduction degree, while simultaneously increasing the signal amplitude of the frequency point signal whose phase matches the pickup suppression direction according to a preset amplification degree, thereby obtaining the second directional enhanced frequency domain signal.

[0130] For example, the reduction and increase levels can be represented as percentages, and can be the same or different, or any value between 0% and 100%. For instance, both the reduction and increase levels can be 50%, which the user can set according to their actual needs. Users can send control commands carrying the reduction and increase levels to the binaural hearing aid setup via an input terminal, thereby achieving flexible settings for the reduction / increase levels.

[0131] S1120. Perform time-frequency transformation processing on the first directional enhancement frequency domain signal and the second directional enhancement frequency domain signal respectively to obtain the first directional enhancement audio signal and the second directional enhancement audio signal.

[0132] Optionally, having obtained a first directional enhancement frequency domain signal and a second directional enhancement frequency domain signal, the computer device can perform time-frequency conversion processing on the first directional enhancement frequency domain signal to convert it from the frequency domain to the time domain, thereby obtaining a first directional enhancement audio signal. Similarly, the computer device can perform time-frequency conversion processing on the second directional enhancement frequency domain signal to convert it from the frequency domain to the time domain, thereby obtaining a second directional enhancement audio signal.

[0133] In this embodiment, by acquiring the phases of a first frequency domain signal and a second frequency domain signal, signal amplitude suppression is performed on the first and second frequency domain signals based on the phase of each frequency point and the pickup suppression direction, resulting in a first directional enhanced audio signal and a second directional enhanced audio signal. Specifically, the signal amplitude of the frequency point signal whose phase matches the pickup suppression direction in the first and second frequency domain signals can be reduced, and / or the signal amplitude of the frequency point signal whose phase matches the pickup suppression direction in the first and second frequency domain signals can be increased, resulting in a first directional enhanced frequency domain signal and a second directional enhanced frequency domain signal. Time-frequency transformation processing is then performed on the first and second directional enhanced frequency domain signals respectively to obtain the first directional enhanced audio signal and the second directional enhanced audio signal. In the above method, the correlation between phase and direction is used to suppress / enhance signal amplitude, improving the accuracy of the suppression / enhancement direction, and correspondingly improving the directional accuracy of directional signal enhancement.

[0134] To facilitate understanding by those skilled in the art, the audio processing method provided in this application is described in detail below, such as... Figure 12 As shown, the method may include:

[0135] S1201. Acquire the audio frequency domain signal of each microphone in the two microphone arrays; the two microphone arrays are located on the first hearing aid unit and the second hearing aid unit respectively, and each microphone array includes two microphones;

[0136] S1202. Obtain the phase difference between the two audio frequency domain signals of the first hearing aid unit as the first feature;

[0137] S1203. Obtain the phase difference between the two audio frequency domain signals of the second hearing aid unit as the second feature;

[0138] S1204. Use the first feature and the second feature as phase difference features;

[0139] S1205. Obtain the first amplitude average value of the two audio frequency domain signals of the first hearing aid unit at the same frequency point;

[0140] S1206. Obtain the second amplitude average value of the two audio frequency domain signals of the second hearing aid unit at the same frequency point;

[0141] S1207. Obtain the amplitude difference between the first amplitude mean and the second amplitude mean at the same frequency point, and use it as the amplitude difference feature;

[0142] S1208: Receive the pickup direction indication information and determine the pickup suppression direction based on the direction indication information;

[0143] S1209. Input the phase difference characteristics, amplitude difference characteristics, and pickup suppression direction into the signal enhancement model. Perform signal processing on each audio frequency domain signal through the signal enhancement model to obtain the first directional enhanced audio signal and the second directional enhanced audio signal. The signal processing includes signal amplitude suppression in the pickup suppression direction on each audio frequency domain signal.

[0144] S1210. Play a first directional enhanced audio signal through the speaker in the first hearing aid unit, and play a second directional enhanced audio signal through the speaker in the second hearing aid unit.

[0145] It should be noted that the descriptions in S1201-S1210 above can be found in the relevant descriptions in the above embodiments, and their effects are similar, so they will not be repeated here.

[0146] For example, the hearing aid device includes a first hearing aid unit and a second hearing aid unit, respectively disposed on the user's left and right ear sides. Both the first and second hearing aid units include a microphone array and a speaker, with each microphone array including two microphones. Figure 13 As shown, the control chip in the hearing aid device can perform STFT on the four audio signals collected by the microphone arrays in the first and second hearing aid units to obtain four audio frequency domain signals. It then extracts the first and second features, namely the phase difference (i.e., the phase difference between the single ear and multi-microphone on the same side in the figure) and the amplitude difference feature (i.e., the amplitude difference between the two ears and multi-microphone on both sides in the figure). Simultaneously, it receives the user-input sound pickup direction indication information (i.e., the suppression direction information / enhancement direction information in the figure). The control chip inputs the aforementioned first feature, second feature, amplitude difference feature, and sound pickup direction indication information into a pre-trained signal enhancement model (the neural network in the figure, structured as shown). Figure 14 As shown in the figure, the audio frequency domain signals are fused and the signal amplitude is suppressed. The first directional enhanced audio signal (i.e., the left ear enhanced signal in the figure) and the second directional enhanced audio signal (i.e., the right ear enhanced signal in the figure) are obtained by ISTFT. The control chip then reproduces the first directional enhanced audio signal through the speaker in the first hearing aid unit (i.e., the left ear speaker) and reproduces the second directional enhanced audio signal through the speaker in the second hearing aid unit (i.e., the right ear speaker).

[0147] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0148] Based on the same inventive concept, this application also provides an audio processing apparatus for implementing the audio processing method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more audio processing apparatus embodiments provided below can be found in the limitations of the audio processing method described above, and will not be repeated here.

[0149] In one embodiment, such as Figure 15 As shown, an audio processing device is provided, including: an information acquisition module 1501, a suppression determination module 1502, and a signal processing module 1503, wherein:

[0150] The information acquisition module 1501 is used to acquire the audio frequency domain signal of each microphone in the two microphone arrays; the two microphone arrays are located on the first hearing aid unit and the second hearing aid unit of the hearing aid device, respectively;

[0151] The suppression determination module 1502 is used to receive the pickup direction indication information and determine the pickup suppression direction based on the pickup direction indication information;

[0152] The signal processing module 1503 is used to process the audio frequency domain signals according to the audio frequency domain signals and the pickup direction indication information to obtain a first directional enhanced audio signal and a second directional enhanced audio signal; the signal processing includes adjusting the signal amplitude according to the pickup suppression direction; the first directional enhanced audio signal is played through the first hearing aid unit, and the second directional enhanced audio signal is played through the second hearing aid unit.

[0153] Each module in the aforementioned audio processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0154] In one embodiment, a hearing aid device is provided, including a first hearing aid unit, a second hearing aid unit, and a control chip; both the first and second hearing aid units include a microphone array and a speaker; the control chip is used to implement the steps of any of the above audio processing methods.

[0155] In one embodiment, a hearing aid system is provided, including an input terminal and the aforementioned hearing aid device that communicate with each other. The input terminal is used to send user-inputted pickup direction indication information to the hearing aid device, and the hearing aid device is used to implement the steps of any of the aforementioned audio processing methods.

[0156] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-described audio processing methods.

[0157] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of any of the above-described audio processing methods.

[0158] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0159] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0160] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An audio processing method for a hearing aid device, the hearing aid device comprising a first hearing aid unit and a second hearing aid unit; characterized in that, The method includes: Acquire the audio frequency domain signals of each microphone in two microphone arrays; the two microphone arrays are located on the first hearing aid unit and the second hearing aid unit, respectively; Receive the pickup direction indication information and determine the pickup suppression direction based on the pickup direction indication information; The audio frequency domain signals of each channel are processed according to the pickup suppression direction to obtain a first directional enhanced audio signal and a second directional enhanced audio signal; the signal processing includes adjusting the signal amplitude according to the pickup suppression direction; the first directional enhanced audio signal is played through the first hearing aid unit, and the second directional enhanced audio signal is played through the second hearing aid unit.

2. The method according to claim 1, characterized in that, The step of processing the audio frequency domain signals according to the audio frequency domain signals of each channel and the pickup suppression direction to obtain a first directional enhanced audio signal and a second directional enhanced audio signal includes: Feature extraction is performed on each of the audio frequency domain signals to obtain feature information; The feature information and the pickup suppression direction are input into the signal enhancement model. The signal enhancement model processes each audio frequency domain signal to obtain the first directional enhanced audio signal and the second directional enhanced audio signal. The signal enhancement model is a pre-trained neural network model.

3. The method according to claim 2, characterized in that, The step of processing the audio frequency domain signals of each channel to obtain the first directional enhanced audio signal and the second directional enhanced audio signal includes: The audio frequency domain signals of the first hearing aid unit are fused to obtain the fused first frequency domain signal; The audio frequency domain signals of the second hearing aid unit are fused to obtain the fused second frequency domain signal; The first frequency domain signal and the second frequency domain signal are subjected to signal amplitude suppression in the pickup suppression direction, respectively, to obtain the first directional enhanced audio signal and the second directional enhanced audio signal.

4. The method according to claim 2 or 3, characterized in that, The feature information includes amplitude difference features and phase difference features; The step of extracting features from each of the audio frequency domain signals to obtain feature information includes: The phase difference feature is obtained by extracting the phase information of each audio frequency domain signal from each microphone in the microphone array of the same hearing aid unit. The amplitude difference feature is obtained by extracting the amplitude information of each audio frequency domain signal from each microphone in the microphone array of different hearing aid units.

5. The method according to claim 4, characterized in that, Each of the microphone arrays includes two microphones; the phase difference feature is obtained by extracting the phase information of each audio frequency domain signal from each microphone in the microphone array of the same hearing aid unit, including: The phase difference between the two audio frequency domain signals of the first hearing aid unit is obtained as the first feature; The phase difference between the two audio frequency domain signals of the second hearing aid unit is obtained as a second feature; The first feature and the second feature are used as the phase difference feature.

6. The method according to claim 4, characterized in that, The microphone array includes two microphones; the step of obtaining the amplitude difference feature by extracting the amplitude information of each audio frequency domain signal from each microphone in the microphone array of different hearing aid units includes: The first amplitude average value between the two audio frequency domain signals of the first hearing aid unit at the same frequency point is obtained, and the second amplitude average value between the two audio frequency domain signals of the second hearing aid unit at the same frequency point is obtained. The amplitude difference between the first and second amplitude averages at the same frequency points is obtained as the amplitude difference feature.

7. The method according to claim 3, characterized in that, The step of performing signal amplitude suppression in the pickup suppression direction on the first frequency domain signal and the second frequency domain signal respectively to obtain the first directional enhanced audio signal and the second directional enhanced audio signal includes: Obtain the phase of each frequency point in the first frequency domain signal and the second frequency domain signal; Based on the phase of each frequency point in the first frequency domain signal and the second frequency domain signal and the pickup suppression direction, signal amplitude suppression is performed on the first frequency domain signal and the second frequency domain signal to obtain the first directional enhanced audio signal and the second directional enhanced audio signal.

8. The method according to claim 7, characterized in that, The step of performing signal amplitude suppression on the first frequency domain signal and the second frequency domain signal based on the phase of each frequency point in the first frequency domain signal and the pickup suppression direction to obtain the first directional enhanced audio signal and the second directional enhanced audio signal includes: Reduce the signal amplitude of the frequency point signal whose phase matches the pickup suppression direction in the first frequency domain signal and the second frequency domain signal, and / or increase the signal amplitude of the frequency point signal whose phase does not match the pickup suppression direction in the first frequency domain signal and the second frequency domain signal to obtain the first directional enhancement frequency domain signal and the second directional enhancement frequency domain signal; The first directional enhancement frequency domain signal and the second directional enhancement frequency domain signal are respectively subjected to time-frequency transformation processing to obtain the first directional enhancement audio signal and the second directional enhancement audio signal.

9. A hearing aid device, comprising a first hearing aid unit, a second hearing aid unit, and a control chip; both the first hearing aid unit and the second hearing aid unit include a microphone array and a speaker, characterized in that, The control chip is used to implement the steps of the method according to any one of claims 1 to 8.

10. A hearing aid system, comprising an input terminal and a hearing aid device that communicate with each other, characterized in that, The input terminal is used to send pickup direction indication information to the hearing aid device, and the hearing aid device is used to implement the steps of the method according to any one of claims 1 to 8.

Images