Signal Processing in a Hearing Device

Abstract

The present application discloses signal processing in a hearing device. A method for defining and setting non-linear signal processing of a hearing device by machine learning includes: providing at least one electrical input signal representing at least one input sound signal from the user environment of the hearing device; determining a normal hearing representation of the at least one electrical input signal based on a normal hearing auditory model; determining a hearing-impaired representation of the at least one electrical input signal based on a hearing-impaired auditory model; determining optimized training parameters by machine learning, where determining the optimized training parameters includes repeatedly adjusting the training parameters and comparing the normal hearing representation and the hearing-impaired representation to determine a matching degree between the normal hearing representation and the hearing-impaired representation until the matching degree meets a predetermined requirement; and when the matching degree meets the predetermined requirement, determining corresponding signal processing parameters of the hearing device based on the optimized training parameters.

Description

Technical Field

[0001] The present application relates to hearing devices such as hearing aids or headsets configured to be worn by a user at or in the ear or fully or partially implanted in the head at the user's ear. Background Art

[0002] Cochlear (sensorineural) hearing loss causes many degenerations in the inner ear (cochlea), thus altering the non-linear behavior of a healthy cochlea in many ways. The dynamic hearing range is reduced, and the spectral and temporal resolutions are worse. Consequently, the ability to hear and recognize speech in noisy situations is reduced. In modern hearing aids, this is addressed to some extent by non-linear compression and noise reduction.

[0003] In hearing research, both normal and impaired hearing have been studied using auditory models. These models are based on the physiology of the ear (physiological models), psychoacoustics (behavioral models), or a combination of both. All of these models transform an input signal into some kind of "neural" or "auditory" representation, and the processing steps during the transformation are generally highly non-linear. These models are not two-way models, so they cannot be directly used to "reverse engineer" hearing or derive the mapping of signal processing required to transform an impaired auditory representation into a normal auditory representation.

[0004] In a cochlear implant, the stimulation of the auditory nerve is performed via electrodes inserted into the ear. Compared to the number of available synapses in the cochlea, the implant has a very small number of electrodes, for example 24 electrodes. Each electrode stimulates a series of nearby synapses, and due to current spread, the stimulation is not very specific. Due to the simple speech processing in cochlear implants, limited number of channels and electrodes, and current spread, the sound perception of cochlear implant users is severely degraded compared to normal hearing.

[0005] Hearing loss is usually measured as the auditory threshold of a hearing-impaired user for pure tones (reflected in an audiogram), and a hearing impairment model should enable the average setting of all model parameters based on the audiogram. However, this does not capture all aspects of the individual hearing impairment of a hearing device user. Summary of the Invention

[0006] In one aspect of the present application, a method for defining and setting non-linear signal processing of a hearing device (e.g., for compensating a user's hearing impairment) is provided. The hearing device can be a hearing aid or a headset.

[0007] The method for defining and setting non-linear signal processing of a hearing device can be implemented by machine learning (e.g., supervised machine learning).

[0008] Machine learning can include providing a neural network, such as a deep neural network.

[0009] The method can be a method of training a neural network for defining and setting non-linear signal processing of a hearing device.

[0010] The hearing device can be configured to be worn by a user at or in the ear or to be fully or partially implanted in the head at the user's ear.

[0011] Defining and setting non-linear signal processing and / or training the neural network can be performed during the manufacture of the hearing device product, and / or after the manufacture of the hearing device, and / or after the hearing device has been handed over to the hearing device user (e.g., during use of the hearing device).

[0012] The method can include providing at least one electrical input signal. The electrical input signal can represent at least one input sound signal from the user environment of the hearing device. The at least one input sound signal can include a speech component originating from one or more speech sound sources. The electrical input signal can include a sound containing a noise signal component. The noise signal component can originate from one or more noise sources. The electrical input signal can represent the sound in the user environment of the hearing device. The at least one electrical input signal can include a large amount of material of corresponding electrical input signals from the daily life of the hearing device user. For example, the at least one electrical input signal can include a combination of speech and many types of background noise, can include pure speech, and / or can include music, etc.

[0013] The method can include determining a normal hearing representation of the at least one electrical input signal, such as a "neural representation" [9] similar to the signal in the auditory nerve or an "excitation map"

[10] similar to the psychoacoustic masking map. The normal hearing representation can be determined based on a normal hearing auditory model. The normal hearing representation can refer to the perception of the audio of the at least one electrical input signal, which is assumed to be the perception available to the brain in a normal hearing listener.

[0014] Determining the normal hearing representation can refer to collecting or selecting one or more electrical input signals and corresponding representations. The one or more electrical input signals can originate from a library of speech sound signals and / or noise sound signals and / or other sound signals, such as including music, sounds from a television, electronic sounds, and / or animal sounds. Thereby, a good default version of the normal hearing representation can be obtained during the initial training.

[0015] The method may include determining a hearing impairment representation of at least one electrical input signal. The hearing impairment representation may be based on a hearing impairment auditory model. The hearing impairment representation may refer to the perception of the audio of at least one electrical input signal, which is assumed to be the perception available to the brain of a hearing impaired listener (such as a hearing device user). The main clinical description of an individual's hearing loss is an audiogram (e.g.,

[11] ), where 0 dB HL represents normal hearing and X dB HL represents an X dB hearing loss. Thus, the audiogram can be used as an individual input parameter for the auditory model. The audiogram may include one or more of the following: age-related hearing loss, noise-induced hearing loss, genetic hearing loss, reverse slope hearing loss, and cookie bite hearing loss.

[0016] Determining the hearing impairment representation may refer to collecting or selecting one or more electrical input signals and an individual or general audiogram (such as a prototype audiogram

[11] ), which will be affected by the degree of hearing loss described by the audiogram (such as a prototype audiogram). The one or more electrical input signals may be sourced from a library of speech sound signals and / or noise sound signals and / or other sound signals, such as including music, sounds from a television, electronic sounds, and / or animal sounds. Thereby, a good default version of the hearing impairment representation can be obtained during initial training.

[0017] The hearing impairment representation, the normal hearing representation, and the corresponding auditory model may be used during initial training (and possibly further training), but need not be stored in the hearing device.

[0018] The method may include determining optimized training parameters through machine learning.

[0019] The method may include determining optimized training parameters (weights) for a neural network (algorithm).

[0020] Machine learning can be used to define and set the non-linear signal processing of the hearing device.

[0021] A neural network can be used to define and set the non-linear signal processing of the hearing device by training the neural network.

[0022] Machine learning and / or a neural network can be configured to determine the enhancement and / or attenuation required for at least one electrical input signal at one or more frequencies.

[0023] For example, a neural network can be trained based on a prototype audiogram and a corresponding library of sound signals during the product development phase, such that a good default version of the neural network parameters (and signal processing parameters) can be obtained after the initial training of the neural network.

[0024] Determining the optimized training parameters may include repeatedly adjusting the training parameters.

[0025] Training the neural network may include repeatedly adjusting the training parameters of the neural network.

[0026] Determining optimized training parameters and / or training a neural network may include repeatedly comparing a normal hearing representation and a hearing-impaired representation to determine a degree of match (error metric) between the normal hearing representation and the hearing-impaired representation.

[0027] Determining optimized training parameters and / or training a neural network may include repeatedly adjusting the parameters of the neural network and comparing the normal hearing representation and the hearing-impaired representation to determine a degree of match between the normal hearing representation and the hearing-impaired representation until the degree of match meets a predetermined requirement, such as based on a value function or a loss function, such as mean squared error (MSE).

[0028] The parameters of the neural network may refer to the weights of the neural network.

[0029] The comparison may include comparing one or more parameters that define / determine the normal hearing representation and the hearing-impaired representation, as defined by the respective auditory models applied.

[0030] The predetermined requirement may refer to one or more predetermined values. For example, the predetermined requirement may be that the deviation between the normal hearing representation and the hearing-impaired representation (at all comparison values such as frequencies) must be below a predetermined value in total (such as <20%, <10%, <5%, <2% or others). For example, the predetermined requirement may be that the deviation between the normal hearing representation and the hearing-impaired representation at each comparison value (such as at each frequency) must be below a predetermined value (such as <20%, <10%, <5%, <2% or others).

[0031] The predetermined requirement may include minimizing a value function, i.e., minimizing the difference between the normal hearing representation and the hearing-impaired representation, which will be below a predetermined value.

[0032] The method may include, when the degree of match meets the predetermined requirement, determining corresponding signal processing parameters of the hearing device based on the optimized training parameters.

[0033] The signal processing parameters may refer to gain (such as including providing a mask), noise reduction, enhancement (such as spectral shaping), and / or other parameters related to signal processing.

[0034] Determining the corresponding signal processing parameters of the hearing device may include converting the optimized parameters of the neural network into the corresponding signal processing parameters of the hearing device. For example, determining the corresponding signal processing parameters of the hearing device may include converting the optimized weights of the neural network into the corresponding signal processing parameters of the hearing device.

[0035] Thus, an output signal may be provided in the ear of the hearing-impaired user, which has an improved similarity to the corresponding normal hearing perception.

[0036] The method includes providing at least one output signal representing a stimulus that can be perceived as sound by a user of a hearing device, based on signal processing parameters. Providing at least one output signal based on signal processing parameters may include enhancing and / or amplifying an electrical input signal to provide the output signal.

[0037] Providing at least one electrical input signal may include providing a plurality of electrical input signals. For example, at least two electrical input signals may be provided. For example, at least three electrical input signals may be provided.

[0038] The plurality of electrical input signals may include a large collection of corresponding electrical input signals from the daily life of the hearing device user, such as a combination of speech and many types of background noise, pure speech, music, etc. The corresponding electrical input signals may be stored on a storage device as a library of audio signals and corresponding audiograms. Thus, a good representation of hearing impairment (and corresponding signal processing parameters) may be obtained after initial training.

[0039] The plurality of electrical input signals may be provided one at a time. Thereby, training parameters of a neural network and corresponding signal processing parameters may be determined for a first electrical input signal, and further adjusted for a second electrical input signal, and so on, until the neural network has been trained based on all of the electrical input signals in the plurality of electrical input signals.

[0040] More than two of the plurality of electrical input signals may be provided simultaneously. Thereby, the training parameters of the neural network and the corresponding signal processing parameters may be determined at once based on more than two electrical input signals, which is a time - efficient way, until the neural network has been trained based on all of the electrical input signals in the plurality of electrical input signals.

[0041] The method may further include transforming the electrical input signal into a spectrum.

[0042] The method may transform the electrical input signal into a spectrum using an analysis filter bank.

[0043] The method may transform the electrical input signal into a spectrum by performing a Fourier transform on the electrical input signal.

[0044] A hearing device such as an input unit and / or an antenna and transceiver circuit may include a TF conversion unit for providing a time - frequency representation of at least one electrical input signal. The time - frequency representation may include an array or mapping of corresponding complex - valued or real - valued numbers of the signal involved in a specific time and frequency range. The TF conversion unit may include an analysis filter bank for filtering the (time - varying) input signal and providing a plurality of (time - varying) output signals, each output signal including a distinct frequency range of the input signal. The TF conversion unit may include a Fourier transform unit for converting the time - varying input signal into a (time -) frequency - domain (time - varying) signal. Thereby, a frequency - decomposed electrical input signal is provided.

[0045] Analysis filter banks can also be designed to mimic frequency decomposition similar to human hearing, such as the Gammatone filter bank (

[12] ).

[0046] The normal auditory threshold of hearing and the hearing-impaired threshold of hearing can be parameterized by an audiogram.

[0047] An audiogram can represent a normal-hearing user by approaching zero dB HL (e.g., up to 15 dB HL maximum).

[0048] An audiogram can represent a hearing-impaired user by having values greater than zero dB HL (e.g., > 30 dB HL).

[0049] The physiology of the ear (physiological model), psychoacoustics (behavioral model), or a combination of both can be parameterized by an audiogram of normal hearing and / or an audiogram of hearing impairment.

[0050] Thereby, input parameters that are easy to compare and clinically acceptable are provided.

[0051] Determining optimized training parameters through machine learning can include determining the optimized training parameters of a neural network by training the neural network.

[0052] The neural network can be a deep neural network.

[0053] The neural network and / or the deep neural network provide the possibility of performing computationally intensive training.

[0054] A deep neural network (DNN) can transform an input signal into N output samples / coefficients of the same type using N samples / coefficients. The neural network can be a traditional feedforward DNN without memory [1], or a long short-term memory (LSTM) or convolutional recurrent neural network (CRNN) [1], both of which contain memory and are thus able to learn from previous input samples.

[0055] The DNN can include an autoencoder.

[0056] When using a traditional feedforward DNN, it can also be modified into a so-called autoencoder [2], where the middle layer of the network has a dimension smaller than the input and output dimensions N. This transforms the input into a simpler representation containing essential features, which can then be modified to obtain a given result. These denoising and super-resolution autoencoders have been successfully used to enhance noisy and blurred images back to noise-free high-resolution images [3].

[0057] The method can also include providing at least one supra-threshold (super-threshold) metric.

[0058] By providing at least one supra-threshold metric as an input to an auditory model, the ability of a neural network to adjust its parameters is increased until the best match between the normal hearing representation and the hearing impaired representation is achieved.

[0059] Determining the hearing impaired representation of at least one electrical input signal may also be based on at least one supra-threshold metric.

[0060] Hearing loss is typically measured as the auditory threshold (audiogram) of a hearing impaired user for pure tones, and the hearing impaired model should enable the setting of all other model parameters on average based on the audiogram. However, this does not capture all aspects of individual hearing impairment. Supra-threshold metrics can characterize, for example, extended auditory filtering [4], loss of cochlear compression [5], or spectral-temporal modulation detection [6]. Thus, the parameters in the hearing impaired representation (the impaired auditory model) can be further based on (adapted to) the supra-threshold metrics. The supra-threshold metrics can be individual measurements (custom measurements) [5]. The supra-threshold metrics can provide individualized (customized) training of the neural network of a hearing device.

[0061] In a physiological model, for example, cochlear compression can be measured and then inserted into the model [5]. Another important factor in the model is the estimation of the relative loss of inner and outer hair cells from the total hair cell loss (estimated from the audiogram). This can be estimated via a threshold equalising noise (TEN) test, where the auditory threshold is measured using masking noise

[15] . In a psychoacoustic model, the equivalent rectangular bandwidth (ERB) can be measured using a notch noise method [4].

[0062] Thus, the training of the neural network can be further based on at least one supra-threshold metric obtained from a hearing device user. Thereby, the neural network (and thus the hearing device) can be further trained to adapt to the needs of an individual hearing device user. The further training of the neural network can be referred to as "transfer learning" [1]. The further training of the neural network can be performed during production, during the initial fitting of the hearing device to the user, and / or after the hearing device user has worn the hearing device for a certain period of time.

[0063] The further training of the neural network can be performed after the hearing device user has started using the hearing device and audio samples (measurements) of the actual sound environment encountered by the hearing device user in their daily life have been collected.

[0064] At least one supra-threshold metric can include extended auditory filtering.

[0065] At least one supra-threshold metric can include loss of cochlear compression.

[0066] At least one supra-threshold metric can include spectral-temporal modulation detection.

[0067] At least one suprathreshold metric may include two or more of extended auditory filtering, loss of cochlear compression, and spectro-temporal modulation detection.

[0068] The normal-hearing auditory model and the hearing-impaired auditory model may be based on the same auditory model.

[0069] The method may also include providing at least one audiogram.

[0070] Determining the hearing-impaired representation of at least one electrical input signal may also be based on the at least one audiogram.

[0071] At least one audiogram may include an audiogram specific to the hearing device user and / or a general (generic) audiogram.

[0072] The hearing-impaired auditory model may be based on the normal-hearing auditory model.

[0073] The output representations (parameters) of the normal-hearing auditory model and the hearing-impaired auditory model may be the same. Thus, optimal training of the neural network may occur because the output parameters of the normal-hearing auditory model and the hearing-impaired auditory model may be directly compared.

[0074] The normal-hearing auditory model and the hearing-impaired auditory model may be the same. Thus, the types of input parameters required by the auditory model and the types of output parameters of the auditory model may be the same, but the specific input parameters (such as values) of the auditory model may vary depending on whether the user has normal hearing or is hearing-impaired. Thus, optimal training of the neural network may occur because the output parameters of the normal-hearing auditory model and the hearing-impaired auditory model may be directly compared.

[0075] An audiogram, an auditory filter bandwidth, and / or cochlear compression measurement results may be used to represent the parameters of the auditory model.

[0076] An error metric may be provided. The error metric may include, for example, the root mean square (RMS) error across channels, or may be a perceptually-based different error based on different weightings in the model. A simple example of perceptual weighting may be the band weighting used in the Speech Intelligibility Index (SII) (ANSI S3.5, 2007), where the relative importance of each band to speech intelligibility is multiplied by the speech level in the same band and then summed across bands. Thus, the aggregate error metric across the entire training set may be used as a result metric for training quality and matching to normal hearing. It may also be possible to construct other error metrics based on internal variables or representations in the auditory model, which will reflect the matching quality between the normal-hearing representation and the hearing-impaired representation.

[0077] In one aspect of the present application, a hearing device is provided. The hearing device may include a hearing aid or a headset. The hearing device may be adapted to be worn in or at the user's ear. The hearing device may be adapted to be fully or partially implanted in the user's head.

[0078] The hearing device may include an input unit for receiving an input sound signal from the environment of the hearing device user. The hearing device may provide at least one electrical input signal representative of the input sound signal.

[0079] The hearing device may include an output unit for providing at least one set of stimuli perceptible by the user as sound based on a processed version of at least one electrical input signal.

[0080] The hearing device may include a processing unit.

[0081] The processing unit may be connected to the input unit.

[0082] The processing unit may be connected to the output unit.

[0083] The processing unit may include the signal processing parameters of the hearing device.

[0084] The processing unit may include the signal processing parameters of the hearing device to provide a processed version of at least one electrical input signal.

[0085] The signal processing parameters may be determined based on optimized training parameters. The optimized training parameters may be determined by machine learning. The optimized training parameters may belong to a neural network. The neural network may include a deep neural network.

[0086] The neural network may be trained according to the method described above.

[0087] The training of the neural network may be performed in a server such as a cloud server. Thus, the training may be distributed to the server, and the hearing device may receive a trained version of the signal processing parameters.

[0088] The training of the neural network may be performed at least partially in an external device such as a mobile device. Thus, the training may be at least partially distributed to the external device, and the hearing device may receive a trained version of the signal processing parameters.

[0089] Since training a neural network is a computationally intensive task, performing the training outside the hearing device, such as in a server or an external device, may reduce the power consumption of the hearing device.

[0090] The training of the neural network, or at least part of the training, can be performed by a smartphone connected to the hearing device or by the hearing device itself. Thus, the hearing device can be configured to be trained during initial production and fitting for the hearing device user, but can also be trained after the user has received the hearing device, where the hearing device does not have to be connected to a server or external device, or at least only needs to be connected to a server or external device only at certain intervals.

[0091] The neural network can provide a processed version of at least one electrical input signal. The neural network can, for example, be configured to provide a signal processing factor (varying with frequency and time) to be applied to at least one electrical input signal (or its processed version). The signal processing factor can be configured to compensate for the hearing impairment of the hearing device wearer (user). The hearing device can be configured to provide a processed version of at least one electrical input signal via the neural network.

[0092] The hearing device can be configured to provide at least one set of stimuli perceptible by the user as sound based on the processed version of at least one electrical input signal based on signal processing parameters.

[0093] The hearing device can be configured to provide at least one output signal by the output unit based on signal processing parameters, which represents a stimulus perceptible by the hearing device user as sound.

[0094] The processing unit can include a deep neural network trained according to the method described above. The processing unit can be configured to provide a processed version of at least one electrical input signal. The processing unit can be configured to provide a processed version of at least one electrical input signal based on the trained deep neural network described above.

[0095] The hearing device can be configured to be further trained based on audio representing sounds in the user's environment.

[0096] The audio representing sounds in the user's environment can include speech from the user or from other people, music, audio from the television, naturally occurring audio, etc.

[0097] The further training of the neural network for defining and setting the non-linear signal processing of the hearing device can be performed after the user has started using the hearing device. Thus, the further training can be regarded as a further customization of the hearing device for the user, such that the hearing device provides / applies the optimal signal processing parameters in the user's environment (i.e., in the environment where the hearing device operates normally).

[0098] The hearing device can include an analysis filter bank for transforming an electrical input signal into a spectrum.

[0099] The analysis filter bank can be arranged behind (downstream) the input unit. The analysis filter bank can be arranged in front of the processing unit.

[0100] The hearing device may include a Fourier transform of an electrical input signal.

[0101] The analysis filter bank (and the hearing device) may be configured to provide a frequency decomposition (spectrum) of the electrical input signal.

[0102] The analysis filter bank (and the hearing device) may be configured to provide the frequency-decomposed version (spectrum) of the electrical input signal to a processing unit.

[0103] The electrical input signal may be provided directly in the time domain as a waveform or may be transformed to the frequency domain using, for example, an analysis filter bank or a Fourier transform.

[0104] The processing unit may provide a normal hearing representation (based on a normal hearing auditory model) corresponding to the electrical input signal to a neural network.

[0105] The processing unit may provide a hearing-impaired representation (based on a hearing-impaired auditory model) corresponding to the electrical input signal to a neural network.

[0106] The hearing device may include a synthesis filter bank for transforming at least one electrical output signal into a signal representing a stimulus that can be perceived as sound by the user of the hearing device.

[0107] The synthesis filter bank may be arranged behind the processing unit. The synthesis filter bank may be arranged in front of the output unit of the hearing device.

[0108] The synthesis filter bank may convert the frequency-decomposed signal (spectrum) into a time-decomposed signal.

[0109] The hearing device may include a gain module. The hearing device may include a mask. The hearing device may include a gain module incorporating a mask.

[0110] The hearing device may provide a non-linear time-varying gain. The gain module of the hearing device may provide a non-linear time-varying gain. The non-linear time-varying gain may be calculated / determined by machine learning, such as by a neural network, and may be applied directly.

[0111] The hearing device (such as the gain module) may include a time-frequency mask. For example, the time-frequency mask may be used in combination with noise reduction and / or beamforming, where a multiplication factor (e.g., between 0 and 1, e.g., a binary mask) that is a function of frequency and time may be applied.

[0112] The mask and / or the gain module may be arranged in front of the output unit. The mask and / or the gain module may be arranged in front of the synthesis filter bank. The mask may be arranged behind the processing unit that provides the signal processing parameters of the hearing device.

[0113] The neural network may estimate a time-frequency mask, which may then be applied to the frequency-decomposed electrical input signal (spectrum).

[0114] A hearing device may be configured as or include a hearing aid.

[0115] The hearing aid may be configured to be fully or partially implanted in the head at the user's ear.

[0116] A hearing device may be configured as or include headphones.

[0117] A hearing device may be configured as or include an earphone.

[0118] A hearing device may be configured as or include an ear protection device.

[0119] A hearing device may be configured as or include a combination of a hearing aid, headphones, an earphone, and an ear protection device.

[0120] The hearing device may be adapted to provide gain that varies with frequency and / or compression that varies with level and / or frequency shifting from one or more frequency ranges to one or more other frequency ranges (with or without frequency compression) to compensate for the user's hearing impairment. The hearing device may include a signal processor for enhancing the input signal and providing a processed output signal.

[0121] The hearing device may include an output unit for providing a stimulus that is perceived by the user as an acoustic signal based on the processed electrical signal. The output unit may include a plurality of electrodes of a cochlear implant (for CI-type hearing devices) or a vibrator of a bone-conduction hearing device. The output unit may include an output transducer. The output transducer may include a receiver (loudspeaker) for providing the stimulus as an acoustic signal to the user (e.g., in an acoustic (air-conduction-based) hearing device). The output transducer may include a vibrator for providing the stimulus as a mechanical vibration of the skull to the user (e.g., in a bone-attached or bone-anchored hearing device).

[0122] The hearing device may include an input unit for providing an electrical input signal representative of sound. The input unit may include an input transducer such as a microphone for converting the input sound into an electrical input signal. The input unit may include a wireless receiver for receiving a wireless signal including or representing sound and providing an electrical input signal representative of the sound. The wireless receiver may be configured, for example, to receive electromagnetic signals in the radio frequency range (3 kHz to 300 GHz). The wireless receiver may be configured, for example, to receive electromagnetic signals in the optical frequency range (e.g., infrared light 300 GHz to 430 THz, or visible light, e.g., 430 THz to 770 THz).

[0123] A hearing device may include a directional microphone system that is adapted to spatially filter sounds from the environment so as to enhance a target sound source among a plurality of sound sources in the local environment of a user wearing the hearing device. The directional system is adapted to detect (such as adaptively detect) from which direction a particular part of a microphone signal originates. This can be achieved in a variety of different ways as described in the prior art. In hearing devices, microphone array beamformers are typically used to spatially attenuate background noise sources. Many beamformer variants can be found in the literature. The minimum variance distortionless response (MVDR) beamformer is widely used in microphone array signal processing. Ideally, the MVDR beamformer leaves the signal from the target direction (also known as the look direction) unchanged while maximally attenuating sound signals from other directions. The generalized sidelobe canceller (GSC) structure is an equivalent representation of the MVDR beamformer, which offers computational and digital representation advantages over a direct implementation of the original form.

[0124] A hearing device may include an antenna and transceiver circuitry (such as a wireless receiver) for receiving a direct electrical input signal from another device such as from an entertainment device (such as a television), a communication device, a wireless microphone, or another hearing device. The direct electrical input signal may represent or include an audio signal and / or a control signal and / or an information signal. The hearing device may include demodulation circuitry for demodulating the received direct electrical input to provide a direct electrical input signal representing an audio signal and / or a control signal, for example for setting operating parameters (such as volume) and / or processing parameters of the hearing device. Generally, the wireless link established by the antenna and transceiver circuitry of the hearing device can be of any type. The wireless link is established between two devices, for example between an entertainment device (such as a TV) and the hearing device, or between two hearing devices, for example via a third intermediate device (such as a processing device, such as a remote control device, a smart phone, etc.). The wireless link is used under power constraints, for example since the hearing device may be or include a portable (usually battery-powered) device. The wireless link is a near-field communication-based link, for example an inductive link based on inductive coupling between the antenna coils of a transmitter part and a receiver part. The wireless link may be based on far-field electromagnetic radiation. Communication via the wireless link is arranged according to a particular modulation scheme, for example an analog modulation scheme such as FM (frequency modulation) or AM (amplitude modulation) or PM (phase modulation), or a digital modulation scheme such as ASK (amplitude shift keying) such as on-off keying, FSK (frequency shift keying), PSK (phase shift keying) such as MSK (minimum shift keying) or QAM (quadrature amplitude modulation), etc.

[0125] Communication between the hearing device and another device can be in the baseband (audio frequency range, e.g., between 0 and 20 kHz). Preferably, the frequency used to establish a communication link between the hearing device and another device is below 70 GHz, e.g., in the range from 50 MHz to 70 GHz, e.g., above 300 MHz, e.g., in the ISM range above 300 MHz, e.g., in the 900 MHz range or in the 2.4 GHz range or in the 5.8 GHz range or in the 60 GHz range (ISM = Industrial, Scientific and Medical, such standardized ranges are defined, for example, by the International Telecommunication Union ITU). The wireless link is based on a standardized or proprietary technology. The wireless link is based on Bluetooth technology (such as Bluetooth Low Energy technology).

[0126] The hearing device can be a portable (i.e., configured to be wearable) device or form part of it, such as a device including a native energy source like a battery, e.g., a rechargeable battery. The hearing device can be a lightweight and easily wearable device, e.g., having a total weight of less than 100 g, e.g., less than 10 g.

[0127] The hearing device can include a forward or signal path between an input unit (such as an input transducer, e.g., a microphone or a microphone system and / or a direct electrical input (such as a wireless receiver)) and an output unit such as an output transducer. A signal processor is located in this forward path. The signal processor is adapted to provide frequency-dependent gain according to the specific needs of the user. The hearing device can include an analysis path having functions for analyzing the input signal (such as determining the level, modulation, signal type, acoustic feedback estimate, etc.). Part or all of the signal processing in the analysis path and / or the signal path can be performed in the frequency domain. Part or all of the signal processing in the analysis path and / or the signal path can be performed in the time domain.

[0128] The analog electrical signal representing the acoustic signal can be converted to a digital audio signal during an analog-to-digital (AD) conversion process, where the analog signal is sampled at a predetermined sampling frequency or sampling rate f s and sampled, f s e.g., in the range from 8 kHz to 48 kHz (adapting to the specific needs of the application) at discrete time points t n (or n) to provide digital samples x n (or x[n]), each audio sample being represented by a predetermined N b bits representing the value of the acoustic signal at t n and N b e.g., in the range from 1 to 48 bits, such as 24 bits. Each audio sample is thus quantized using N b bits (resulting in 2 Nb different possible values for the audio sample). The digital sample x has a time length of 1 / f s e.g., 50 μs, for f s= 20 kHz. Multiple audio samples may be arranged in time frames. A time frame may include 64 or 128 audio data samples. Other frame lengths may be used according to the actual application.

[0129] The hearing device may include an analog-to-digital (AD) converter to digitize an analog input (e.g., from an input transducer such as a microphone) at a predetermined sampling rate, such as 20 kHz. The hearing device includes a digital-to-analog (DA) converter to convert the digital signal into an analog output signal, e.g., for presentation to the user via an output transducer.

[0130] The hearing device, such as the input unit and / or the antenna and transceiver circuitry, includes a TF conversion unit for providing a time-frequency representation of the input signal. The time-frequency representation may include an array or mapping of the corresponding complex or real values of the signal involved in a specific time and frequency range. The TF conversion unit may include a filter bank for filtering the (time-varying) input signal and providing multiple (time-varying) output signals, each output signal including a distinct input signal frequency range. The TF conversion unit may include a Fourier transform unit for converting the time-varying input signal into a (time-)frequency domain (time-varying) signal. The frequency range considered by the hearing device, from the minimum frequency f min to the maximum frequency f max may include a part of the typical human audible frequency range from 20 Hz to 20 kHz, e.g., a part of the range from 20 Hz to 12 kHz. Generally, the sampling rate f s is greater than or equal to twice the maximum frequency f max , i.e., f s ≥ 2f max . The signals in the forward path and / or the analysis path of the hearing device may be split into NI (e.g., uniformly wide) frequency bands, where NI is, for example, greater than 5, such as greater than 10, such as greater than 50, such as greater than 100, such as greater than 500, and at least some of them are processed individually. The hearing device is adapted to process the signals in the forward and / or analysis path in NP different channels (NP ≤ NI). The channels may have the same or different widths (e.g., the width increases with frequency), and may overlap or not overlap.

[0131] The hearing device may be configured to operate in different modes, such as a normal mode and one or more specific modes, e.g., which may be selected by the user or may be automatically selected. The operating mode may be optimized for a specific acoustic situation or environment. The operating mode may include a low-power mode, in which the functions of the hearing device are reduced (e.g., for energy saving), e.g., disabling wireless communication and / or disabling specific features of the hearing device.

[0132] The hearing device may include a plurality of detectors configured to provide status signals related to the current network environment of the hearing device, such as the current acoustic environment, and / or related to the current state of the user wearing the hearing device, and / or related to the current state or operating mode of the hearing device. As an alternative or in addition, one or more of the detectors may form part of an external device communicating with the hearing device (such as wirelessly). The external device may for example include another hearing device, a remote control, an audio transmission device, a telephone (such as a smart phone), an external sensor, etc.

[0133] One or more of the plurality of detectors may act on the full-band signal (time domain). One or more of the plurality of detectors may act on the frequency-band split signal ((time-)frequency domain), for example in a limited number of frequency bands.

[0134] The plurality of detectors may include a level (L) detector for estimating the current level of the signal in the forward path. The detector may be configured to determine whether the current level of the signal in the forward path is above or below a given (L-)threshold. The level detector acts on the full-frequency band signal (time domain). The level detector acts on the frequency-band split signal ((time-)frequency domain).

[0135] The hearing device may include a voice activity detector (VAD) for estimating whether (or with what probability) the input signal (at a particular point in time) includes a voice signal. In this specification, the voice signal includes speech signals from humans. It may also include other forms of vocalization (such as singing) produced by the human voice system. The voice activity detector unit is adapted to classify the user's current acoustic environment as a "voice" or "no voice" environment. This has the advantage that time periods of the microphone signal including human vocalization (such as speech) in the user's environment can be identified and thus separated from time periods including only (or mainly) other sound sources (such as artificially generated noise). The voice activity detector may be adapted to also detect the user's own voice as "voice". As an alternative, the voice activity detector may be adapted to exclude the user's own voice from the detection of "voice".

[0136] The hearing device may include a self-voice detector for estimating whether (or with what probability) a particular input sound (such as a voice, such as speech) originates from the voice of the user of the hearing system. The microphone system of the hearing device may be adapted to be able to distinguish the user's own voice from the voice of another person and possibly from non-voice sounds.

[0137] The plurality of detectors may include a motion detector, such as an acceleration sensor. The motion detector may be configured to detect motion of the user's facial muscles and / or bones caused for example by speech or chewing (such as jaw movement) and provide a detector signal indicating the motion.

[0138] A hearing device may include a classification unit configured to classify the current situation based on an input signal from (at least in part) a detector and possibly other inputs. In this specification, the "current situation" is defined by one or more of the following:

[0139] a) The physical environment (such as including the current electromagnetic environment, for example, the presence of electromagnetic signals (including audio and / or control signals) that are planned or unplanned to be received by the hearing device, or other properties of the current environment that are different from acoustic);

[0140] b) The current acoustic situation (input level, feedback, etc.);

[0141] c) The current mode or state of the user (motion, temperature, cognitive load, etc.);

[0142] d) The current mode or state of the hearing device and / or another device communicating with the hearing device (selected program, time elapsed since the last user interaction, etc.).

[0143] The classification unit may be based on or include a neural network, such as a trained neural network.

[0144] The hearing device may also include other suitable functions for the applications involved, such as compression, noise reduction, feedback control, etc.

[0145] The hearing device may include a listening device such as a hearing aid, a hearing instrument, for example, a hearing instrument adapted to be located at the user's ear or fully or partially located in the ear canal, such as a headset, an earphone, an ear protection device, or a combination thereof. The hearing assistance system may include a loudspeaker (including a plurality of input transducers and a plurality of output transducers, for example, used in an audio conferencing situation), for example, including a beamforming filter unit, for example, providing multiple beamforming capabilities.

[0146] When appropriately replaced by corresponding processes, some or all of the structural features of the device described above, detailed in the "Detailed Description" or defined in the claims, may be combined with the implementation of the method of the present invention, and vice versa. The implementation of the method has the same advantages as the corresponding device.

[0147] Application

[0148] On the one hand, there is provided an application of the hearing device as described above, detailed in the "Detailed Description" section and defined in the claims. An application in a system including audio distribution can be provided, such as a system including a microphone and a speaker that are close enough to each other to cause feedback from the speaker to the microphone during user operation. Applications in systems including one or more hearing aids (such as hearing instruments), headsets, earphones, active ear protection systems, etc. can be provided, such as uses in hands-free telephone systems, remote conferencing systems (such as including loudspeakers), broadcast systems, karaoke systems, classroom amplification systems, etc.

[0149] Computer-readable medium or data carrier

[0150] The present invention further provides a tangible computer-readable medium (data carrier) storing a computer program including program code (instructions), which, when the computer program runs on a data processing system, causes the data processing system (computer) to execute (complete) at least part (such as most or all) of the steps of the method described above, detailed in the "Detailed Description" and defined in the claims.

[0151] By way of example and not limitation, the foregoing tangible computer-readable medium may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store or execute the required program code in the form of instructions or data structures and can be accessed by a computer. As used herein, a disk includes a compact disk (CD), laser disk, optical disk, digital versatile disk (DVD), floppy disk and Blu-ray disk, where these disks typically magnetically replicate data while these disks can optically replicate data using a laser. Other storage media include those stored in DNA (such as in synthetic DNA strands). Combinations of the above disks should also be included within the scope of computer-readable media. In addition to being stored on a tangible medium, a computer program may also be transmitted via a transmission medium such as a wired or wireless link or network such as the Internet and loaded into the data processing system to run at a location different from the tangible medium.

[0152] Computer program

[0153] Furthermore, the present application provides a computer program (product) including instructions, which, when the program runs on a computer, causes the computer to execute the method (steps) described above, detailed in the "Detailed Description" and defined in the claims.

[0154] Data processing system

[0155] On the one hand, the present invention further provides a data processing system, including a processor and program code, the program code causing the processor to execute at least part (such as most or all) of the steps of the methods described above, detailed in the "Detailed Description" and defined in the claims.

[0156] Hearing system

[0157] On the other hand, there is provided a hearing system including the hearing device and the auxiliary device described above, detailed in the "Detailed Description" and defined in the claims.

[0158] The hearing system is adapted to establish a communication link between the hearing device and the auxiliary device so that information (such as control and status signals, possibly audio signals) can be exchanged or forwarded from one device to another device.

[0159] The auxiliary device may include a remote control, a smart phone, or other portable or wearable electronic devices such as a smart watch, etc.

[0160] The auxiliary device may be constituted by a remote control or may include a remote control for controlling the functions and operations of the hearing device. The functions of the remote control are implemented in a smart phone, and the smart phone may run an APP enabling the functions of controlling the audio processing device via the smart phone (the hearing device includes a suitable wireless interface to the smart phone, for example, based on Bluetooth or some other standardized or proprietary solution).

[0161] The auxiliary device may be or include an audio gateway device, which is adapted to receive multiple audio signals (for example, from entertainment devices such as TVs or music players, from telephone devices such as mobile phones, or from computers such as PCs) and is adapted to select and / or combine appropriate signals (or signal combinations) among the received audio signals for transmission to the hearing device.

[0162] The auxiliary device may be constituted by another hearing device or may include another hearing device. The hearing system may include two hearing devices adapted to implement a binaural hearing system such as a binaural hearing aid system.

[0163] There is disclosed a hearing system including left and right hearing devices according to the above. The left and right hearing devices are configured to be worn respectively in or at the left and right ears of a user, and / or to be respectively implanted completely or partially in the head at the left and right ears of the user, and are configured to establish a wired or wireless connection therebetween so that data such as audio data can be exchanged therebetween, optionally via an intermediate device.

[0164] APP

[0165] On the other hand, the present invention also provides a non-transitory application called APP. The APP includes executable instructions configured to run on an auxiliary device to implement a user interface for the hearing device or hearing system described above, detailed in the "Detailed Description" and defined in the claims. The APP is configured to run on a mobile phone such as a smart phone or another portable device enabling communication with the hearing device or hearing system.

[0166] Define

[0167] In this specification, a "hearing device" refers to a device suitable for improving, enhancing, and / or protecting a user's auditory ability, such as a hearing aid, for example, a hearing instrument or an active ear protection device or other audio processing device, which is achieved by receiving an acoustic signal from the user's environment, generating a corresponding audio signal, possibly modifying the audio signal, and providing the possibly modified audio signal as an audible signal to at least one ear of the user. A "hearing device" also refers to a device suitable for electronically receiving an audio signal, possibly modifying the audio signal, and providing the possibly modified audio signal as an audible signal to at least one ear of the user, such as a headset or earphone. The audible signal can be provided, for example, in the form of an acoustic signal radiated into the user's outer ear, an acoustic signal transmitted as mechanical vibrations through the bone structure of the user's head and / or through parts of the middle ear to the user's inner ear, and an electrical signal transmitted directly or indirectly to the user's cochlear nerve.

[0168] The hearing device can be configured to be worn in any known manner, such as a unit worn behind the ear (with a tube for guiding the radiated acoustic signal into the ear canal or with an output transducer arranged close to or in the ear canal, such as a speaker), a unit arranged wholly or partly in the auricle and / or ear canal, a unit connected to a fixed structure implanted in the skull, such as a vibrator, or a unit that can be connected or wholly or partly implanted, etc. The hearing device can include a single unit or several units communicating with each other (for example, acoustically, electrically, or optically). The speaker can be provided in a housing together with other components of the hearing device, or it can itself be an external unit (possibly combined with a flexible guiding element such as a dome-shaped element).

[0169] More generally, a hearing device includes an input transducer for receiving an acoustic signal from the user's environment and providing a corresponding input audio signal and / or a receiver for receiving the input audio signal electronically (i.e., wired or wirelessly), a (usually configurable) signal processing circuit for processing the input audio signal (such as a signal processor, for example including a configurable (programmable) processor, such as a digital signal processor), and an output unit for providing an audible signal to the user based on the processed audio signal. The signal processor may be adapted to process the input signal in the time domain or in multiple frequency bands. In some hearing devices, an amplifier and / or a compressor may form part of the signal processing circuit. The signal processing circuit typically includes one or more (integrated or separate) storage elements for executing programs and / or for storing parameters used (or potentially used) in the processing and / or for storing information suitable for the function of the hearing device and / or for storing information used, for example, in connection with the interface to the user and / or to a programming device (such as processed information, for example provided by the signal processing circuit). In some hearing devices, the output unit may include an output transducer, such as a loudspeaker for providing an air-conducted acoustic signal or a vibrator for providing a structure-borne or fluid-borne acoustic signal. In some hearing devices, the output unit may include one or more output electrodes for providing an electrical signal (for example to a multi-electrode array) for electrically stimulating the cochlear nerve (cochlear implant hearing aids). The hearing device may include a horn loudspeaker (including multiple input transducers and multiple output transducers), for example used in an audio conferencing scenario.

[0170] In some hearing devices, the vibrator may be adapted to transmit a structure-borne acoustic signal to the skull transcutaneously or through the skin. In some hearing devices, the vibrator may be implanted in the middle ear and / or the inner ear. In some hearing devices, the vibrator may be adapted to provide a structure-borne acoustic signal to the middle ear bones and / or the cochlea. In some hearing devices, the vibrator may be adapted to provide a fluid-borne acoustic signal to the cochlear fluid, for example through the oval window. In some hearing devices, the output electrodes may be implanted in the cochlea or on the inner side of the skull and may be adapted to provide an electrical signal to the hair cells of the cochlea, one or more auditory nerves, the auditory brainstem, the auditory midbrain, the auditory cortex, and / or other parts of the cerebral cortex.

[0171] Hearing devices such as hearing aids can be adapted to the needs of a particular user, such as hearing impairment. The configurable signal processing circuit of the hearing device may be adapted to apply compression amplification that varies with frequency and level to the input signal. Customized gain (amplification or compression) that varies with frequency and level can be determined during the fitting process by a fitting system based on the user's hearing data, such as an audiogram, using fitting principles (such as adapted to speech). The gain that varies with frequency and level can be embodied, for example, in processing parameters, such as uploaded to the hearing device via an interface to a programming device (fitting system) and used by a processing algorithm executed by the configurable signal processing circuit of the hearing device.

[0172] "Hearing system" refers to a system including one or two hearing devices. "Binaural hearing system" refers to a system including two hearing devices and adapted to cooperatively provide audible signals to both ears of a user. The hearing system or binaural hearing system may also include one or more "auxiliary devices" that communicate with the hearing devices and affect and / or benefit from the functions of the hearing devices. The aforementioned auxiliary devices may include at least one of the following: a remote control, a remote microphone, an audio gateway device, an entertainment device such as a music player, a wireless communication device such as a mobile phone (e.g., a smart phone), a tablet computer, or another device such as one including a graphical interface. The hearing device, hearing system, or binaural hearing system may be used, for example, to compensate for the loss of auditory ability of a hearing-impaired person, enhance or protect the auditory ability of a person with normal hearing, and / or transmit an electronic audio signal to a person. The hearing device or hearing system may form a part of or interact with, for example, a broadcast system, an active ear protection system, a hands-free phone system, an automotive audio system, an entertainment (such as TV, music playing, or karaoke) system, a teleconference system, a classroom amplification system, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0173] Various aspects of the present invention will be best understood from the following detailed description taken in conjunction with the accompanying drawings. For clarity, these drawings are schematic and simplified, showing only the details necessary for understanding the present invention and omitting other details. Throughout the specification, the same reference numerals are used for the same or corresponding parts. Each feature of each aspect may be combined with any or all features of other aspects. These and other aspects, features, and / or technical effects will be apparent from and elucidated in conjunction with the following drawings, in which:

[0174] Figure 1 An exemplary application scenario of the training of a neural network for defining and setting non-linear signal processing of a hearing device system according to the present invention is shown;

[0175] Figure 2 An exemplary application scenario of the training of a neural network for defining and setting non-linear signal processing of a hearing device system according to the present invention is shown;

[0176] Figure 3 An exemplary application scenario of the training of a neural network for defining and setting non-linear signal processing of a hearing device system according to the present invention is shown;

[0177] Figure 4 An exemplary application scenario of a hearing device according to the present invention is shown;

[0178] Figure 5 An exemplary application scenario of a hearing device according to the present invention is shown;

[0179] Figure 6Shows an exemplary application scenario of a hearing device according to the present invention;

[0180] Figure 7 Shows a hearing device according to an embodiment of the present invention, which uses a trained (user-personalized) neural network to control the processing of a signal representing sound in the hearing device before the processed signal is presented to the user wearing the hearing device;

[0181] Figure 8 Shows an exemplary application scenario of an auditory model for providing a representation of normal hearing or hearing impairment.

[0182] The further scope of applicability of the present invention will become apparent from the detailed description given below. However, it should be understood that while the detailed description and the specific examples, which indicate preferred embodiments of the invention, are given by way of illustration only, for a person skilled in the art other embodiments of the present invention will become apparent based on the following detailed description. Detailed Description of the Invention

[0183] The following detailed description presented in conjunction with the accompanying drawings serves as a description of various different configurations. The detailed description includes specific details for providing a thorough understanding of a plurality of different concepts. However, it will be apparent to those skilled in the art that these concepts may be implemented without these specific details. Several aspects of the apparatus and method are described by means of a plurality of different blocks, functional units, modules, elements, circuits, steps, processes, algorithms, etc. (collectively referred to as "elements"). Depending on a particular application, design constraint or other reasons, these elements may be implemented using electronic hardware, computer programs or any combination thereof.

[0184] The electronic hardware may include microelectromechanical systems (MEMS), (e.g., application-specific) integrated circuits, microprocessors, microcontrollers, digital signal processors (DSPs), field-programmable gate arrays (FPGAs), programmable logic devices (PLDs), gated logic, discrete hardware circuits, printed circuit boards (PCBs) (such as flexible PCBs), and other suitable hardware configured to perform the various different functions described in this specification, such as sensors for sensing and / or recording physical properties of the environment, the device, the user, etc. A computer program should be construed broadly as instructions, instruction sets, code, code segments, program code, programs, subroutines, software modules, applications, software applications, software packages, routines, subroutines, objects, executables, execution threads, programs, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language or otherwise.

[0185] This application relates to the field of hearing devices, such as hearing aids, headsets, earphones, and / or ear protection devices.

[0186] Figure 1An exemplary application scenario for the training of a neural network for defining and setting non-linear signal processing of a hearing device system according to the present invention is shown.

[0187] In Figure 1 , at S1, at least one electrical input signal may be provided. The at least one electrical input signal may represent at least one input sound signal from the user environment of the hearing device. The at least one input sound signal may include one or more speech signals from one or more sound sources and may include additional signal components (referred to as noise signal components) from one or more other sound sources. The electrical input signal may be provided in the time domain as a waveform. The electrical input signal may be provided by an input unit of the hearing device.

[0188] At S2, an analysis filter bank (and / or Fourier transform unit) may be provided. The at least one electrical input signal may be provided to (and processed by) the analysis filter bank (and / or Fourier transform unit). In the analysis filter bank, the electrical input signal may be transformed into the frequency domain. The analysis filter bank may be configured to transform the at least one electrical input signal into a (frequency-decomposed) spectrum. The analysis filter bank may be provided behind the input unit of the hearing device.

[0189] At S4, based on the at least one electrical input signal in the time domain or frequency domain, a normal hearing representation may be determined. The determination of the normal hearing representation may be based on the normal hearing auditory model provided at S3. The normal hearing representation may be an output in the form of an audiogram.

[0190] At S6, based on the at least one electrical input signal in the time domain or frequency domain, an initial hearing-impaired representation may be determined. The determination of the hearing-impaired representation may be based on the hearing-impaired auditory model provided at S5. The hearing-impaired representation may be an output in the form of an audiogram.

[0191] The normal hearing representation and the hearing-impaired representation may be provided, for example, to an error metric module for providing an error metric at S16 (to determine the degree of match).

[0192] Based on an input including the electrical input signal, the normal hearing representation, and the hearing-impaired representation, optimized training parameters of the provided neural network may be determined at S7. As Figure 1 shown, the neural network may be a deep neural network. The step of determining the optimized training parameters of the neural network may result in signal processing parameters of the hearing device.

[0193] Auditory models have long been used as research tools and for developing auditory mechanisms and further as approximate front-ends for further analysis and processing of sound signals for different purposes, such as

[14] .

[0194] There are two basic and different types of auditory models:

[0195] 1) Physiological models, which represent different functional stages of ear anatomy: the outer ear, the middle ear, and the inner ear. The inner ear is described by the basilar membrane, outer hair cells, inner hair cells, synapses, spiral ganglion, auditory nerve, midbrain, etc. These models have generally been designed and validated using animal data such as auditory nerve fiber recordings in cats.

[0196] 2) Psychoacoustic models (sometimes also called phenomenological models), which are based on functional metrics of the human ear, such as frequency masking, loudness growth, etc. The advantage of this model is that it can be validated through different classical psychoacoustic tests. On the other hand, compared with physiological models, the output is less rich / detailed.

[0197] Depending on the type of auditory model, the representation can be based on different interpretations. In physiological models, the representation can represent the output of the auditory nerve [7] or midbrain (brainstem) neural activity [9]. In psychoacoustic models, the representation can be an "excitation map"

[11]

[13] , similar to a mask map or a "specific" loudness map, which is the loudness subdivided into frequency bands

[13]

[14] .

[0198] Training of the neural network can include repeatedly performing the following steps: adjusting the training parameters of the neural network, comparing the normal hearing representation with the hearing-impaired representation to determine the degree of match between the normal hearing representation and the hearing-impaired representation. Adjusting the training parameters of the neural network can include adjusting the weights of the neural network.

[0199] For example, training can include comparing an initial hearing-impaired audiogram and a normal hearing audiogram by providing an error metric at S16, and repeatedly adjusting the training parameters of the neural network (and the corresponding signal processing parameters of the hearing-impaired representation) such that the hearing-impaired audiogram to which the determined signal processing has been applied approaches the normal hearing audiogram.

[0200] Based on the comparison between the normal hearing representation and the hearing-impaired representation, the degree of match between the normal hearing representation and the hearing-impaired representation can be determined. For example, the comparison can include comparing the normal hearing audiogram and the hearing-impaired audiogram at corresponding electrical input signals.

[0201] Training can be performed until the degree of match meets a predetermined requirement. For example, the predetermined requirement can refer to one or more predetermined values. For example, the predetermined requirement can be that the deviation between the normal hearing representation and the hearing-impaired representation (at all comparison values such as frequencies) must be below a predetermined value in total (such as <20%, <10%, <5%, <2% or others). For example, the predetermined requirement can be that the deviation between the normal hearing representation and the hearing-impaired representation at each comparison value (such as at each frequency) must be below a predetermined value (such as <20%, <10%, <5%, <2% or others).

[0202] In Figure 1In this case, as shown in the figure, steps S3, S4, S5, S6, S7, and S16 can be executed in a unit (marked by a dotted line box). This unit can be a processing unit of the hearing device.

[0203] This unit can also be a server such as a cloud server or a mobile device. Thus, the server or the mobile device can perform computationally intensive training, and the hearing device can receive the trained version of the neural network or the resulting signal processing parameters.

[0204] After the training has been completed based on the electrical input signal, the training can be repeated based on one or more additional electrical input signals, so that the neural network and the corresponding signal processing parameters can be further adjusted (fine-tuned) to make the output signal for the hearing-impaired user as close as possible to the audio signal received by a normal-hearing user.

[0205] During the (initial) training of the neural network, that is, before the hearing device user starts using the hearing device, the user can provide additional audiological data for training the neural network. Providing additional audiological data can include providing, in S8, one or more audiograms, for example, from the daily life (environment) of a hearing-impaired user, such as speech combined with many types of background noise, pure speech, music, etc. The one or more audiograms can be based on one or more of the following: age-related hearing loss, noise-induced hearing loss, genetic hearing loss, reverse-slope hearing loss, and cookie-bite hearing loss.

[0206] Providing additional audiological data can also include providing, in S9, supra-threshold metrics, such as extended auditory filtering, loss of cochlear compression, or spectral-temporal modulation detection. The supra-threshold metrics in S9 can be represented as frequency-specific Q values (vectors of filter slopes), frequency-specific compression ratios (compression vectors (CR)), scalar values such as spectral-temporal modulation thresholds (dB) [6], or other forms of input parameters.

[0207] Additionally or alternatively, after the hearing device user has started using the hearing device, the neural network can be further trained based on additional audiological data.

[0208] When the matching degree meets the predetermined requirements, the corresponding signal processing parameters for the hearing device can be determined.

[0209] In the case where the electrical input signal is provided to the analysis filter bank or Fourier-transformed, in S10, the output from the neural network can be provided to the synthesis filter bank. In the synthesis filter bank, the output from the neural network can be transformed back to the time domain.

[0210] Based on the output from the neural network (the processed version of at least one electrical input signal) or the output from the synthesis filter bank, in S11, at least one output signal representing a stimulus that can be perceived as sound by the hearing device user is provided.

[0211] Figure 2 Shows an exemplary application scenario for the training of a neural network for defining and setting non - linear signal processing of a hearing device system according to the present invention.

[0212] Figure 1 The main part of the training steps of the neural network shown in is similar to Figure 2 the training steps in. Thus, reference is made to the description above Figure 1 .

[0213] Figure 2 Differing from Figure 1 is that, in S12, a gain is provided by a gain module or, as an alternative, a mask such as a processed version of at least one electrical input signal is provided.

[0214] As shown, the mask or gain module can receive inputs directly from the analysis filter bank and / or the neural network. The neural network can estimate the gain as a time - varying gain, and the time - varying gain can be provided for hearing loss compensation. The neural network can estimate the mask as a time - frequency mask, which can then be applied to the frequency - decomposed electrical input signal (spectrum) during neural network training. The mask or gain module can be arranged in front of the output unit of the hearing device. The mask or gain module can be arranged in front of the synthesis filter bank. Thus, the mask or gain module can be applied to the adjusted hearing - impaired representation during neural network training. In addition, the mask or gain module can be applied to the processed signal from the neural network to the synthesis filter bank of the hearing device and to the output unit.

[0215] In addition, Figure 2 the method of the exemplary application scenario of can further include providing standard hearing aid multi - channel compression or other standard hearing aid algorithms. The standard hearing aid multi - channel compression or other standard hearing aid algorithms can be provided before or after the gain module (or time - frequency mask). Thus, the neural network and the mask and / or gain module may be providing the unknown "rest" of the processing required for time - frequency processing.

[0216] Figure 3 Shows an exemplary application scenario for the training of a neural network for defining and setting non - linear signal processing of a hearing device system according to the present invention.

[0217] Figure 1 and Figure 2 the main part of the training steps of the neural network shown in is similar to Figure 3 the training steps in. Thus, reference is made to the description above Figure 1 and Figure 2 .

[0218] Figure 3 Differing from Figure 1 and Figure 2The difference lies in providing a model for electrical stimulation, such as

[16] , instead of an auditory model for the hearing-impaired. The model for electrical stimulation can be used, for example, in the case where the hearing device is a cochlear implant.

[0219] The input of such a model can be an electric current. Therefore, the cochlear implant itself needs to be included in the system to provide the transformation from acoustic stimulation to electrical stimulation using pulse coding strategies and audio signal processing. The model can be established for stimulating an individual's hearing (such as the user's electro-auditory threshold and discomfort, both of which are standard metrics). Otherwise, the training and optimization procedures can be similar to Figure 1 and Figure 2 the similar ones.

[0220] Thus, in Figure 3 the pulse generation can be provided by the cochlear implant at S13. The cochlear implant for pulse generation can be fitted through the fitting parameters provided at S14 (such as hearing level, discomfort, etc.). The fitting parameters can be based on one or more of the following: age-related hearing loss, noise-induced hearing loss, genetic hearing loss, etc.

[0221] The pulse generation can be provided to the electrically stimulated auditory model provided at S15. The electrically stimulated auditory model can also be provided with supra-threshold metrics. The hearing impairment indication can be provided at S6 based on the electrically stimulated auditory model.

[0222] When the matching degree meets a predetermined requirement (such as when an error metric is provided at S16), an output signal can be provided at S11.

[0223] Figure 4 An exemplary application scenario of the hearing device according to the present invention is shown.

[0224] In Figure 4 it shows a hearing device 1, which can be a hearing aid, adapted to be worn in or at the user's ear, and / or adapted to be fully or partially implanted in the user's head.

[0225] The hearing device 1 can include an input unit 2 for receiving an input sound signal from the user environment of the hearing device and providing at least one electrical input signal representing the input sound signal.

[0226] The hearing device 1 can include an analysis filter bank 3. In the analysis filter bank 3, at least one electrical input signal can be transformed into the frequency domain. Therefore, the analysis filter bank 3 can be configured to transform at least one electrical input signal into a (frequency-decomposed) spectrum.

[0227] The processing unit 4 of the hearing device 1 can be connected to the input unit 2, for example, via the analysis filter bank 3. The processing unit 4 can include a trained version of a neural network. In Figure 4In [the figure], it is shown that the neural network can be a deep neural network 5. When the neural network is trained (the initial training has been completed), the parameters of the neural network are optimized and fixed. The processing unit 4 (and the hearing device 1) can be configured to determine the corresponding signal processing parameters of the hearing device 1 based on the fixed training parameters of the neural network. In addition, the processing unit 4 (and the hearing device 1) can be configured to provide, for example, a processed version (i.e., the processed signal) of at least one electrical input signal from the input unit 2 via the analysis filter bank 3.

[0228] The processing unit 4 (and the hearing device 1) can be configured to provide the processed version of at least one electrical input signal, for example, via the synthesis filter bank 6, to the output unit 7 of the hearing device 1. The output unit 7 can be configured to convert the processed signal or a signal derived therefrom into at least one set of stimuli that can be perceived as sound by the user of the hearing device.

[0229] Figure 5 An exemplary application scenario of the hearing device according to the present invention is shown.

[0230] Figure 5 The main part of the hearing device 1 shown in [the figure] is similar to Figure 4 the hearing device 1 shown in [the figure]. Therefore, refer to the above Figure 4 description.

[0231] Figure 5 It is different from Figure 4 in that a gain module 8 or a mask is applied.

[0232] As shown in the figure, the gain module 8 or the mask can directly receive inputs from the analysis filter bank 3 and / or the neural network (deep neural network 5). The neural network can estimate the non-linear time-varying gain of the gain module 8.

[0233] The neural network can estimate the mask as a time-frequency mask, which can then also be applied to the frequency-decomposed electrical input signal (spectrum) during the neural network training, as described above.

[0234] Therefore, the processing unit 4 (and the hearing device 1) can be configured to apply the gain module 8 or the mask to the processed signal from the neural network, for example, via the synthesis filter bank 6, to the output unit 7 of the hearing device 1.

[0235] As an alternative, instead of applying the processing unit 4 including the deep neural network 5, a conventional hearing device processing unit 4a can be applied.

[0236] Figure 6 An exemplary application scenario of the hearing device according to the present invention is shown.

[0237] Figure 6 The main part of the hearing device 1 shown in [the figure] is similar to Figure 4 andFigure 5 is similar to the hearing device 1 shown therein. Thus, reference is made to the description above Figure 4 and Figure 5 .

[0238] Figure 6 relates to the case where the hearing device can be a cochlear implant. Thus, Figure 6 different from Figure 4 and Figure 5 in that a pulse generation module 6a is applied instead of the synthesis filter bank 6. Optionally, the pulse generation module 6a can be adapted by adaptation parameters 6b. Thus, the output unit 7 can provide, for example, an enhanced pulse sequence as an output signal.

[0239] Figure 7 shows a hearing device HD according to an embodiment of the invention, which uses a trained (personalized for a specific user as described in this specification) neural network NN* to control the processing of the signal representing sound in the hearing device before the processed signal is presented to the user wearing the hearing device. The hearing device HD includes an input unit IU that provides electrical input signals (IN1, IN2) from corresponding microphones (M1, M2), for example. The input unit IU includes, for example, corresponding analysis filter banks to provide the electrical input signals (IN1, IN2) in sub-band representation (k, m), where k and m are frequency and time indices, respectively. The hearing device can include a pre-processor Pre-Pro that receives (at least one) electrical input signal (IN1, IN2) and prepares an appropriate input vector FV for the neural network NN*. The input vector FV can include one or more time frames of the electrical input signal or a processed version thereof (e.g., extracted features of one of the signals). The output signal GAIN of the optimized neural network NN* is fed to the processor PRO of the hearing device. The processor PRO receives the electrical input signals (IN1, IN2) (or their beamformed versions) from the input unit IU and processes the signals according to the output GAIN of the neural network NN*. The output signal GAIN can represent, for example, a time-varying gain G (e.g., a gain G(k, m) varying with time and frequency), which will be applied to the signals in the forward path, e.g., to one of the electrical input signals (IN1, IN2) or its spatially filtered version. On this basis, the processor PRO provides a processed output OUT, which is fed to an output unit OU, for example including an output transducer such as a loudspeaker or a vibrator, for presenting a stimulus that can be perceived as sound by the user of the hearing device. The output unit OU (or the processor) can include a synthesis filter bank (inverse filter bank) for converting the sub-band signals into time-domain signals before presenting them to the output transducer.

[0240] Figure 8 shows an exemplary application scenario of an auditory model for providing a normal hearing representation or a hearing impairment representation.

[0241] In Figure 8 it, the auditory model module 9 can receive input parameters via the input module 10 and can provide an output via the output module 11.

[0242] The input parameters can include an audiogram, an auditory filter bandwidth, cochlear compression measurement results, and / or a sound file.

[0243] The auditory model module 9 can include a control module 12 for controlling the input parameters. The control module 12 can be configured to check that all input parameters are correctly set. When one or more input parameters are not correctly set, default values can be set to ensure that the auditory model functions correctly.

[0244] The auditory model module 9 can also include an aggregation module 13. The aggregation module 13 can be configured to combine / collect one or more functions of the auditory model. The aggregation module 13 can be configured to receive the results of (one or more functions of) the auditory model. The aggregation module 13 can be configured to output and / or transmit the received results to the output module 11.

[0245] One or more functions of the auditory model that can be combined / collected by the aggregation module 13 can include the following modules:

[0246] A read module 14, configured to read a sound signal and provide a matrix. The rows of the matrix can be the desired input frame size, and the columns can be controlled by the sound signal file and the desired input frame size. The output of this function can be set such that it saves each frame row by row in matrix form. This means that the first input frame can be the first row of the matrix, the second frame can be the second row of the matrix, and so on.

[0247] A power spectrum module 15, configured to calculate the power spectrum of one frame at a time. For example, the power spectrum can be calculated by first applying a Hann window to the time signal and then converting it to frequency by calculating the fast Fourier transform (FFT) of the time signal. Thereby, the power spectrum of the first half of the FFT output can be calculated. The first half of the power spectrum can be used to remove the mirror frequency components generated by the FFT.

[0248] A first corrected spectrum module 16, configured to correct the power spectrum to the sound field.

[0249] A second corrected spectrum module 17, configured to correct the power spectrum for equal loudness contours (e.g., equivalent of the outer and middle ear transfer functions).

[0250] An equivalent rectangular bandwidth (ERB) energy module 18, configured to calculate the energy in each ERB frequency band (without overlap). This function can also calculate the ERB for each frame and save it as a vector. The vectors for each frame can be saved row by row such that a matrix can be included finally.

[0251] The excitation map module 19 is configured to calculate an excitation map based on the output of the ERB energy module 18. This module 19 can save the excitation map for each frame, which means the final output can be expected to be in matrix form.

[0252] The excitatory degree module 20 is configured to output the total excitatory degree when outputting each frame.

[0253] The loudness module 21 is configured to calculate the specific loudness and loudness vector in a frame, which can be performed based on the output from the excitation map module 19, or can also be performed based on the outputs from the sound pressure level (SPL) module 22, the hearing threshold level (HTL) module 23, and the discomfort level (UCL) module 24.

[0254] When appropriately replaced by corresponding processes, the structural features of the devices described above, detailed in the "Detailed Description" and defined in the claims, can be combined with the steps of the method of the present invention.

[0255] Unless explicitly stated otherwise, the singular forms "a", "the" used herein are intended to include the plural forms (i.e., having the meaning of "at least one"). It should be further understood that the terms "having", "including" and / or "comprising" used in the specification indicate the presence of the stated features, integers, steps, operations, elements and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or their combinations. It should be understood that unless explicitly stated otherwise, when an element is referred to as being "connected" or "coupled" to another element, it can be directly connected or coupled to other elements, or there can be intervening elements. As used herein, the term "and / or" includes any and all combinations of one or more of the listed related items. Unless explicitly stated otherwise, the steps of any method disclosed herein do not necessarily have to be performed in the exact order disclosed.

[0256] It should be realized that the mention of "an embodiment" or "embodiments" or "aspect" or "may" include features in the specification means that the specific features, structures or characteristics described in connection with that embodiment are included in at least one embodiment of the present invention. In addition, the specific features, structures or characteristics can be appropriately combined in one or more embodiments of the present invention. The foregoing description is provided to enable those skilled in the art to implement the various aspects described herein. Various modifications will be apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects.

[0257] The claims are not limited to the aspects shown herein, but cover the full scope consistent with the claim language, where unless explicitly stated otherwise, an element referred to in the singular does not mean "one and only one", but rather "one or more". Unless explicitly stated otherwise, the term "some" means one or more.

[0258] Accordingly, the scope of the present invention should be determined by the claims.

[0259] References

[0260] [1]Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning, MIT Press.

[0261] [2]Bengio, Y. (2009). “Learning Deep Architectures for AI,” Found. Mach. Learn., doi:10.1561 / 2200000006. doi:10.1561 / 2200000006.

[0262] [3]Kun, Z., Yu, J., Wang, R., Li, C., and Tao, D. (2015). “Coupled Deep Autoencoder for Single Image Super-Resolution,” IEEE Trans. Cybern., 47, 1–11.

[0263] [4]Stone, M. A., Glasberg, B. R., and Moore, B. C. J. (1992). “Simplified measurement of impaired auditory filter shapes using the notched-noise method,” Br. J. Audiol., 26, 329–334.

[0264] [5]Jepsen, M. L., and Dau, T. (2011). “Characterizing auditory processing and perception in individual listeners with sensorineural hearing loss,” J. Acoust. Soc. Am., 129, 262–281. doi:10.1121 / 1.3518768.

[0265] [6]Bernstein, J.G.W., Mehraei, G., Shamma, S., Gallun, F.J., Theodoroff, S.M., and Leek, M.R. (2013). “Spectrotemporal modulation sensitivity as a predictor of speech intelligibility for hearing-impaired listeners,” J. Am. Acad. Audiol., 24, 293–306. doi:10.3766 / jaaa.24.4.5.

[0266] [7]Zilany, M.S.A., and Bruce, I.C. (2006). “Modeling auditory-nerve responses for high sound pressure levels in the normal and impaired auditory periphery,” J. Acoust. Soc. Am., 120, 1446–1466. doi:10.1121 / 1.2225512.

[0267] [8]Jepsen, M.L., and Dau, T. (2011). “Characterizing auditory processing and perception in individual listeners with sensorineural hearing loss,” J. Acoust. Soc. Am., 129, 262–281. doi:10.1121 / 1.3518768.

[0268] [9]Carney, L.H. (2017). “Fluctuation Contrast and Speech-on-Speech Masking: Model Midbrain Responses to Simultaneous Speech,” Proc. Int. Symp. Audit. Audiol. Res., 6, 75–82.

[0269]

[10] Chen, Zhangli, Guangshu Hu, Brian R. Glasberg, and Brian C. J. Moore. "A new method of calculating auditory excitation patterns and loudness for steady sounds." Hearing research 282, no. 1-2 (2011): 204-215.

[0270]

[11] Bisgaard, Nikolai, Marcel S. M. G. Vlaming, and Martin Dahlquist. “Standard Audiograms for the IEC 60118-15 Measurement Procedure.” Trends in Amplification 14, no. 2 (June 2010): 113–20. doi:10.1177 / 1084713810379609.

[0271]

[12] Hohmann, V. (2002). Frequency analysis and synthesis using a Gammatone filterbank. Acta Acustica united with Acustica, 88(3), 433-442.

[0272]

[13] Fastl, H., and Zwicker, E. (2007). Psychoacoustics, Psychoacoustics Facts Model., Springer Berlin Heidelberg, Berlin, Heidelberg. doi:10.1007 / 978-3-540-68888-4.

[0273]

[14] L. (2004). “An Objective Estimate of the Perceived Quality of Reproduced Sound in Normal and Impaired Hearing,” Acta Acust. united with Acust., 90, 1007–1018.

[0274]

[15] Moore,B.C.J.,Glasberg,B.R.,and Stone,M.A.(2004).“New version ofthe TEN test with calibrations in dB HL,”Ear Hear.,doi:10.1097 / 01.aud.0000145992.31135.89.

[0275]

[16] Grange,J.A.,Culling,J.F.,Harris,N.S.L.,and Bergfeld,S.(2017).“Cochlear implant simulator with independent representation of the fullspiral ganglion,”J.Acoust.Soc.Am.,doi:10.1121 / 1.5009602.

Claims

1. A method for defining and setting non-linear signal processing of a hearing device by machine learning, the hearing device being configured to be worn by a user at or in the ear or fully or partially implanted in the head at the user's ear, the method comprising: Provide at least one electrical input signal representing at least one input sound signal from the user environment of the hearing device; Provide at least one supra-threshold metric, the at least one supra-threshold metric including customized measurement results; Determine a normal hearing representation of the at least one electrical input signal based on a normal hearing auditory model; Determine a hearing-impaired representation of the at least one electrical input signal based on a hearing-impaired auditory model and the at least one supra-threshold metric; Determine optimized training parameters by machine learning, wherein determining the optimized training parameters includes repeatedly comparing the normal hearing representation and the hearing-impaired representation to determine a matching degree between the normal hearing representation and the hearing-impaired representation, and adjusting the training parameters based on the matching degree between the normal hearing representation and the hearing-impaired representation; Until the matching degree meets a predetermined requirement; And When the matching degree meets the predetermined requirement, determine corresponding signal processing parameters of the hearing device based on the optimized training parameters.

2. The method according to claim 1, wherein, Providing at least one electrical input signal includes providing a plurality of electrical input signals.

3. The method according to claim 1, wherein the method further comprises transforming an electrical input signal into a spectrum.

4. The method according to claim 1, wherein the signal processing parameters include gain, noise reduction, enhancement and / or other signal processing parameters.

5. The method according to claim 1, wherein, Determining optimized training parameters by machine learning includes determining optimized training parameters of a neural network by training the neural network, and wherein the neural network is a deep neural network.

6. The method according to claim 5, wherein the deep neural network includes an autoencoder.

7. The method according to claim 1, wherein at least one supra-threshold metric includes extended auditory filtering, loss of cochlear compression and / or spectral-temporal modulation detection.

8. The method according to claim 1, wherein the normal hearing auditory model and the hearing impaired auditory model are based on the same auditory model.

9. The method according to claim 1, wherein the method further comprises providing at least one audiogram, and wherein determining the hearing impaired representation of at least one electrical input signal is also based on the at least one audiogram.

10. The method according to claim 9, wherein at least one audiogram includes an audiogram specific to the hearing device user and / or a general audiogram.

11. The method according to claim 1, wherein the hearing impaired auditory model is based on the normal hearing auditory model.

12. A hearing device adapted to be worn in or at the user's ear and / or fully or partially implanted in the user's head, the hearing device comprising: An input unit for receiving an input sound signal from the environment of the hearing device user and providing at least one electrical input signal representing the input sound signal; An output unit for providing at least one set of stimuli perceptible by the user as sound based on a processed version of the at least one electrical input signal; A processing unit connected to the input unit and the output unit, which includes signal processing parameters of the hearing device to provide a processed version of the at least one electrical input signal, wherein the signal processing parameters are determined based on the optimized training parameters determined according to any one of claims 1-11.

13. The hearing device according to claim 12, wherein the processing unit includes a deep neural network that provides optimized training parameters, and the deep neural network is trained according to the method of any one of claims 1-11.

14. The hearing device according to claim 12, wherein the hearing device is configured to be further trained based on audio representing sounds in the user's environment.

15. The hearing device according to claim 12, wherein the hearing device includes an analysis filter bank for transforming an electrical input signal into a spectrum.

16. The hearing device according to claim 15, wherein the hearing device includes a synthesis filter bank for transforming the spectrum into a time-domain signal.

17. The hearing device according to claim 12, wherein the hearing device includes a mask and / or gain module.

18. The hearing device according to claim 12, which constitutes or includes a hearing aid, a headset, an earphone, an ear protection device, or a combination thereof.

19. A hearing system, comprising left and right hearing devices according to any one of claims 12-18, wherein the left and right hearing devices are configured to be worn respectively in or at the user's left and right ears, and / or configured to be fully or partially implanted respectively in the head at the user's left and right ears, and configured to establish a wired or wireless connection therebetween to enable data exchange therebetween.

20. A computer-readable medium having stored thereon a program comprising instructions that, when executed by a computer, cause the computer to perform the steps of the method of any one of claims 1-11.

Images