Hearing devices or systems including communication interfaces

By introducing a signal predictor and hybrid processing technology into the hearing aid, the problem of signal transmission delay between the external wireless microphone and the hearing aid is solved, achieving better real-time sound processing and improving the user experience of the hearing aid.

CN115226016BActive Publication Date: 2026-11-10OTICON
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202210397429.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2021-04-15
Filing Date
2022-04-15
Publication Date
2026-11-10
Estimated Expiration
2042-04-15

AI Technical Summary

Technical Problem

In existing hearing aids, the signal transmission delay between the external wireless microphone and the hearing aid causes audiovisual synchronization problems and comb filter effects, affecting the real-time sound processing effect.

Method used

The system estimates the future value of the sound signal using a signal predictor, combines the acoustic and wireless received signals for mixing, compensates for transmission delay, and adjusts the mixing ratio using signal quality metrics to achieve real-time sound processing.

Benefits of technology

It effectively reduces signal delay, improves the real-time performance and clarity of sound signals, and enhances the user experience of hearing devices.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115226016B_ABST
    Figure CN115226016B_ABST
Patent Text Reader

Abstract

This application discloses a hearing device or system including a communication interface, wherein the hearing device includes at least one input converter, a wireless receiver, and a processor; the processor is configured to receive at least one acoustically received electrical input signal, receive a wirelessly received electrical input signal, and provide a processed signal; the processor includes: a signal predictor for estimating future values ​​of the wirelessly received electrical input signal based on a plurality of past values ​​of the wirelessly received electrical input signal to provide a predicted signal; and an output converter for presenting a perceptible sound-like output stimulus to a user based on the processed signal from the processor or a further processed version thereof; wherein the processor is configured to provide the processed signal, either alone or in combination with the at least one acoustically received electrical input signal or a further processed version thereof, based on the predicted signal or its processed version.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to hearing systems or hearing devices, such as hearing aids or headphones or similar portable audio processing devices. Background Technology

[0002] Hearing systems such as hearing aid (HA) systems (e.g., including one or two hearing aids) can be connected to one or more external wireless microphones, such as tabletop microphones, to facilitate the hearing aid user's perception of speech from speakers in the user's environment. When the wireless microphone picks up the sound of interest, the signal is buffered, encoded, and possibly packetized before being transmitted electromagnetically to the HA. This process delays the sound signal of interest. The precise delay is a function of the audio encoding algorithm and transmission scheme used in the wireless system (e.g., Bluetooth or UWB protocols). However, the introduced delay can be significant, i.e., several times 10 ms. If such a delayed sound signal is presented to the hearing aid user, it can cause audiovisual synchronization problems, such as lip reading, and / or a comb-filter effect due to the direct sound arriving at the user's eardrum much earlier. In other words, such a delay is essentially useless for real-time processing / presentation of the received sound signal to the user wearing the HA. Summary of the Invention

[0003] On one hand, the present invention relates, for example, to a situation where an external sound capture device, such as a wireless microphone, transmits audio to a hearing device, such as a hearing aid, and the signal is predicted in the hearing device based on a wirelessly received signal. The signal presented to the user by the speaker of the hearing device can be the predicted signal, or a) a mixture of the predicted signal and b) the acoustically received signal (picked up by the microphone of the hearing device). The mixture can be based on a frequency range level (e.g., a frequency band level) and can vary over time (e.g., according to a signal quality estimate).

[0004] Instead of estimating the signal to be enhanced, we can also estimate the gain (across time and frequency) to amplify the desired portion of the signal and reduce noise when multiplied by the gain signal. We can also predict future values ​​of such gain signals. The signal predictor can be configured to estimate the future value of the noise reduction gain signal based on multiple past values ​​of the input signal and / or past values ​​of the gain signal.

[0005] In an embodiment, the signal presented to the user by the speaker of the hearing device may be a mixture of a) a predicted signal, b) an acoustically received signal (picked up by the microphone of the hearing device), and c) a (unpredicted) wirelessly received signal from a sound capture device. The signal components from the predicted signal a) are ideally "timely" and "clean"; the signal components from the acoustically received signal b) are "timely" but noisy; and the signal components from the (unpredicted) wirelessly received signal c) are "not timely" (previously) and "clean". In certain situations (time periods) and / or frequency ranges, presenting previously received signal components instead of noisy (or predicted) signal components may be useful. This may depend on signal quality metrics such as signal-to-noise ratio (SNR) or speech intelligibility (SI) index.

[0006] In another aspect, the present invention provides a hearing device or system, such as a hearing aid, comprising at least one earpiece configured to be worn on or in a user's ear, and a separate audio processing device (in communication with the earpiece). The earpiece or audio processing device may include a signal predictor for estimating future values ​​of the signal (originally received in the earpiece) or a processed version thereof based on multiple past values ​​of the acoustically received signal, thereby providing a predicted signal. The objective of the signal predictor is to compensate for or reduce delays caused by processing performed in an external processing device. The signal predictor may be configured to fully or partially compensate for processing delays caused by one or more, for example, all of: a) the transmission of the acoustically received electrical input signal from the hearing device to the audio processing device, b) processing in the audio processing device, and c) the transmission of the predicted signal or a processed version thereof to the earpiece and its reception in the earpiece.

[0007] In this invention, a solution is proposed that makes a portion of the received sound signal useful for real-time processing at the HA regardless of the specific signal received. In fact, the proposed solution is a general one and can be found in a very wide range of wireless audio applications. Our basic solution is based on the idea of ​​predicting future portions of the sound signal given a current signal. The use of predictive algorithms to address the problem of eliminating acoustically propagated sound reaching the eardrum in hearing devices has been addressed, for example, in EP3681175A1.

[0008] Hearing device

[0009] In one aspect of this application, a hearing device, such as a hearing aid, is provided. The hearing device includes:

[0010] - At least one input converter for converting sounds in the environment of the hearing device into electrical input signals representing at least one acoustically received sound;

[0011] - A wireless receiver for receiving audio signals from the wireless transmitter of a sound capture device that picks up sounds in the environment and providing an electrical input signal representing the wireless reception of the sounds;

[0012] - Processor, configured as

[0013] --Receive the at least one acoustically received electrical input signal or a processed version thereof;

[0014] --Receive the wirelessly received electrical input signal; and

[0015] --Provide processed signals.

[0016] The processor may include a signal predictor for providing a predicted signal by estimating future values ​​of the signal (or estimating future values ​​of the gain to be applied to the signal) based on multiple past values ​​of the wirelessly received electrical input signal.

[0017] The hearing device also includes an output converter for presenting a perceptible sound-like output stimulus to the user based on the processed signal from the processor or a further processed version thereof.

[0018] The processor can be configured to provide the processed signal based on the predicted signal or a processed version thereof in the following manner:

[0019] - alone, or

[0020] - Mixed with the at least one acoustically received electrical input signal or its processed version.

[0021] The signal presented to the user by the output transducer (e.g., a loudspeaker) of the hearing device (processed signal) can be a mixture of a) a predicted signal, b) an electrical input signal received at least acoustically (picked up by at least one input transducer (e.g., a microphone) of the hearing device, and c) an electrical input signal received wirelessly (unpredicted) from a sound capture device. In certain cases (time periods) and / or frequency ranges, a trade-off may be made between presenting previously (unpredicted but clean) signal components and noisy (or predicted) (but timely) signal components. Mixing can be performed based on signal quality metrics (e.g., signal-to-noise ratio (SNR) or speech intelligibility (SI) index, and / or speech presence probability (SPP) index), for example, at the frequency band level.

[0022] Thus, a hearing device can be provided that enhances the utilization of sound transmitted from the environment.

[0023] The term "or its processed version" in this specification means the original audio signal that has been subjected to a processing algorithm that applies gain or attenuation to the original audio signal, resulting in a modified audio signal (preferably enhanced in some sense, such as by reducing noise relative to the target signal). However, it also covers "features or parameters extracted" from the original audio signal (e.g., gain or quality parameters, etc.).

[0024] The hearing device may be configured to be worn by a user. The hearing device (or a portion thereof) may be configured to be located at or in the user's ear. The hearing device may include several separate parts, such as one part adapted to be located at or in the ear, and another part adapted to be located elsewhere on the user's body, the two parts being configured to communicate with each other via a wired or wireless communication link.

[0025] The sound capturing device may be, for example, a "wireless microphone" or a device with audio transmission capability that includes a microphone. The sound capturing device may also be, for example, the contralateral hearing device (e.g., a hearing aid) of a binaural hearing system (e.g., a binaural hearing aid system).

[0026] The term "mixing" in this specification means that a portion of the acoustic signal is completely replaced by the predicted signal, or a combination of both, such as a linear or nonlinear combination, for example, a weighted mixing in the time domain, such as a mixing that varies over time. Furthermore, mixing can occur in the frequency domain, such that a portion (e.g., all) of the frequency bands of the acoustic signal is mixed with a portion (e.g., all) of the frequency bands of the predicted signal. The number of frequency bands used in mixing can vary over time, for example, as a function of a quality estimate of the predicted signal (e.g., weighted according to an estimated quality metric, such that time-frequency units with relatively high quality metrics (e.g., SNR) have higher weights than time-frequency units with relatively low quality metrics).

[0027] The processor may include a delay estimator configured to estimate the time difference between arrival times of sound from a given sound source in the environment and the following signals:

[0028] - The acoustically received electrical input signal or its processed version; and

[0029] - Wireless reception of electrical input signals.

[0030] The time difference of arrival (T) can be fed into the signal predictor to determine the prediction time period. The time difference of arrival (T) can be determined, for example, by correlating the acoustically received signal with the wirelessly received signal. Alternatively, the delay estimator can be based on ultra-wideband technology.

[0031] Alternatively, the signal predictor can be located within the sound capture device. In this case, the hearing device should be configured to predict the time difference of arrival (TDOA, see, for example) Figure 4The signal T) in the sound capture device is transmitted, specifically, a) the TDOA between the acoustically received electrical input signal or its processed version and b) the wirelessly received electrical input signal.

[0032] Hearing devices may include a wireless transmitter for transmitting data to another device. The wireless transmitter may be configured to transmit audio signals (e.g., an acoustically received electrical input signal or a processed version thereof (e.g., a low-pass filtered version, and / or, for example, a transformed signal, in which case the "signal" may be transmitted according to a transformation factor)), or information or control signals, such as the time difference (T) of sound arriving at the processor from a given sound source in the environment. The processor, or a portion thereof, may be located in a different device than the portion located at the ear (earpiece). Hearing devices may, for example, include an earpiece adapted for location at the ear and a processing device adapted for wear or carrying by a user (or otherwise accessible by the earpiece), see, for example, see [link to relevant documentation]. Figure 6A , 6B 6C.

[0033] The processor may include a selection controller configured to include the estimated predicted signal or a portion thereof in the processed signal based on a sound quality metric. The predicted signal may be included in the signal presented to the user in the time region where the predicted signal satisfies a sound quality criterion (e.g., a sound quality metric greater than a threshold). For example, if the signal-to-noise ratio (SNR) is sufficiently high, the predicted signal may simply replace the (noisy) acoustically received electrical input signal picked up at the microphone of the hearing device, such as in a time or time-frequency region where the SNR of the predicted signal is higher than the SNR of the acoustically received electrical input signal (estimated by the hearing device's onboard SNR estimation algorithm). The acoustically received electrical input signal, the wirelessly received electrical input signal, and the predicted signal may be time-domain signals (represented by a corresponding digital audio sample stream).

[0034] The hearing device may include a transformation unit or a corresponding transformation unit for providing at least an acoustically received electrical input signal or a processed version thereof and / or a wirelessly received electrical input signal in the transformation domain. The transformation domain may be, for example, the frequency domain, such as the short-time Fourier transform (STFT) domain. Other transformation domains may be the Laplace domain, cosine transform domain, wavelet transform domain, etc. The wirelessly received electrical input signal may already be in the transformation domain, as provided by a wireless receiver.

[0035] Other nonlinear mappings are foreseeable, which, strictly speaking, are not transformations. For example, a signal can be nonlinearly mapped to some "inner domain" via a neural network, where prediction / signal substitution can occur. Subsequently, the resulting time-domain signal can be reconstructed by applying approximately the opposite mapping from the inner domain to the time domain. The neural network performing these nonlinear mappings can be an autoencoder network.

[0036] The transformation unit can be configured to provide the signal in the frequency domain. The transformation unit may be, for example, an analysis filter bank or a Fourier transform algorithm such as the Discrete Fourier Transform (DFT), Short-Time Fourier Transform (STFT), or a similar algorithm. Thus, signal processing can be performed in a sub-band or in a time-frequency representation (m,q), where m and q are the time and frequency exponents, respectively. The sub-band or time-frequency representation (m,q) may, for example, cover at least a portion of the normal human hearing frequency range (from 20 Hz to 20 kHz).

[0037] The signal predictor can be configured to estimate the future value of a wirelessly received electrical input signal in a transformed domain, such as the frequency domain (or time-frequency domain), based on the past value of the signal.

[0038] The processor can be configured to include estimated future values ​​of the wirelessly received electrical input signal in the processed signal only within a limited portion of the operating frequency range of the hearing device. The operating frequency range of the hearing device is a portion of the normal human hearing range (20 Hz to 20 kHz), for example, up to 12 kHz, up to 10 kHz, or up to 8 kHz. This limited portion of the frequency range may, for example, include a low-frequency component, such as frequencies below 4 kHz, below 2 kHz, or below 1 kHz.

[0039] A finite portion of the operating frequency range of a hearing device can be predetermined, for example, determined before the use of the device, such as by adapting to the user's hearing condition (e.g., audiogram). The finite portion of the operating frequency range can be adaptively determined (over time), for example, based on predicted signal sound quality parameters or criteria (possibly compared to the sound quality parameters of at least one acoustically received electrical input signal). Sound quality criteria can be based on SNR estimates or similar parameters estimating sound quality.

[0040] The processor, if selected by the controller, can be configured to allow some time-frequency units, such as STFT units, to be replaced while others remain unreplaced. Furthermore, the replaced STFT units do not need to be all connected (they must be directly adjacent to each other).

[0041] The processed signal may include future values ​​of the wirelessly received electrical input signal only in the frequency band or time-frequency region that satisfy the sound quality criteria. For example, the predicted signal (z(n)) can be decomposed and replaced at the channel (e.g., a low-frequency channel) because the quality of the predicted signal is generally known to be better than that of the (noisy) hearing aid microphone signal. The replacement of the acoustically received signal portion with the predicted portion can be done in the time-frequency domain based on an estimate of the SNR within the time-frequency window:

[0042]

[0043] Where s(m,q) and e(m,q) refer to the time-frequency representations (e.g., short-time Fourier transform) of signals s(n) and e(n), where s(n) is the electrical input signal received wirelessly, e(n) is the estimation error during prediction (when the predicted signal z(n) is written as z(n) = s(n) + e(n), T' is the delay of the wirelessly received signal relative to the acoustically received signal (in time unit m'), K'-1 is the number of past time units (m') on which the prediction is based, and m(m') and q are the time index and frequency index, respectively.

[0044] More generally, the processor (e.g., a selection controller) can be configured to receive the acoustic signal (Y,Y) BF The combination of the predicted signal (Z) and the predicted signal (Z) (e.g., a linear combination, see weight α) yields the resulting signal. (For example, see) Figure 3 , 4 ):

[0045]

[0046] Where 0 ≤ α ≤ 1, α can be a function of the predicted signal quality estimation parameters, Z(m,q) is the TF cell of the predicted signal, and Y(m,q) (or Y... BF (m,q), see also Figure 3 ( ) represents the TF unit of the acoustically received signal. Predicted signal quality estimation parameters (e.g., SNR) can vary with time and frequency. In this case, or independently of it, the weighting parameter α can vary with time and frequency.

[0047] The hearing device may include a beamformer configured to provide a beamforming signal based on at least one acoustically received electrical input signal and the predicted signal. In several different beamforming schemes, the predicted signal z(n) (or Z(m,q)) may be combined with one or more microphone signals from the hearing device to produce a final noise-reduced signal. In this case, the signal z(n) is simply considered as another microphone signal with a noisy realization of the target signal.

[0048] Hearing aids can be configured to apply spatial cues to a predicted signal before it is presented to the user. These spatial cues can be based on the location of an external device (wireless transmitter) or the location of the speaker. The hearing aid may include or access a database of acoustic transfer functions from multiple locations / directions around the user. When the location of the wireless transmitter or speaker is known, an appropriate acoustic transfer function can be selected and applied to the predicted signal. This makes it easier for hearing aid users to locate the received sound.

[0049] The hearing device can be configured to activate the signal predictor only when the time difference of arrival is greater than a minimum. Prediction should not be applied when T = TDOA is small (e.g., negative), such as when T > 0, T > 5 ms, or T > 10 ms (where T = T1 – T2, T1 and T2 are the times when the wirelessly received signal and the (“corresponding”) acoustically received signal arrive at the hearing device). Even without prediction, a suitable substitution of the signal or a portion thereof may still be applied.

[0050] Hearing devices including earpieces and external audio processing devices

[0051] The present invention further provides a hearing device or system, such as a hearing aid. The hearing device or system includes:

[0052] -Configured to be worn on or in the user's ear; and

[0053] - A separate audio processing unit.

[0054] At least one earpiece includes:

[0055] - Input converter for converting ambient sounds of the hearing device into an electrical input signal representing the acoustic reception of said sounds;

[0056] - A wireless transmitter for transmitting the acoustically received electrical input signal or a portion thereof to the audio processing device;

[0057] - A wireless receiver, configured to receive a first-processed signal from the audio processing device, at least in the normal operating mode of the hearing device; and

[0058] - Output converter, used to convert the final processed signal into a stimulus that is perceived by the user as sound.

[0059] The audio processing device includes:

[0060] - A wireless receiver for receiving an acoustically received electrical input signal or a portion thereof from the earpiece and providing a received signal representing that signal;

[0061] - A computing device for processing the received signal or a signal derived therefrom and providing a first processed signal;

[0062] - A transmitter for transmitting the first processed signal to the earpiece.

[0063] The earpiece or audio processing device may include a signal predictor for estimating future values ​​of the received signal or a processed version thereof (or estimating future values ​​of the gain to be applied to the signal) based on a plurality of past values ​​of the received signal, thereby providing a predicted signal.

[0064] The signal predictor can be configured to fully or partially compensate for processing delays caused by one or more of the following, for example, all of them:

[0065] - The transmission of acoustically received electrical input signals from the hearing device to the audio processing device;

[0066] - Processing in audio processing devices; and

[0067] - The transmission of the predicted signal or its processed version to the earpiece and its reception in the earpiece.

[0068] At least in the normal operating mode of the hearing device, the final processed signal may consist of or include at least a portion of the predicted signal.

[0069] The signal predictor may include a prediction algorithm (either operating in the time domain or in a transformed domain such as the time-frequency domain) configured to predict the future value of the input signal based on knowledge of past values ​​of the input signal and a processing delay (T) between a first future value and a last past value of the input signal. The processing delay (T) may include delay caused by the wireless link between the earpiece and a separate processing device. The processing delay (T) may also include processing delay within the audio processing device. The processing delay (T) of the wireless link... link The processing latency (T) of the wireless link depends on the technology (communication protocol) used to establish the link, which can be Bluetooth, Bluetooth Low Energy, Ultra Wideband, Zigbee, or any other standardized or proprietary (short-range) communication technology. link The processing delay (T) of the audio processing device can be measured or is known or estimated from the communication protocol. apd The processing delay (T) depends on the processing modules in the audio path from the receiver to the transmitter through the audio processing device. apd It is known or measurable, or estimated based on basic data of the processing device (sampling frequency, processing algorithm, etc.). The earpiece includes a forward path (e.g., a signal processing unit (see...)). Figure 6B , 6C In the case of HAPep), and in the case where the signal processed in the forward path will be mixed with the predicted signal received from the audio processing device (see, for example, HAPep), Figure 6B , 6C The processing delay (T) used as the input to the signal predictor (PRED) in the audio processing device may be less than the delay of the signal processing unit of the earpiece.

[0070] The computing device may consist of or include a signal processor (such as an audio signal processor). The computing device may be configured to perform computations using neural networks or similar learning algorithms.

[0071] The first processed signal transmitted from the audio processing device (via a wireless link) to the earpiece does not necessarily have to be an "audio signal." It can also be a feature derived from an audio signal. Instead of transmitting the audio signal back to the earpiece, parameters derived from the audio signal, such as gain derived from a predicted signal, can be transmitted to the earpiece.

[0072] The term "or its processed version" may, for example, cover features extracted from the original audio signal. The term "or its processed version" may also, for example, cover the original audio signal that has been subjected to a processing algorithm that applies gain or attenuation to the original audio signal and results in a modified audio signal (preferably enhanced in some sense, such as by reducing noise relative to the target signal).

[0073] The signal predictor may be located in an audio processing device (such as a telephone or a dedicated processing device, such as a remote control) (which may be configured to have more processing power than the earpiece). However, the signal predictor may be located in the earpiece, for example, using a first processed signal from the audio processing device as the input to the signal predictor (because the total round-trip delay T in the two devices may be assumed to be known, for example, stored in the memory of the two devices, or available from the two devices).

[0074] The audio processing apparatus may include the signal predictor. The first processed signal may include the predicted signal or a processed version thereof.

[0075] The earpiece may include an earpiece computing device configured to process the acoustically received electrical input signal and / or the first processed signal received from an audio processing device and provide the final processed signal. The earpiece computing device may, for example, include a digital signal processor such as an audio processor, and / or may be configured to perform calculations of neural networks or other learning algorithms.

[0076] The hearing device may include a transformation unit configured to convert the received signal into a signal in the transform domain. The transform domain may be, for example, the frequency domain, such as the short-time Fourier transform (STFT) domain. Other transform domains may be the Laplace domain, cosine transform domain, wavelet transform domain, etc. The hearing device may also include an inverse transform domain unit configured to convert the transform domain signal into a time domain signal. The hearing device may include an inverse transform domain unit, such as an inverse Fourier transform algorithm or a synthesis filter bank.

[0077] The hearing device can be configured to operate the signal predictor in the time-frequency domain. However, the hearing device can also be configured to operate the signal predictor in the time domain, for example, by placing an inverse transform domain module upstream of the signal predictor (if the audio signal is processed in the transform domain before the signal predictor).

[0078] The earpiece computing device can be configured, at least in the normal operating mode of the hearing device, to mix an acoustically received electrical input signal or a modified signal with a predicted signal received from an audio processing device and provide the mixed signal as a first processed signal to an output converter.

[0079] In ear-device operation mode (where the first processed signal is not received from the audio processing device or is received at a lower quality), the ear-device computing device is configured to provide the final processed signal to the output converter based on the acoustically received input signal. "Ear-device operation mode" is assumed to be the default operating mode for the hearing device in cases where a link cannot be established between the ear and the audio processing device, or only a poorly connected link can be established. Therefore, even in the absence of the audio processing device (or in the event of a wireless link failure), some minimal processing of the acoustically received electrical input signal (e.g., basic hearing loss compensation and / or noise reduction) can be provided.

[0080] A separate audio processing device can be configured to be worn or carried by the user. Alternatively, the separate audio processing device can be configured to be located on a table or other support (e.g., in a drawer). The separate audio processing device can, for example, be located in another room, provided that the wireless link between the earpiece and the separate audio processing device has sufficient transmission capability to enable the exchange of data (including audio data) therebetween at a sufficient quality.

[0081] Another hearing device includes an earpiece and an external audio processing unit.

[0082] On the other hand, the present invention provides a hearing device or system. The hearing device or system includes:

[0083] - At least one earpiece configured to be worn on or in a user's ear, to receive acoustic signals, and to present the final processed signal to the user; and

[0084] - A separate audio processing device that communicates with at least one earpiece;

[0085] The earpiece is configured to transmit the acoustic signal to an audio processing device;

[0086] The audio processing apparatus includes a signal predictor for estimating future values ​​of the acoustic signal or a processed version thereof based on multiple past values ​​of the acoustic signal received by at least one earpiece, thereby providing a predicted signal; and

[0087] The signal predictor is configured to compensate for or reduce delays caused by processing being performed in an external processing device.

[0088] An audio processing device may be configured to transmit a predicted signal or a processed version thereof to an earpiece, wherein the earpiece is configured to determine the final processed signal based on the predicted signal.

[0089] The signal predictor can be configured to fully or partially compensate for processing delays caused by one or more of the following: a) the transmission of an acoustically received electrical input signal from the hearing device to the audio processing device; b) the processing of the predicted signal in the audio processing device; and c) the transmission of the predicted signal or its processed version to the earpiece and its reception in the earpiece.

[0090] Other hearing aid features

[0091] The following features may also be combined with a hearing device, which includes the earpiece and external audio processing device described above, described in detail in the specific embodiments, or defined in the claims.

[0092] Hearing devices may consist of or include air-conduction hearing aids, bone-conduction hearing aids, cochlear implant hearing aids, or combinations thereof. Hearing devices such as hearing aids may include hearing instruments, such as hearing instruments adapted to be located in the ear or wholly or partially in the user's ear canal, such as headphones, headsets, ear protection devices, or combinations thereof.

[0093] Hearing devices may be adapted to provide frequency-varying gain and / or level-varying compression and / or frequency shifting (with or without frequency compression) from one or more frequency ranges to one or more other frequency ranges to compensate for a user's hearing loss. Hearing devices may include a signal processor for amplifying the input signal and providing a processed output signal.

[0094] Hearing devices may include an output unit for providing stimulation, perceived by the user as an acoustic signal, based on processed electrical signals. The output unit may include an output transducer. The output transducer may include a receiver (speaker) for providing the stimulation as an acoustic signal to the user (e.g., in acoustic (air conduction-based) hearing aids or headphones). The output transducer may include a vibrator for providing the stimulation as mechanical vibrations of the skull to the user (e.g., in bone-attached or bone-anchored hearing devices). The output transducer may include multiple electrodes of a cochlear implant (for CI-type hearing aids) or a vibrator of a bone conduction hearing aid.

[0095] The hearing device may include an input unit for providing an electrical input signal representing sound. The input unit may include an input transducer, such as a microphone, for converting the input sound into an electrical input signal. The input unit may include a wireless receiver for receiving wireless signals that include or represent sound and providing an electrical input signal representing said sound. The wireless receiver may, for example, be configured to receive electromagnetic signals in the radio frequency range (3 kHz to 300 GHz). The wireless receiver may, for example, be configured to receive electromagnetic signals in the optical frequency range (e.g., infrared light 300 GHz to 430 THz or visible light such as 430 THz to 770 THz).

[0096] Hearing devices may include directional microphone systems adapted to spatially filter sound from the environment, thereby enhancing a target sound source among multiple sound sources in the local environment of the user wearing the hearing device. The directional system may be adapted to detect (e.g., adaptive detection) the direction from which a specific portion of the microphone signal originates. This can be achieved, for example, in a variety of different ways described in the prior art. In hearing devices, microphone array beamformers are commonly used to spatially attenuate background noise sources. Many beamformer variations can be found in the literature. Minimum variance distortionless response (MVDR) beamformers are widely used in microphone array signal processing. Ideally, an MVDR beamformer keeps the signal from the target direction (also known as the line of sight) unchanged while attenuating sound signals from other directions to the greatest extent possible. A generalized sidelobe canceller (GSC) structure is an equivalent representation of an MVDR beamformer, offering computational and digital representation advantages over a direct implementation of the original form.

[0097] Hearing devices may include antennas and transceiver circuitry that enables the establishment of wireless links to entertainment devices (e.g., televisions), communication devices (e.g., telephones), wireless microphones, or other hearing devices such as hearing aids. The hearing device can thus be configured to wirelessly receive direct electrical input signals from another device. Similarly, the hearing device can be configured to wirelessly transmit direct electrical input signals to another device. The direct electrical input signals may represent or include audio signals and / or control signals and / or information signals.

[0098] Generally, the wireless link established by the antenna and transceiver circuitry of a hearing aid can be of any type. The wireless link can be a near-field communication-based link, such as an inductive link based on inductive coupling between the antenna coils of the transmitter and receiver sections. The wireless link can also be based on far-field electromagnetic radiation. Preferably, the frequency used to establish the communication link between the hearing aid and another device is below 70 GHz, for example, in the range from 50 MHz to 70 GHz, or above 300 MHz, for example, in the ISM range above 300 MHz, or in the 900 MHz range, or in the 2.4 GHz range, or in the 5.8 GHz range, or in the 60 GHz range (ISM = Industrial, Scientific and Medical, such standardized ranges are defined, for example, by the International Telecommunication Union ITU). The wireless link can be based on standardized or proprietary technologies. The wireless link can be based on Bluetooth technology (such as Bluetooth Low Energy) or Ultra Wideband (UWB) technology. From UWB, spatial or directional information about the location of an external device relative to the hearing aid can be provided. Such information, such as time-of-arrival delay, can be used for prediction.

[0099] Hearing devices can be portable (i.e., configured to be wearable) devices or part of them, such as devices that include local power sources such as batteries, for example, rechargeable batteries. Hearing devices such as hearing aids or headphones can be lightweight, easy-to-wear devices, for example, having a total weight of less than 100g, such as less than 20g.

[0100] A hearing aid may include a "forward" (or "signal") path for processing audio signals between the input and output units of the hearing aid. A signal processor may be located in this forward path. The signal processor may be adapted to provide frequency-varying gain (e.g., in hearing instruments) according to the specific needs of the user. The hearing aid may include an "analysis" path having functionalities for analyzing signals and / or controlling the processing of the forward path. Some or all of the signal processing in the analysis path and / or the forward path may be performed in the frequency domain, in which case the hearing aid includes appropriate analysis and synthesis filter banks. Some or all of the signal processing in the analysis path and / or the forward path may be performed in the time domain.

[0101] Analog electrical signals representing sound signals can be converted into digital audio signals during analog-to-digital (AD) conversion, where the analog signal is sampled at a predetermined sampling frequency or sampling rate f. s Perform sampling, f s For example, in the range from 8kHz to 48kHz (to suit specific application needs) at discrete time points t n (or n) provides digital samples x n (or x[n]), each audio sample passes through a predetermined N b Bit represents the acoustic signal at t n The value of N at time bFor example, in a range from 1 to 48 bits, such as 24 bits. Each audio sample therefore uses N. b Bit quantization (resulting in 2^n voltammetry of audio samples) Nb (Number of different possible values). The numerical sample x has 1 / f s The duration of time, such as 50 μs, for f s =20kHz. Multiple audio samples can be arranged in time frames. A time frame can include 64 or 128 audio data samples. Other frame lengths can be used depending on the application.

[0102] Hearing devices may include analog-to-digital (AD) converters to digitize analog inputs (e.g., from an input converter such as a microphone) at a predetermined sampling rate, such as 20 kHz. Hearing devices may also include digital-to-analog (DA) converters to convert digital signals into analog output signals, for example, for presentation to a user via an output converter.

[0103] Hearing devices, such as input units and / or antenna and transceiver circuitry, may include a transformation unit for converting time-domain signals into signals in a transform domain (e.g., frequency domain or Laplace domain). The transformation unit may constitute or include a time-frequency (TF) conversion unit for providing a time-frequency representation of the input signal. The time-frequency representation may include an array or mapping of corresponding complex or real values ​​of the signal in question over a specific time and frequency range. The TF conversion unit may include an (analytical) filter bank for filtering the (time-varying) input signal and providing multiple (time-varying) output signals, each output signal comprising a distinctly different frequency range of the input signal. The TF conversion unit may include a Fourier transform unit (e.g., including a Discrete Fourier Transform (DFT) algorithm, a Short-Time Fourier Transform (STFT) algorithm, or a similar algorithm) for converting the time-varying input signal into a (time-)frequency signal in the (time-frequency) domain. The hearing device considers a minimum frequency f... min up to the maximum frequency f max The frequency range can include a portion of the typical human hearing range from 20Hz to 20kHz, such as a portion of the range from 20Hz to 12kHz. Typically, the sampling rate f... s Greater than or equal to the maximum frequency f max twice that, i.e., f s ≥2f max The signals from the forward and / or analytical pathways of the hearing aid can be divided into NI (e.g., uniformly wide) frequency bands, where NI is, for example, greater than 5, greater than 10, greater than 50, greater than 100, or greater than 500, and at least some of these bands are processed individually. The hearing aid can be adapted to process the signals from the forward and / or analytical pathways (NP≤NI) on NP different channels. The channels can have consistent or inconsistent widths (e.g., width increases with frequency), and can overlap or not overlap.

[0104] The hearing device can be configured to operate in different modes, such as a normal mode and one or more specific modes, which may be user-selectable or automatically selected. Operating modes can be optimized for specific acoustic conditions or environments. Operating modes may include low-power modes, in which the functions of the hearing device are reduced (e.g., for energy saving), such as disabling wireless communication and / or disabling specific features of the hearing device.

[0105] The hearing device may include multiple detectors configured to provide status signals relating to the hearing device's current network environment (such as the current acoustic environment), and / or the current state of the user wearing the hearing device, and / or the current state or operating mode of the hearing device. Alternatively or additionally, one or more detectors may form part of an external device that communicates with the hearing device (e.g., wirelessly). The external device may include, for example, another hearing device (e.g., a hearing aid), a remote control, an audio transmission device, a telephone (e.g., a smartphone), external sensors, etc.

[0106] One or more of a plurality of detectors can operate on a full-band signal (time domain). One or more of a plurality of detectors can operate on a band-split signal ((time-)frequency domain), for example, in a finite number of frequency bands.

[0107] Multiple detectors may include level detectors for estimating the current level of the signal in the forward path. Detectors may be configured to determine whether the current level of the signal in the forward path is higher or lower than a given (L-) threshold. Level detectors operate on full-band signals (time domain). Level detectors operate on band-split signals ((time-)frequency domain).

[0108] The hearing device may include a voice activity detector (VAD) for estimating whether (or with what probability) an input signal (at a specific point in time) includes a voice signal. In this specification, a voice signal may include speech signals from humans. It may also include other forms of vocalization produced by the human speech system (such as singing). The voice activity detector unit may be adapted to classify the user's current acoustic environment as a "voice" or "no-voice" environment. This has the advantage that time periods or (time-frequency) components of the electro-acoustic signal that include human vocalizations (such as speech) in the user's environment can be identified and thus separated from time periods that only (or primarily) include other sound sources (such as natural or artificial noise). The voice activity detector may be adapted to also detect the user's own voice as "voice." Alternatively, the voice activity detector may be adapted to exclude the user's own voice from the detection of "voice."

[0109] The hearing device may include a self-voice detector for estimating whether (or with what probability) a particular input sound (such as speech) originates from the system user's voice. The microphone system of the hearing device may be adapted to distinguish between the user's own voice and the voice of another person, and possibly from non-voice sounds.

[0110] Multiple detectors may include motion detectors, such as accelerometers. Motion detectors may be configured to detect movements of a user's facial muscles and / or bones, such as those caused by speech or chewing (e.g., jaw movements), and provide detector signals indicating those movements.

[0111] The hearing device may include a classification unit configured to classify the current situation based on input signals from (at least partially) a detector and possibly other inputs. In this specification, "current situation" may be defined by one or more of the following:

[0112] a) Physical environment (including the current electromagnetic environment, such as the presence of electromagnetic signals (including audio and / or control signals) that are planned or unplanned to be received by the hearing device, or other properties of the current environment that are different from acoustics);

[0113] b) Current acoustic conditions (input level, feedback, etc.);

[0114] c) The user's current mode or state (movement, temperature, cognitive load, etc.);

[0115] d) The current mode or state of the hearing device and / or another device communicating with the hearing device (selected program, time elapsed since the last user interaction, etc.).

[0116] The classification unit may be based on or include neural networks, such as trained neural networks.

[0117] Hearing devices may include acoustic (and / or mechanical) feedback control (such as suppression) or echo cancellation systems. Adaptive feedback cancellation is capable of tracking changes in the feedback path over time. It is typically based on a linear time-invariant filter that estimates the feedback path, but the filter weights are updated over time. The filter update can be computed using stochastic gradient algorithms, including some form of least mean square (LMS) or normalized LMS (NLMS) algorithm. Both have the property of minimizing the error signal in terms of mean square, with NLMS additionally normalizing the filter update with respect to the square of the Euclidean norm of a reference signal.

[0118] Hearing devices may also include other suitable functions for the applications involved, such as compression and noise reduction.

[0119] Hearing devices such as hearing aids may include hearing instruments adapted to be located at the user's ear or wholly or partially in the ear canal, such as headphones, headsets, ear protection devices, or combinations thereof. Hearing systems may include loudspeaker amplifiers (containing multiple input converters and multiple output converters, for example, for use in audio conferencing scenarios), and may include beamforming filter units, for example, providing multiple beamforming capabilities.

[0120] application

[0121] On the one hand, applications of the hearing device described in detail in the "Detailed Description" section and defined in the claims are provided. Applications can be provided in systems including one or more hearing devices (such as hearing aids), headphones, headsets, active ear protection systems, etc., such as hands-free telephone systems, teleconferencing systems (e.g., including loudspeakers), broadcasting systems, karaoke systems, classroom amplification systems, etc.

[0122] Hearing system

[0123] On the other hand, hearing devices such as hearing aids and hearing systems including assistive devices are provided, including those described above, described in detail in the "Detailed Description" section, and defined in the claims.

[0124] Hearing systems can be adapted to establish a communication link between hearing devices and assistive devices so that information (such as control and status signals, possibly audio signals) can be exchanged or forwarded from one device to another.

[0125] Assistive devices may include remote controls, smartphones, or other portable or wearable electronic devices such as smartwatches. Assistive devices may be or may include dedicated processing devices (e.g., worn by the user during normal use of the hearing system). The dedicated processing device may communicate with the hearing aid (e.g., the earpiece) and be configured to perform at least a portion of the processing of the hearing aid system, such as "high-power" portions and / or processing-intensive portions, such as portions related to learning algorithms like neural networks.

[0126] The auxiliary device may consist of or include a remote control for controlling the functions and operation of the hearing device. The remote control functions are implemented in a smartphone, which may run an app that enables control of the audio processing device via the smartphone (the hearing device includes a suitable wireless interface to the smartphone, such as Bluetooth or some other standardized or proprietary solution).

[0127] The auxiliary device may be constituted by or include an audio gateway device, which is adapted to receive multiple audio signals (e.g., from an entertainment device such as a TV or music player, from a telephone device such as a mobile phone, or from a computer such as a PC) and to select and / or combine appropriate signals (or combinations of signals) from the received audio signals for transmission to the hearing device.

[0128] The assistive device may constitute or include another hearing device such as a hearing aid. The hearing system may include two hearing devices suitable for implementing a binaural hearing system such as a binaural hearing aid system.

[0129] APP

[0130] On the other hand, the present invention also provides a non-transitory application called an APP. The APP includes executable instructions configured to run on an assistive device to implement a user interface for the hearing device, such as a hearing aid or hearing system, described above, in detail in the "Detailed Description," and as defined in the claims. The APP can be configured to run on a mobile phone, such as a smartphone, or another portable device enabled to communicate with said hearing device or hearing system.

[0131] The invention can be applied, for example, to hearing devices that communicate wirelessly with other devices in the immediate environment of the user wearing the hearing device. Attached Figure Description

[0132] Various aspects of the invention will be best understood from the following detailed description taken in conjunction with the accompanying drawings. For clarity, these drawings are schematic and simplified, showing only the details necessary for understanding the invention while omitting other details. Throughout the specification, the same reference numerals are used for the same or corresponding parts. Features of each aspect may be combined with any or all features of other aspects. These and other aspects, features, and / or technical effects will be apparent from and illustrated in the following figures, wherein:

[0133] Figure 1A The diagram illustrates a hearing aid system including left and right hearing aids receiving a wireless speech signal s(nT) and an acoustic signal s(n), where it is assumed that the wireless signal arrives at the hearing aid later than the acoustic signal (T>0).

[0134] Figure 1B An exemplary amplitude-time waveform of a wirelessly received signal representing speech is shown;

[0135] Figure 1C schematically shown Figure 1B The time-frequency representation of the waveform;

[0136] Figure 1D schematically shown Figure 1B The time-domain representation of the waveform;

[0137] Figure 2A The diagram schematically illustrates time-varying analog signals (amplitude-time) and their digitization in samples arranged across multiple time frames, each time frame comprising N... s One sample;

[0138] Figure 2B schematically shown Figure 2A The time-frequency representation of the time-varying electrical signal is related to the prediction algorithm according to the present invention;

[0139] Figure 3 An embodiment of a hearing system according to the invention, including a hearing device communicating with a sound capture device, is shown;

[0140] Figure 4 An embodiment of a hearing aid according to the present invention, comprising a signal predictor and a corresponding signal quality estimator, is shown.

[0141] Figure 5 An embodiment of a hearing aid according to the present invention, including a signal predictor and a beamformer, is shown;

[0142] Figure 6A An embodiment of a hearing aid is shown, comprising an earpiece and a (e.g., body-worn) processing device that communicate with each other, wherein the body-worn processing device includes a signal predictor according to the invention;

[0143] Figure 6B As shown Figure 6A The embodiment of the hearing aid shown includes an earpiece and (e.g., body-worn) processing device, but the earpiece also includes a processing unit that enables signals from a microphone and / or from an audio processing device to be processed, such as providing a predicted signal in the earpiece;

[0144] Figure 6C As shown Figure 6A The embodiment of the hearing aid shown includes an earpiece and (e.g., a body-worn) processing device, wherein the earpiece further includes a processing unit that enables signals from a microphone and / or from an audio processing device to be processed, wherein a signal predictor is located in the audio processing device (e.g., Figure 6A (Same as in the middle) but works in the time domain.

[0145] The further applicability of the invention will become apparent from the detailed description given below. However, it should be understood that while the detailed description and specific examples illustrate preferred embodiments of the invention, they are given for illustrative purposes only. Other embodiments of the invention will become apparent to those skilled in the art based on the following detailed description. Detailed Implementation

[0146] The detailed description below, taken in conjunction with the accompanying drawings, serves as a description of various different configurations. This detailed description includes specific details to provide a thorough understanding of several different concepts. However, it will be apparent to those skilled in the art that these concepts can be implemented without these specific details. Several aspects of the apparatus and method are described by various different blocks, functional units, modules, elements, circuits, steps, processes, algorithms, etc. (collectively, “elements”). Depending on the specific application, design constraints, or other reasons, these elements may be implemented using electronic hardware, computer programs, or any combination thereof.

[0147] Electronic hardware may include microelectromechanical systems (MEMS), (e.g., application-specific integrated circuits), microprocessors, microcontrollers, digital signal processors (DSPs), field-programmable gate arrays (FPGAs), programmable logic devices (PLDs), gating logic, discrete hardware circuits, printed circuit boards (PCBs) (e.g., flexible PCBs), and other suitable hardware configured to perform the various functions described in this specification, such as sensors for sensing and / or recording the physical properties of the environment, equipment, users, etc. Computer programs should be interpreted broadly as instructions, instruction sets, code, code segments, program code, programs, subroutines, software modules, applications, software applications, software packages, routines, subroutines, objects, executables, threads of execution, programs, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description languages, or other names.

[0148] This application relates to the field of hearing systems such as hearing aids or headphones. More particularly, it relates to the acoustic and electromagnetic transmission of sound received by a hearing system wearer from the surrounding acoustic environment. The arrival timing of the two representations has a significant impact (especially when their propagation / processing times differ). A large time difference (e.g., more than 10-20 ms) between the arrival of the same sound "content" from both representations at the user's ear leads to confusion and interference, rather than improving the wearer's perception (e.g., speech intelligibility).

[0149] Consider a scenario where a user is in a very noisy environment, a target speaker is speaking, but the user has a low SNR because he or she is located at a distance from the target speaker and cannot understand the speaker. However, a wireless microphone is positioned on a table, very close to the target speaker, and can pick up a substantially noise-free version of the target speech signal. Unfortunately, the substantially noise-free signal picked up by the wireless microphone (e.g., delayed by a certain amount of time) T relative to the direct sound before reaching the user (and thus the user's hearing system). D (e.g., 30ms) cannot be presented to the user for the reasons mentioned above.

[0150] However, the hearing system can use the received (essentially pure but delayed) signal to predict future T.D The prediction of a pure signal at a time (e.g., 30ms) will obviously be imperfect, but parts of the predicted signal (especially the low-frequency part) will provide a basis for future signals. D A good representation of the actual pure signal at (e.g., 30ms). This predicted portion can be usefully presented to the user, either directly or in combination with the microphone signal from the hearing system.

[0151] Furthermore, the gain, which varies with time and frequency, can be extracted from the predicted signal and applied to the hearing aid microphone signal. This gain can, for example, depend on the level of the signal of interest, such that only the high-energy time-frequency region of the external signal is preserved (the low-energy (or low SNR) region is attenuated).

[0152] More specifically, consider Figure 1A , 1B In the scenario shown in 1C, the voice signal of the target speaker TT is transmitted through a wireless microphone M. ex Pick it up. Figure 1A The diagram illustrates a hearing aid system comprising left and right hearing aids (HD1, HD2) receiving a wireless speech signal s(nT) and an acoustic signal s(n), wherein it is assumed that the wireless signal arrives at the hearing aids a time interval T (e.g., ms) later than the acoustic signal (i.e., T>0). Figure 1B An exemplary amplitude-time waveform of a wirelessly received (of fairly high quality) voice signal is shown.

[0153] It should be noted that this is not always the case. Sound takes approximately 3 ms to travel 1 meter. For example, if the sound source is 5 meters away, the transmission delay through the air is 15 ms, so the wirelessly transmitted sound can actually arrive before the sound is picked up by the microphone. Therefore, knowing TDOA is advantageous, and prediction is only applied when T>0 or T>5 ms or T>10 ms (where T = T1–T2, where T1 and T2 are the times when the wirelessly received signal and the (“corresponding”) acoustically received signal arrive at the hearing device, see below).

[0154] In wireless microphone M ex In this process, the speech signal is encoded and transmitted to the hearing aid user U, where it is received after T1ms (e.g., in one or both hearing aids (HD1, HD2), or in a separate processing device communicating with the hearing aids). Simultaneously, the acoustic speech signal ATS emitted from the target speaker TT is received at the microphone of the hearing aid after T2ms. Therefore, the wirelessly transmitted signal WTS is delayed by T = T1 - T2 relative to the acoustic signal ATS received at the hearing aid user U. There may be a difference in the arrival times of the acoustic signal between the two hearing aids (HD1, HD2) (equal to the interaural time difference ITD) (theoretically, there should also be a difference in the arrival times of the wirelessly transmitted signal).

[0155] In practice, the time difference of arrival (TDOA) T can be estimated by measuring the similarity between the signals, for example, by correlating the acoustic signal of a given hearing aid with the wirelessly received signal. Alternatively, the time difference of arrival T can be estimated by other means, such as via ultra-wideband (UWB) technology.

[0156] Including left and right hearing instruments (see, for example) Figure 1A In a binaural hearing aid configuration (HD1, HD2), TDOA can preferably be jointly estimated, for example, estimated as the minimum TDOA for each hearing aid. Alternatively, TDOA can be estimated individually for each hearing aid.

[0157] If T is too large, for example, greater than 10 ms, the wirelessly received signal cannot be used to present it to the user in real time. In this case, we recommend using signal samples s(nT-K+1),…,s(nT) available in the hearing aid system to predict future samples s(n) (relative to the signals available in the hearing aid system (HD1,HD2)), where K-1 is the number of past samples of the wirelessly received signal used for prediction (s(nT) is the previous wirelessly received sample, such that K is the number of samples used for prediction). Prediction can be performed in the time domain or in other domains, such as the (time-)frequency domain.

[0158] Figure 1C schematically shown Figure 1B The waveform shown is represented in the time-frequency domain. Figure 1B The time samples are transformed to the frequency domain as multiple (Q) (e.g., complex) values ​​of the time-domain signal s(n), provided, for example, through Fourier transform algorithms (such as Short-Time Fourier Transform, STFT, etc.). Figure 1C In this context, the same time exponent n is used for both time samples and time frames. This is done for simplicity of illustration. In practice, multiple samples, such as 64, are included in a single time frame; on the other hand, samples can overlap. Within each frequency band, or as... Figure 1C As shown, in lower frequency bands, for example below the threshold frequency f th For the q-th frequency band FB q (i.e., for frequencies below the threshold frequency f) th The frequency bands FB1,...FB qth In the future, "samples" q (n) Based on K-1 past samples (or frames) s q (nT-K+1),…,s q (nT) is used for prediction. The predicted value is in Figure 1C The bright spots in the time-frequency window are indicated by shading at time exponent n. The predicted values ​​are based on values ​​in... Figure 1CThe black dot shading marks the time-frequency window at time exponents nT-(K-1),….,nT. The prediction is limited to the low-frequency band below the threshold (FB1,…FB). qth In the case of f, which is higher than the threshold frequency f th frequency band (FB) qth+1 ,…,FB Q It can be represented by one of the noisy microphone signals or by a beamforming signal that is a combination of two or more microphone signals (as indicated by the cross-shaded time-frequency window at time exponent n).

[0159] Different uses of the concept of this invention can be foreseen, such as the following:

[0160] 1) The wireless microphone is an external wireless microphone, such as a microphone unit clipped to the target speaker's clothing, a desktop microphone, or a microphone in a smartphone (see, for example, [link to relevant documentation]). Figure 1A M in ex );

[0161] 2) The “wireless microphone” is actually the hearing device on the opposite side (HDx, x = 1 or 2 depending on the situation), and the target signal is an external sound source: the sound signal is picked up by the left hearing device (e.g., HD2) and sent to the right hearing device (e.g., HD1) which uses signal prediction to reduce latency, so that a portion of the microphone signal from the left hearing device (HD2) can be replaced by the right (HD1) microphone signal and / or combined with the right microphone signal (see below for a more detailed description);

[0162] 3) "Wireless microphone" is not a microphone. Consider the scenario where computation is output to an external (processing) device such as a smartphone: the sound signal is picked up by the microphone of the hearing device user, sent from the hearing device to the external device for computation (possibly using signal prediction to reduce latency), and sent from the external device back to the hearing device (possibly using signal prediction to reduce latency).

[0163] Predict future samples s(n) based on past samples s(nT-K+1),…,s(nT), such as Figure 1C , 1D As shown in the image. Figure 1C schematically shown Figure 1B The time-frequency representation of the waveform, Figure 1D schematically shown Figure 1B The time-domain representation of the waveform. For example... Figure 1D As shown, the predicted signal z(n) is the K past samples (from t) of the wirelessly received signal s. now = nT backwards) s(nT-K+1),…,s(nT) is a function “f” of time t. now=nT is the time it takes for the wireless signal to reach the hearing device and indicates the delay T relative to the arrival time of the (corresponding) acoustically propagating signal y. Therefore, in t now =nT, the hearing device can be used at t=t now The samples and s in t now -1,t now -2 or any number of past sample values.

[0164] Predicting future samples s(n) based on past samples s(nT-K+1),…,s(nT) is a well-known problem with existing solutions. For example, the prediction of future samples s(n) can be based on linear prediction, see, for example, [1], where the estimator z(n) of s(n) is formed as a linear combination of past samples, i.e.:

[0165]

[0166] Among them, a k ,k=0,…,P-1 are time-varying coefficients derived from past samples s(nT-K+1),…,s(nT)[1], where P refers to the order of the linear predictor.

[0167] There are many other methods for predicting s(n). For example, the estimator z(n) of s(n) can be computed using nonlinear methods such as deep neural networks (DNNs), i.e.:

[0168] z(n)=G(s(nT-K+1),…,s(nT); Θ,T) (2)

[0169] Here, Θ refers to the parameter set of the DNN, and G(.,Θ,T) represents the network. In this case, the network G(.,Θ,T) should be trained offline using a training set of clean speech signals before deployment to predict signal samples separated into T samples, see, for example, Chapter 5 in [2].

[0170] More generally, the estimator z(n) of s(n) can be computed using a DNN of this form:

[0171] z(n)=G(s(nT-K+1),…,s(nT),x(n); Θ) (3)

[0172] Here, x(n) represents the microphone signal captured at the hearing aid (i.e., the noisy version the network attempts to predict). In this case, we eliminate the network's dependence on T, as it can be estimated internally within the DNN by comparing the wirelessly received samples s(nT-K+1),…,s(nT) with the local microphone signal x(n), for example, by correlating the two sequences. This configuration, which has access to an updated but potentially very noisy signal x(n), is particularly well-suited for predicting transient / speech initiation events, which would otherwise be challenging.

[0173] In another generalized version of the predictor

[0174] z(n)=G(s(nT-K+1),…,s(nT),x1(n),…,x M (n); Θ) (4)

[0175] The estimator z(n) is the (outdated but possibly relatively noise-free) received wireless signal and multiple local microphone signals x1(n),…,x M The latter configuration has a special case where z(n) (partially) is calculated as a function of the beamforming signal y(n) calculated at the hearing aid using the local microphone signal:

[0176] y(n)=H(x1(n),…,x M (n)) (5)

[0177] Where H(.) represents beamforming operation. Other prediction methods exist.

[0178] Clearly, prediction is not limited to the time-domain signal s(n) as described above. For example, (linear) prediction can also occur in the time-frequency domain or in other domains (such as cosine, wavelet, Laplace, etc.).

[0179] Generally, the predicted signal can be written as:

[0180] z(n)=s(n)+e(n) (6)

[0181] Where e(n) is the estimation error. If e(n) is considered as a noise term, the prediction process can be viewed as a “trade” between a delayed (outdated) essentially noiseless signal s(nT) and a newer but usually noisy signal z(n) = s(n) + e(n). The more accurate the prediction, the smaller the noise (prediction error). The signal-to-noise ratio (SNR) ξ(n) in the predicted signal can be estimated from available past samples, for example, as follows:

[0182]

[0183] This involves summing across available past samples. Alternatively, the SNR can be calculated offline as the long-term average SNR expected for a specific T value.

[0184] SNR can also be estimated in the time-frequency domain (m,q):

[0185]

[0186] Where s(m,q) and e(m,q) refer to the time-frequency representations of signals s(n) and e(n) (e.g., short-time Fourier transform), T′ and K′ are the time-frequency analogies of T and K, m is the time index (e.g., time frame index), and q is the frequency index (e.g., frequency band index).

[0187] The predicted signal z(n) can be used in hearing devices in several ways.

[0188] For example, if the SNRξ(n) is high enough, the noisy signal x(n) picked up at the microphone of the hearing device can be simply replaced with the predicted signal z(n), for example in the signal region where the SNRξ(n) in the predicted signal is higher than the SNR in the microphone signal x(n) (as estimated by the SNR estimation algorithm on the hearing device).

[0189] As an alternative, instead of replacing the signal sample z(n), substitution can be performed in the frequency band. For example, z(n) can be decomposed into channels (e.g., low frequencies) and z(n) substituted in the channels, since the quality of the predicted signal is known to be generally better than that of the hearing aid microphone signal. More generally, substitution can even be performed in the time-frequency domain based on the SNR estimate in the time-frequency window, see equation (8) above.

[0190] Alternatively, in several different beamforming schemes, the signal z(n) can be combined with one or more microphone signals from the hearing device to produce a final, noise-reduced signal. In this case, the signal z(n) is simply considered as yet another microphone signal with a noisy realization of the target signal.

[0191] The above description assumes the predictor is part of the receiver (i.e., the hearing system, such as a hearing aid). However, prediction may also occur in a wireless microphone, assuming it has processing capabilities and can be notified of the time difference of arrival T. In other words, the predicted signal z(n) is formed in the wireless microphone and transmitted to the hearing system (such as the hearing aid), possibly along with auxiliary information such as estimated... Together.

[0192] The above description assumes that the wireless microphone is a single microphone that captures a substantially noise-free signal s(n). However, it can also consist of a microphone array (i.e., one or more microphones). In this case, a beamforming system can be implemented in the wireless device, with the output of the beamformer acting as the substantially noise-free signal s(n). Furthermore, the external (sound capture) device can, for example, consist of or include a tabletop microphone capable of extracting at least one noise-free signal.

[0193] Figure 2A The diagram schematically illustrates a time-varying analog signal (amplitude-time) and its digitized form x(n) in samples arranged across multiple time frames, each time frame comprising N. s One sample. Figure 2A The diagram illustrates an analog electrical signal (solid line curve), for example, representing an acoustic input signal from a microphone, which is converted into a digital audio signal x(n) during analog-to-digital conversion. During the analog-to-digital conversion, the analog signal is sampled at a predetermined rate f. s Perform sampling, f s For example, in the range from 8kHz to 48kHz (adapting to specific application needs), digital samples x(n) are provided at discrete time points n, as shown by a vertical line extending from the time axis with solid dots at its endpoints "coinciding" with the curve, representing the digital sample values ​​at corresponding different time points n. Each (audio) sample x(n) represents the value obtained through a predetermined number (N) b The (quantization) bits represent the value of the acoustic signal at time n, N b For example, in the range of 1 to 48 bits, such as 24 bits. Each audio sample therefore uses N. b Bit quantization (resulting in 2^n audio samples) Nb (a number of different possible values).

[0194] In the Modulus-Digital (AD) process, the digit sample x(n) has 1 / f s The length of time, for example, for f s =20kHz, the duration is 50μs. Multiple (audio) samples N s For example, it can be arranged in a time frame, such as Figure 2A The lower schematic diagram shows that the samples (evenly spaced here) are grouped by time frame x(m) (including each sample element #1,2,…,N). s ), where m is the number of frames. Similarly, as... Figure 2A As shown in the lower diagram, time frames can be arranged continuously in a non-overlapping manner (time frames 1, 2, ..., m, ..., N). M ), where m is the time frame index. Alternatively, time frames can overlap (e.g., 50% or more, e.g. Figure 2A(See below). In this embodiment, a time frame comprises 64 audio data samples. Other frame lengths may be used depending on the application. A time frame may, for example, have a duration of 3.2 ms (e.g., corresponding to 64 samples at a sampling rate of 20 kHz).

[0195] Figure 2B schematically shown Figure 2A The time-varying electrical signal x(n) is represented by a time-frequency diagram (or sub-band) related to the prediction algorithm according to the present invention. This time-frequency representation X m (q) (q = 1, ..., Q, where q is the frequency exponent) comprises an array or mapping of the corresponding complex or real values ​​of the signal over a specific time and frequency range. This time-frequency representation can, for example, be the result of a Fourier transform of a time-varying input signal x(n) into a time-frequency domain (time-varying) signal X(m,q). In embodiments, the Fourier transform includes a Discrete Fourier Transform (DFT) algorithm, such as a Short-Time Fourier Transform (STFT) algorithm. Typical hearing devices consider frequencies starting from the minimum frequency f. min up to the maximum frequency f max The frequency range includes a portion of the typical human hearing range from 20Hz to 20kHz, such as a portion of the range from 20Hz to 12kHz. Figure 2B In this context, the time-frequency representation X(m,q) of a signal x(n) includes the complex values ​​of the signal's magnitude and / or phase within multiple DFT windows (or watts) determined by the exponent (m,q), where q = 1, ..., Q represents Q frequency values ​​(see [reference]). Figure 2B The vertical q-axis in the diagram, and m = 1, ..., N M N represents M Time frames (see Figure 2B (Horizontal m-axis in the diagram). The time frame is determined by a specific time index m and the corresponding Q DFT windows (see [reference]). Figure 2B The indication of time frame m in the time frame (or X). m () represents the spectrum of signal x at time m. It includes the DFT window or W(m,q) of the (real or) complex values ​​of the signal involved in X(m,q). Figure 2B The shaded plot of the corresponding field in the time-frequency graph (see...) Figure 2B In the DFT window = time-frequency unit (m,q): Where |X| represents the magnitude, and (This represents the phase of the signal within that time-frequency unit). Each value of the frequency index q corresponds to a frequency range Δf. q ,like Figure 2B The frequency is indicated by the vertical frequency axis f. Each value of the time index m represents a time frame. The continuous time index spans a time T. F Depending on the length of the time frame and the degree of overlap between adjacent time frames (see...) Figure 2B(Horizontal time axis in the text).

[0196] The time frame of an electrical signal may, for example, include N, a digitized electrical signal representing sound. s A series of consecutive samples, for example, 64 (written as a vector x). m ), where m is the time exponent, for example, see Figure 2A However, as an alternative, the time frame of an electrical signal can be defined as including the amplitude spectrum of the electrical signal at a given time point (written as vector X). m (For example, provided via Fourier transform algorithms such as the STFT (Short Time Fourier Transform) algorithm, see, for example, [link to relevant documentation]) Figure 2B (A schematic illustration of the TF graph in the image). A time frame x representing multiple time samples. m and the time frame X representing the amplitude spectrum of the electrical signal (with samples taken at the same time). m They are linked together through Fourier transform, for example, by the expression Given, among which Let be the matrix representing the Fourier transform.

[0197] The electrical input signal representing sound can be provided as multiple sub-band signals. Sub-band signals can be provided, for example, by analyzing a filter bank, such as based on multiple bandpass filters, or based on the Fourier transform algorithm mentioned above (e.g., by continuously extracting the corresponding amplitude spectrum from the Fourier transformed data).

[0198] like Figure 2B As specified in the document, the prediction algorithm according to the invention can provide sub-band level signals (instead of the full-band (time domain) signals described above, see, for example, see...) Figure 1D This provides a downsampling (e.g., a factor of 20 or greater) for the update rate of the corresponding (sub-band) prediction algorithm. Figure 2B The coarse "trapezoidal" polygon (from the time "now" (exponent m, see [reference]) surrounding the (wirelessly received) input signal, representing multiple historical time-frequency units (DFT windows). Figure 1C , 1D The time "nT" in the middle and the time backward K q (Each time unit) indicates a portion of the known input data, which, for a given frequency band q, is used to predict the (wirelessly received) signal s. WLR For the prediction of the future value z after time T (exponentially m+T), see the thick rectangle with dot fill in time unit m+T. The prediction algorithm can be performed in all frequency bands q = 1, ..., Q, and, for example, can use the same number (K) q The algorithm uses historical values ​​(or different values ​​for different frequency bands) to predict future values. However, the prediction algorithm can be performed only in selected frequency bands, such as those most important for speech intelligibility, or below the threshold frequency (see, for example, [link to relevant documentation]). Figure 1C f in th) frequency band, or such Figure 2B The schematic diagram indicates that the frequency f is higher than the low-frequency threshold. th,low And below the high-frequency threshold frequency f th,high The frequency band. High-frequency threshold frequency f th,high For example, it could be 4kHz (which is generally difficult to predict at higher frequencies), 3kHz, 2kHz, or even lower, such as 1kHz. This is partly because speech origins above the high-frequency threshold are primarily caused by the rushing airflow from the mouth and throat regions, and are inherently less predictable than speech at frequencies below the low-frequency threshold (which are mainly produced by vocal cord vibration). Low-frequency threshold f th,low For example, it can be greater than or equal to 100Hz (human hearing is typically below 100Hz), or greater than or equal to 200Hz, or greater than or equal to 500Hz. The parameter K indicates the number of past values ​​of time-frequency units used to predict future frequencies. q They can differ; for example, they decrease as the frequency increases (e.g.) Figure 2B As shown in the diagram, for example, it simulates a time period in which the fundamental frequency increases as the frequency decreases. Similarly, a weighting factor a is applied to each previous value (time-frequency unit) of a given sub-band signal. i It can vary with frequency, a i =a i (q)=a i,q Even predicting time T (e.g., by parameter K) q Different values ​​cause this, and it can vary with frequency (T = T(q) = T). q Various prediction algorithms can be executed according to the present invention, as discussed above for full-band signals. Instead of... Figure 2B The effect shown is applied to a uniform frequency band (bandwidth Δf) q Independent of the frequency exponent q), the prediction algorithm can be applied to non-uniform frequency bands, such as those with increasing bandwidth as the frequency increases (reflecting the logarithmic characteristics of the human auditory system).

[0199] Figure 3 It shows a device including a sound capture device (M) ex An embodiment of a hearing system according to the present invention, comprising a hearing device HD with -Tx) communication. The hearing device HD, such as a hearing aid, includes an input unit IU, which includes corresponding input converters (IT1, ..., IT1). M M (M≥2) input units (IU1,…,IU) of a microphone M ), used to input sound from the environment of the hearing device ("sound input", y1(t),...,y M (t) is converted into the corresponding multiple acoustic receiver electrical input signals (y'1(n),…,y'). M(n)) thereby representing the sound as a data sample stream. Input units (such as input converters) may include suitable analog-to-digital converters to provide the acoustically received electrical input signal as digital samples. Input units (IU1,…,IU) M It also includes a corresponding analysis filter bank AFB, used to provide the acoustically received electrical input signal in time-frequency representation (m,q) as the signal Y = (Y1(m,q),…,Y…). M (m,q)). The hearing device HD (in this case, the input unit IU) also includes an auxiliary input unit IU. aux It includes a sound capture device M for picking up (target) sound S from the environment. ex The wireless transmitter Tx receives audio signals (“audio input”, s(t)) and provides a digital sample stream (s) representing the sound. WLR (n)) wirelessly received electrical input signal s WLR (n) wireless receiver Rx. Auxiliary input unit IU aux It may include a suitable analog-to-digital converter to provide the wirelessly received electrical input signal as a digital sample (S). WLR (n)). Input units (IU1,…,IU) M It also includes a corresponding analysis filter bank AFB, used to convert the wirelessly received electrical input signal (s) into a time-frequency representation (m,q). WLR (n) is provided as signal S WLR (m,q). The hearing aid also includes a beamformer BF, configured to receive electrical input signals (Y1(m,q),…,Y) based on multiple acoustic receivers. M (m,q) provides the beamforming signal Y. BF (m,q). The hearing aid also includes a processor PRO, configured to receive beamforming signals Y. BF (m,q) and the wirelessly received electrical input signal S WLR (m,q). The processor PRO includes a signal predictor PRED, configured to estimate the wirelessly received electrical input signal s based on multiple past samples (or time-frequency units) of the signal. WLR (n)(or S) WLR The signal predictor PRED can be configured to run the estimation algorithm outlined above (and in the prior art, see, for example, EP3681175A1). The signal predictor PRED can also be configured to estimate the electrical input signal received acoustically (e.g., y'). i (n), i = 1, ..., M, or beamforming signal Y BF (m,q)) and the wirelessly received electrical input signal (s) WLR (n') or S WLRThe arrival time difference between (m,q) is shown in the example below. Figure 4 The delay estimator DEST in [the context of the original text] estimates, for example, by finding the time delay that optimizes the correlation function between two signals. (See also: Time difference of arrival) Figure 4 The time difference (T) can be fed into the prediction algorithm to determine its prediction time. Figure 4 The T in the time-frequency domain can be based on the time-frequency domain signal (Y). BF (m,q) and S WLR (m,q) is determined, see the dashed input Y of the signal predictor PRED. BF (m,q). The hearing device also includes a selection controller SEL-MIX-CTR, configured to adjust the processed signal based on control signals such as sound quality metrics, such as SNR estimates. This includes signals predicted based on wirelessly received electrical input signals (see, for example, [link to relevant documentation]). Figure 4 Processed signal It may include the predicted signal Z(m,q) or may be composed of the predicted signal. The processed signal. It can be the beamforming signal Y received acoustically. BF The mixture of (m,q) and the wirelessly received electrical input signal Z(m,q). Figure 3 The hearing device embodiment also includes an output unit OU, which includes a synthesis filter bank FBS and a method for processing signals from a processor. Or, a further processed version thereof presents the user with an output transducer that is perceived as an auditory stimulus (see, for example, see...). Figure 4 The output unit may include a digital-to-analog converter, depending on the situation. The output converter may include a speaker in an air-conduction hearing device. The output converter may include a vibrator in a bone-conduction hearing device. The output converter may include a multi-electrode array in a cochlear implant hearing device. In the latter case, the synthesis filter bank may be omitted.

[0200] Figure 4An embodiment of a hearing aid according to the present invention is illustrated. The hearing aid includes an input converter (here, a microphone M) for converting sound in the hearing aid's environment ("acoustic input y(t)", where t represents time) into an acoustically received electrical input signal (y(n) = s(n) + v(n)), thereby representing the sound as a digital sample stream, where n is the time exponent. The input converter assumes that the electrical input signal is provided as a digital sample stream y(n), for example, by using a MEMS microphone or by including an analog-to-digital converter (where appropriate). The acoustic input y(t) (based on which the electrical input signal v(n)) may include a target sound s and noise v from the environment (e.g., from the user, or from the hearing aid (e.g., acoustic feedback)). The hearing aid also includes a wireless receiver Rx for receiving audio signals from a wireless transmitter of a sound capture device that picks up sound from the environment. The wireless receiver Rx may include, for example, an antenna and corresponding electronic circuitry for receiving and extracting the payload (audio) signal and providing the wirelessly received electrical input signal s. WLR (n'), thus representing the sound as a digital sample stream, where n' is the time exponent. The hearing aid includes a corresponding analysis filter bank (or Fourier transform algorithm) for representing Y(m,q) and S in sub-bands or in time-frequency representations, respectively. WLR (m',q) provides each digitized electrical input signal y(n) and s WLR (n'), where m and m' are time exponents and q is the frequency exponent. The hearing aid also includes a signal predictor PRED, configured to predict the frequency based on the wirelessly received electrical input signal S. WLR Multiple past samples S of (m,q) WLR (m',q) estimates its future samples (e.g., estimated as the value of time-frequency unit (m,q) in one or more frequency bands q'), thereby providing a predicted signal Z(m,q). The hearing aid also includes a delay estimator DEST, configured to estimate the sound from a given sound source in the environment at the acoustically received electrical input signal y(n) and the wirelessly received electrical input signal s. WLRThe arrival time difference T between (n') and the signal (e.g., the input of a delay estimator) is the time difference T between the acoustically received signal and the signal (e.g., the input of a delay estimator). The arrival time difference T is provided as the output of the delay estimator is fed to a signal predictor PRED to determine the prediction time period of the signal predictor. The arrival time difference T can be determined, for example, by correlating the acoustically received signal with the wirelessly received signal. The hearing aid also includes corresponding SNR estimators (SNRestA, SNRestP) configured to provide estimates of the signal-to-noise ratio (SNR) of the acoustically received electrical input signal Y(m,q) and the predicted signal Z(m,q), respectively. SNR estimation can generally be provided in various ways, such as involving a speech activity detector and, for example, estimating the noise level during speech pauses.<N(m0,q)> The noise estimate is provided as Y(m,q) / <N(m0,q)> Where m0 is the time exponent when the noise was last estimated (the previous speech pause). More complex SNR estimation schemes or other signal quality estimators are also available, see, for example, US20190378531A1. SNR estimators (SNRestA, SNRestP) applied to the acoustically received signal and the predicted signal, respectively, can be based on different principles. The SNR estimator SNRestP for the predicted signal (Z) can be based on the wirelessly received electrical input signal (S... WLR The previous value of ) (see the dashed input of the SNR estimator SNRestP) and the prediction error signal e are estimated (Z = S + e), as outlined above in combination with equations (7) and (8) (for the corresponding time-domain and time-frequency domain implementations). The hearing aid also includes a selection controller SEL-MIX-CTR, configured to estimate the sound quality metric, such as the SNR estimate SNR based on the acoustically received electrical input signal Y(m,q) and the wirelessly received electrical input signal Z(m,q). Y (m,q) and SNR Z (m,q) includes estimated future samples of the wirelessly received electrical input signal in the processed signal. These estimated future samples can be included in the predicted signal if it satisfies a sound quality criterion (e.g., a sound quality metric, in this case, the SNR estimate SNR). Z (m,q), greater than the first threshold SNR TH1 (q) or a time region greater than the SNR estimate of the acoustically received signal Y(m,q). For example, if the SNR estimate is high enough, such as greater than a second threshold SNR. TH2(q) The selection controller can be configured to replace the (noisy) acoustically received electrical input signal y(n) (or the signal picked up at the microphone of the hearing aid) with a predicted signal z(n), for example, in a time region where the estimated SNR in the predicted signal is higher than the estimated SNR in the microphone signal y(n) (estimated by an onboard SNR estimation algorithm in the hearing aid), where n is the time (sample) exponent. In the time-frequency domain, such a scheme can be adjusted equally at the sub-band level (q) or the time-frequency unit level (i.e., individually for each TF unit (m,q)). The selection controller SEL-MIX-CTR thus receives signals Y(m,q) and Z(m,q) as audio inputs and adjusts them according to the control signal SNR. Y (m,q) and SNR Z (m,q) will enhance the audio signal. Provided as output. The hearing aid also includes a signal processing unit (HAG) configured to provide frequency-varying gain and / or level-varying compression, for example, to compensate for the user's hearing loss. The determined hearing aid gain can thus be applied to the (enhanced) audio signal. And provide (user-adapted) processed signal S out (m,q). The hearing aid also includes a synthesis filter bank (FBS) for synthesizing the time-frequency domain signal S. out (m,q) is converted to a time-domain signal s out (n). The hearing aid also includes an output converter OT, used to adjust the output based on the processed signal s from the processor. out (n) or a further processed version thereof will be presented to the user as an output stimulus that can be perceived as sound. The output transducer may include, for example, a speaker, a vibrator, or an implanted electrode array. Figure 4 Some functional elements of a hearing aid may be included in a (e.g., digital signal) processor. The processor is configured to receive: a) at least an acoustically received electrical input signal or a processed version thereof, b) a wirelessly received electrical input signal, and to provide a processed signal accordingly. The (digital signal) processor may include... Figure 4 The following functional modules are included in the embodiment: a) Analysis Filter Bank (FBA), b) Delay Estimator (DEST), c) Signal Predictor (PRED), d) SNR Estimator (SNRest), e) Selection Controller (SEL-MIX-CTR), f) Signal Processing Unit (HAG), and g) Synthesis Filter Bank (FBS). Other functional modules, such as those related to feedback control or additional analysis and control modules, such as those related to self-voice estimation / voice control, may also be included in the (digital signal) processor.

[0201] Figure 5 An embodiment of a hearing aid according to the present invention, comprising a signal predictor and a beamformer, is shown. Figure 5 The embodiments are similar to Figure 4 The embodiments, except for the unlabeled mixed SNR estimator (SNRestP, SNRestP) that controls the electrical input signal (Y) and the predicted signal (Z) of the acoustic reception. Additionally, instead of Figure 4 Select the controller SEL-MIX-CTR, in Figure 5 The embodiment includes a beamformer filter (BF). Alternatively, in other words, the selection controller SEL-MIX-CTR can be incorporated into the beamformer filter (BF) to provide (enhanced) audio signals. In this embodiment, in several different beamforming schemes, the predicted signal (z(n) or Z(m,q)) is combined with one or more microphone signals from the hearing aid to produce the final noise-reduced signal (only one microphone signal Y(m,q) is shown here, but in other embodiments, there may be multiple (M) electrical input signals, see, for example, [link to relevant documentation]). Figure 3 In this case, the predicted signal is simply regarded as another microphone signal with a noisy realization of the target signal.

[0202] Figure 6A An embodiment of a hearing device HD is shown, comprising an earpiece EP and a wearable audio processing device APD that communicate with each other. The wearable processing device APD may include a computing device CPD containing a signal predictor PRED according to the invention. apd (e.g., an audio signal processor or similar device). Hearing devices such as hearing aids include at least one earpiece (EP) configured to be worn on or in the user's ear and a separate audio processing device (APD) configured to be worn or carried by the user (or at least located close enough to the user to maintain communication with the earpiece via a wireless link implemented by the transceiver of the respective device).

[0203] At least one earpiece EP includes an input converter (here, a microphone M) for converting ambient sound of the hearing device into an acoustically received electrical input signal y(n) representing that sound. The earpiece also includes a wireless transmitter Tx for transmitting the acoustically received electrical input signal y(n) or a portion thereof (e.g., a filtered portion, such as a low-pass filtered portion) to an audio processing device APD. The earpiece EP also includes a wireless receiver for receiving a predicted signal from the audio processing device, at least in the normal operating mode of the hearing device. The wireless transmitter and receiver may be provided as antenna and transceiver circuitry for establishing an audio communication link WL according to a standardized or proprietary (short-range) protocol. The earpiece EP also includes an output converter (here, a speaker SPK) for transmitting the (final) processed signal s' out (n) is converted into a stimulus that is perceived as sound by the user. The processed signal s' out(n) At least in the normal operating mode of the hearing device, it may consist of or include at least a portion of the predicted signal (provided by the audio processing device or by the earpiece, such as...). Figure 6B (As shown below).

[0204] An audio processing device (APD) includes a wireless receiver Rx for receiving an acoustically received electrical input signal y(n) or a portion thereof from an earpiece EP and configured to provide a received signal y(n') representing the signal or a portion thereof. The audio processing device (APD) is similar to a computing device (CPD). apd The device also includes a processor section (HAP) for applying processing algorithms (e.g., including neural networks) to the received signal y(n') or a signal derived therefrom, such as its transformed version Y, and providing a modified signal Y'. The processor section (HAP) may be configured, for example, to address a user's hearing impairment (e.g., by applying compression amplification algorithms, such as providing a gain (attenuation) that varies with frequency and / or level to be applied to the input signal (y(n') or Y)). Audio processing device (APD) (e.g., computing device (CPD)). apd It also includes a signal predictor PRED, which estimates the future value of the modified signal (y', Y') based on multiple past values ​​of the signal to provide a predicted signal (z, Z). The signal predictor PRED may include a prediction algorithm (either operating in the time domain or in a transformed domain such as the time-frequency domain), configured to be based on past values ​​of the input signal and knowledge of the prediction time, such as the processing delay between the first future value and the last past value of the input signal (see input T) (see, for example, [link to relevant documentation]). Figure 1C , 1D The total processing delay T can be the delay caused by the wireless link between the earpiece EP and the separate processing unit APD. link Adding the processing delay T in the audio processing device apd The sum (i.e., T = T) link +T apd The processing latency T of the wireless link. link The processing delay T of the audio processing device depends on the technology (communication protocol) used to establish the link and can be known or estimated in advance (or during use). apd Depending on the processing modules along the audio path from receiver to transmitter through the audio processing device, it can also be known or estimated in advance (or during use). The audio processing device APD also includes a transmitter Tx for transmitting the predicted signal (Z) or its processed version (S). out (n) (referred to as the "first processed signal") is transmitted to the earpiece EP.

[0205] The signal predictor PRED is configured to fully or partially compensate for the transmission of the acoustically received electrical input signal y(n) from the earpiece EP to the audio processing device APD, b) the processing in the audio processing device APD (through its audio signal processing path from receiver Rx to transmitter Tx), and c) the transmission of the predicted signal z(n) or its processed version to the earpiece EP and its reception therein (signal s'). out The processing delay caused by (n) is determined by estimating the total processing delay T (T = T). link +T apd It provides the input implementation for the Prediction Algorithm (PRED).

[0206] exist Figure 6A In this embodiment, the audio processing device APD (e.g., the computing device CPD) apd It also includes corresponding transform domain units and inverse transform domain units (TRF, I-TRF) to transform the time-domain signal (here, the received signal y(n') from the earpiece) to the transform domain (e.g., the time-frequency domain), see signal Y, and then transform it back (here, the predicted signal Z in the transform domain is transformed back to the signal z(n) in the time domain). Figure 6A In this embodiment, the signal predictor PRED is implemented in the transform domain. Figure 6C In this embodiment, the signal predictor PRED is implemented in the time domain. This can be selected as a design feature depending on the specific configuration (e.g., partitioning) of the hearing device / system.

[0207] Figure 6B As shown Figure 6A The embodiment of the hearing aid HD shown includes an earpiece EP and a (e.g., body-worn) processing device APD, but the earpiece also includes signals from the microphone M (signal y(n)) and / or from the wireless receiver Rx (signal s'). out (n) A computing device (CPD) capable of processing in an earpiece ep (e.g., signal processing unit). Computing Device (CPD) ep Provides the final processed signal ( Figure 6B s in out (n)), which is fed to the output converter (in this case, the speaker SPK) to be presented to the user. Figure 6B In this embodiment, the signal predictor PRED is implemented in the time domain.

[0208] Figure 6C As shown Figure 6AThe illustrated embodiment of a hearing aid includes an earpiece and a body-worn processing device, but the signal predictor operates in the time domain (because the order of the inverse transform domain unit I-TRF and the signal predictor PRED has been reversed). Another difference is that the earpiece EP includes a computing device CPD that enables the earpiece to process the acoustically received signal y(n) and / or a first processed signal received from a separate audio processing device APD. ep Optional processing of the acoustically received signal y(n) (such as by a computing device CPD) ep (The dashed input) may be of interest, for example, in operating modes where a connection to the audio processing device APD cannot be established (e.g., to provide the user with basic functions of the hearing device, such as hearing loss compensation).

[0209] about Figure 6A , 6B In the embodiments of 6C, it should be mentioned that the signals transmitted from the earpiece EP to the (external) audio processing device APD via the wireless link WL and / or from the audio processing device APD to the earpiece EP are not necessarily "audio signals". They can also be features derived from audio signals. For example, instead of transmitting the audio signal back to the hearing device, a gain derived from a predicted signal can be transmitted back.

[0210] The term "or its processed version" may, for example, cover features extracted from the original audio signal. The term "or its processed version" may also, for example, cover the original audio signal that has been subjected to a processing algorithm that applies gain or attenuation to the original audio signal and results in a modified audio signal (preferably enhanced in some sense, such as by reducing noise relative to the target signal).

[0211] When appropriately replaced by a corresponding process, the structural features of the apparatus described above, in detail in the "Detailed Description" section, and as defined in the claims can be combined with the steps of the method of the present invention.

[0212] Unless explicitly stated otherwise, the singular forms “a” and “the” used herein include the plural forms (i.e., meaning “at least one”). It should be further understood that the terms “having,” “comprising,” and / or “including” as used in the specification indicate the presence of the stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or combinations thereof. It should be understood that, unless explicitly stated otherwise, when an element is referred to as “connected” or “coupled” to another element, it may be a direct connection or coupling to the other element, or there may be intermediate inserting elements. The term “and / or” as used herein includes any and all combinations of one or more of the listed related items. Unless explicitly stated otherwise, the steps of any method disclosed herein do not necessarily have to be performed in the exact order disclosed.

[0213] It should be understood that references to "an embodiment," "an embodiment," "an aspect," or "may" in this specification mean that a particular feature, structure, or characteristic described in connection with that embodiment is included in at least one embodiment of the invention. Furthermore, particular features, structures, or characteristics may be suitably combined in one or more embodiments of the invention. The foregoing description is provided to enable those skilled in the art to implement the various aspects described herein. Various modifications will be apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects.

[0214] The claims are not limited to the aspects shown herein, but encompass the full scope consistent with the language of the claims, wherein, unless expressly stated, an element referred to in the singular does not mean "one and only one," but rather "one or more." Unless expressly stated, the term "some" means one or more.

[0215] References

[0216] [1]Deller, Hansen and Proakis, "Discrete-Time Processing of SpeechSignals," IEEE Press, 2000.

[0217] [2]Goodfellow, Bengio and Courville, "Deep Learning," MIT Press, 2016.

[0218] ·EP3681175A1(Oticon)15.07.2020.

[0219] ·US20190378531A1(Oticon)12.12.2019.

Claims

1. A hearing device, comprising: - At least one input converter for converting sound in the environment of the hearing device into a corresponding electrical input signal representing at least one acoustically received sound; - A wireless receiver for receiving audio signals from a wireless transmitter of a sound capture device that picks up sounds in the environment and providing an electrical input signal representing the wireless reception of the sounds; - Processor, configured as -- Receive the at least one acoustically received electrical input signal or a processed version thereof; -- Receive the wirelessly received electrical input signal; and -- Provides the processed signal; The processor includes: -- A signal predictor for estimating future values ​​of the wirelessly received electrical input signal based on multiple past values ​​of the wirelessly received electrical input signal, thereby providing a predicted signal; - An output converter for presenting a perceptible sound-like output stimulus to a user based on the processed signal from the processor or a further processed version thereof; The processor is configured to provide the processed signal based on the predicted signal or a processed version thereof in the following manner: -- Alone, or -- Mixed with the electrical input signal received by the at least one acoustic receiver or its processed version.

2. The hearing device according to claim 1, wherein, The processor includes a delay estimator configured to estimate the time difference between the arrival times of a sound from a given sound source in the environment and the following signals: - The acoustically received electrical input signal or its processed version; and - The wirelessly received electrical input signal.

3. The hearing device according to claim 1, comprising a wireless transmitter for transmitting data to another device.

4. The hearing device according to claim 1, wherein, The processor includes a selection controller configured to include an estimated predicted signal or a portion thereof in the processed signal based on a sound quality metric.

5. The hearing device according to claim 1, comprising a transformation unit or a corresponding transformation unit for providing at least one acoustically received electrical input signal or a processed version thereof and / or the wirelessly received electrical input signal in the transformation domain.

6. The hearing device according to claim 5, wherein, The transformation unit is configured to provide a signal in the frequency domain.

7. The hearing device according to claim 6, wherein, The processor is configured to include estimated future values ​​of the wirelessly received electrical input signal in the processed signal only within a limited portion of the operating frequency range of the hearing device.

8. The hearing device according to claim 7, wherein, The processed signal includes future values ​​of the wirelessly received electrical input signal only in the frequency band or time-frequency region that meets the sound quality criteria.

9. The hearing device of claim 1, further comprising a beamformer configured to provide a beamforming signal based on at least one acoustically received electrical input signal and the predicted signal.

10. The hearing device of claim 1, configured to apply spatial cues to the predicted signal before presenting the predicted signal to the user.

11. The hearing device of claim 2, configured to activate the signal predictor only when the arrival time difference is greater than a minimum value.

12. A hearing device, comprising: - Configured to be worn on or in the user's ear; and - Separate audio processing unit; The at least one earpiece includes: -- Input converter for converting ambient sounds of the hearing device into an electrical input signal representing the acoustic reception of the sounds; -- A wireless transmitter for transmitting the acoustically received electrical input signal or a portion thereof to the audio processing device; -- A wireless receiver, for receiving a first-processed signal from the audio processing device, at least in the normal operating mode of the hearing device; and -- Output converter, used to convert the final processed signal into a stimulus that is perceived by the user as sound; The audio processing device includes: -- A wireless receiver for receiving an acoustically received electrical input signal or a portion thereof from the earpiece and providing a received signal representing the signal; -- A computing device for processing the received signal or a signal derived therefrom and providing a first processed signal; -- A transmitter for transmitting the first processed signal to the earpiece; Wherein, the at least one earpiece or the audio processing device includes: -- A signal predictor for estimating future values ​​of the received signal or a processed version thereof based on multiple past values ​​of the received signal, thereby providing a predicted signal; -- Wherein, the signal predictor is configured to fully or partially compensate for processing delays caused by one or more of the following: --- The transmission of acoustically received electrical input signals from the hearing device to the audio processing device; --- Processing in audio processing devices; and --- The transmission of the predicted signal or its processed version to the earpiece and its reception in the earpiece; Wherein, at least in the normal operating mode of the hearing device, the final processed signal consists of or includes at least a portion of the predicted signal.

13. The hearing device according to claim 12, wherein, The audio processing device includes the signal predictor.

14. The hearing device according to claim 12, wherein, The at least one earpiece includes an earpiece computing device configured to process the acoustically received electrical input signal and / or the first processed signal received from an audio processing device and provide the final processed signal; The earpiece computing device is configured, at least in the normal operating mode of the hearing device, to mix an acoustically received electrical input signal or a modified signal with a predicted signal received from an audio processing device and to provide the mixed signal as a first processed signal to an output converter.

15. The hearing device according to claim 14, wherein, In an earphone operating mode where the first processed signal is not received from the audio processing device or is received at a lower quality, the earphone computing device is configured to provide the final processed signal to the output converter based on the acoustically received input signal.

16. The hearing device according to claim 1 or 12, comprising a hearing instrument, headphones, headset, ear protection device or a combination thereof.

Citation Information

Patent Citations

  • A hearing device comprising direct sound compensation

    EP3681175A1

  • Audio processing device and a method for estimating a signal-to-noise-ratio of a sound signal

    US20190378531A1

  • Hearing device and hearing system configured to localize a sound source

    CN107071674A

  • Memory and computation efficient cross-correlation and delay estimation

    US10431238B1