Method, Apparatus and System for Neural Network Hearing Aids - Patent application

Abstract

The present disclosure generally relates to a method, system, and device for improving a user's speech understanding in real-time conversations by processing the speech through a neural network included in a hearing device. The hearing device may be a headphone or a hearing aid. In one embodiment, the present disclosure relates to a device for improving an incoming speech signal. The device includes a controller for receiving an incoming signal and providing a controller output signal, a neural network engine (NNE) circuit in communication with the controller, the NNE circuit being activatable by the controller and configured to generate an NNE output signal from the controller output signal, and a digital signal processing (DSP) circuit for receiving one or more of the controller output signal or the NNE circuit output signal and thereby generating a processed signal, the controller determining a processing path of the controller output signal through one of the DSP or NNE circuit as a function of one or more of predetermined parameters, incoming signal characteristics, and feedback of the NNE circuit.

Description

[Technical field]

[0001] The present disclosure generally relates to methods, apparatus, and systems for neural network-enabled hearing devices. In some embodiments, the present disclosure provides methods, systems, and apparatus for improving the understanding of a user's speech in a real-time conversation by processing the audio through a neural network included in a hearing device such as a headphone or hearing aid.

[0002] The ease of conversation between people in real-world situations is often hindered by background noise. If the background noise is loud relative to the speech, the speech will be drowned out by the background noise. Bars, restaurants, concerts, etc. are examples of environments that are generally difficult for conversation. Although normal-hearing people also struggle with particularly difficult "signal-to-noise" ratios, these environments are particularly difficult for people with hearing loss.

[0003] Hearing loss or impairment makes it difficult to hear, recognize, and understand sounds. Hearing loss can occur at any age and may be due to birth defects, aging, or other causes. The most common type of hearing loss is sensorineural hearing loss. Sensorineural hearing loss is a permanent hearing loss caused by damage to tiny hair-like cells in the inner ear called localization cilia, or to the auditory nerve itself, which prevents or weakens the transmission of nerve signals to the brain. Sensorineural hearing loss usually impairs both volume sensitivity (the ability to hear soft sounds) and frequency selectivity (the ability to distinguish clear sounds in noise). This second impairment has a particularly severe impact on the intelligibility of speech in noise. Compared to people with normal hearing, people with hearing loss experience a reduced ability to follow speech in the presence of background noise, even when the speech is well above the hearing threshold.

[0004] Conventional hearing aids provide the necessary amplification to compensate for the loss of sound sensitivity. This is helpful in quiet environments, but in noisy environments, the amplification has limited effectiveness because it becomes difficult for the hearing-impaired person to selectively hear the sounds they want to hear. Conventional hearing aids use various techniques to increase the wearer's signal-to-noise ratio, such as directional microphones, beamforming techniques, and post-filters. However, none of these methods are particularly effective because they rely on assumptions that are often incorrect, such as the location of the talker and the statistical properties of the signal in different frequency bands. As a result, people with hearing loss still have difficulty following conversations in noisy environments, even with the latest hearing aids.

[0005] Neural networks provide a means to treat sounds differently based on their semantics. Such algorithms can be used to separate speech from background noise in real time, but placing powerful algorithms such as neural networks in the signal path has previously been considered unfeasible in hearing aids and headphones. Hearing aids have limited battery life to compute such algorithms, and such algorithms have struggled to perform well in the variety of environments encountered in the real world. The disclosed embodiments address these and other deficiencies of conventional hearing aids. [Brief description of the drawings]

[0006] The disclosed embodiments will be described with reference to the following illustrative and non-limiting embodiments in which like elements are numbered likewise.

[0007] [Figure 1] FIG. 1 is a system diagram according to one embodiment of the present disclosure. [Diagram 2] FIG. 2 illustrates a schematic diagram of an exemplary front-end receiver according to one embodiment of the present disclosure. [Figure 3A] FIG. 3A is a schematic diagram of an exemplary system according to one embodiment of the present disclosure. [Figure 3B] FIG. 3B shows the speech volume, background noise level controls, and a mode switch. [Figure 4] FIG. 4 is a diagram illustrating a signal processing system according to another embodiment of the present disclosure. [Figure 5A] FIG. 5A illustrates an interaction between user preferences and nonlinear gain applied by an exemplary NNE in accordance with one embodiment of the present disclosure. [Figure 5B] FIG. 5B is an illustration of exemplary NNE circuit logic implemented in accordance with one embodiment of the present disclosure. [Figure 5C] FIG. 5C is a diagram illustrating a schematic of an exemplary architecture for engaging an NNE circuit according to one embodiment of the present disclosure. [Figure 6] FIG. 6 is a flow diagram illustrating an example activation / deactivation of an NNE circuit according to one embodiment of the disclosure. [Figure 7] FIG. 7 is a block diagram of a SOC package according to one embodiment. [Figure 8] FIG. 8 is a block diagram of an exemplary auxiliary processing system that may be used in connection with the disclosed principles. [Figure 9] FIG. 9 is a generalized diagram of a machine learning software stack in accordance with one or more embodiments. [Figure 10] FIG. 10 is a diagram illustrating training and deployment of a deep neural network in accordance with one or more embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] The following description and exemplary embodiments are described to provide a thorough understanding of various embodiments. However, various embodiments may be practiced without specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to obscure certain embodiments. Furthermore, various aspects of the embodiments may be implemented using various means, such as integrated semiconductor circuitry ("hardware"), computer-readable instructions organized into one or more programs ("software"), or some combination of hardware and software. For purposes of this disclosure, references to "logic" shall mean either hardware, software, firmware, or some combination thereof.

[0009] The disclosed embodiments generally relate to the use of neural networks to improve audio data in ear-worn systems such as hearing aids and headphones. Neural network-based audio improvement has also been deployed in other applications such as video conferencing and other telecommunication mediums. In many such applications, these algorithms are used to reduce background noise, making it easier for the user to hear the sound of interest (usually the speech of a person speaking to the user). Neural network-based audio improvement has been considered too difficult for face-to-face applications where the user is in the same location as the person or thing they are trying to hear.

[0010] One of the main reasons that face-to-face communication has been considered impractical is the complexity of the task facing the algorithms: in video communication, the tolerable latency is relatively high (>50 ms), the speaker is usually close to the microphone (resulting in a relatively high signal-to-noise ratio (SNR) in the signal received at the microphone), and the ambient noise is usually limited to that encountered during a face-to-face scenario and is much less tolerant.

[0011] Human hearing is very sensitive to delays introduced by signal processing in ear-worn devices. If the delay is too large, both the original sound and the amplified sound reproduced by the earphones arrive at the ear at different times, resulting in an echo-like perception. Delays can also interfere with the brain's processing of the incoming sound, as a disconnect occurs between visual cues (such as lip movements) and the arrival of the associated sound. Hearing aids are one of the prime examples of ear-worn devices for face-to-face communication. The optimal latency for such devices is 10 milliseconds (ms) or less, but up to 32 ms is acceptable in some circumstances.

[0012] In these face-to-face scenarios, there is high variability in the nature of the background noise, resulting in signals with much lower SNR. Social environments such as bars, restaurants, and outdoor venues often require conversations to take place in the presence of overwhelming background noise. Similarly, general-type environments have much more variety than a typical conference call. Therefore, it is more challenging to create neural networks that are robust to such conditions.

[0013] Neural networks offer a fundamentally different method of filtering sound than conventional hearing aids. The main difference is the power and flexibility in implementing hearing algorithms. In conventional digital signal processing systems, the parameters of the hearing equations must be adjusted manually. In neural networks, the optimal parameters can be discovered by training, a computational process in which the network learns to solve a task by adjusting parameters, gradually improving performance. Whereas a human can optimally tune 100 parameters, a neural network can learn millions of parameters.

[0014] Digital signal processing in conventional hearing devices typically applies a series of filters and gains (interchangeably, weights) that adjust the magnitude of the signal at different frequencies. In conventional hearing aids, these gains specifically compensate for the user's lost frequency sensitivity. These algorithms typically do not adjust the phase of the incoming signal. Neural networks have the computational power to robustly generate fine-grained adjustments to both the magnitude and phase of the incoming signal, with fine granularity in both the time and frequency domains.

[0015] The challenge with incorporating neural network algorithms is the computational cost. There is a well-established positive correlation between network size and network performance found across different domains of deep learning. To obtain the fine-grained responses necessary to robustly handle various acoustic environments, neural networks have thousands of parameters and require millions, or even billions, of operations per second. The size of a viable network is limited by the computational power of the hearing device's processor. Hearing aids must be compact and capable of long-term use to be comfortable and convenient for the wearer. Ideally, hearing aids would be integrated into one device rather than spread across multiple devices (e.g., hearing aid device and smart device).

[0016] Such neural network algorithms are also difficult to incorporate in a way that results in an optimal user experience. Even if a hearing aid can separate sounds from a single source, this behavior is not always desirable. For example, ambient sounds may be important to pedestrians. Even if speech separation is the primary goal, some ambient noise may be desirable. For example, a person in a restaurant may find hearing only speech confusing or unpleasant, and may prefer to let at least low levels of ambient sounds through to provide a sense of ambience. Thus, to achieve the desired user experience, devices must harness the power of neural networks and use their outputs intelligently.

[0017] Another problem in creating a good user experience is dealing with model errors. Even a large, well-trained neural network does not perform perfectly and, in certain environments, may not be able to distinguish one sound source from another. In such scenarios, the device needs to fail gracefully in a way that still provides a comfortable hearing experience for the user. As an example, a conversation interrupted by a loud vehicle may produce an incomprehensible white noise to the listener if the model output is played back without taking into account the model errors. Therefore, a solution is needed that monitors the model output and performance and dynamically adjusts it to produce a suitable user experience.

[0018] Hearing device as used herein generally refers to a hearing aid, active ear protection device, or other sound processing device that can be configured to improve, amplify, and / or protect the hearing ability of a user. A hearing aid can be implemented in one or two earpieces. Such devices typically receive acoustic signals from the user's surroundings, apply possible modifications to the audio signals to generate corresponding audio signals, and provide the modified audio signals as audible signals to the user. The modifications can be implemented in one or both hearing devices corresponding to each ear of the user. In some embodiments, the hearing device can include earphones (individually or as a pair), headsets, or other external devices that can be adapted to provide audible acoustic signals to the outer ear of the user. The transmitted acoustic signals may be fine-tuned via one or more controls to optimally transmit mechanical vibrations to the user's hearing system.

[0019] In one embodiment, the present disclosure relates to a hearing aid that can utilize neural network-based sound improvement in the signal processing chain. As used herein, a neural network in a signal processing chain comprises a system in which the neural network is integrated with an in-ear hearing aid device. In some embodiments, the hearing device comprises a neural network integrated with auxiliary circuitry on an integrated circuit (IC), among other things. The IC may comprise a system on chip (SoC).

[0020] In some embodiments, the exemplary device is configured to, among other things, amplify all ambient sounds, filter incoming sounds down to speech (removing background noise), filter incoming sounds down to one or more target speakers, switch between these modes in response to user input, adjust the volume of the background noise in response to user input, change what types of sounds are considered "noise," adjust the output of the hearing aid in all modes to match the user's hearing profile (including frequency sensitivity and dynamic range), and the like.

[0021] In one embodiment, the neural network is incorporated into a hearing aid. The hearing aid may be equipped with one or more processors optimized to handle the workload of the neural network. The one or more processors may be selectively engaged based on the operating mode of the device. Some embodiments of the present invention address these issues by introducing a dual-path signal chain that may selectively engage the neural network and one or more of the digital signal processors. By creating a dual signal processing path, hearing aid users may enjoy the benefits of neural network-based improvements when neural network engagement is necessary and desirable. These and other embodiments of the present disclosure are discussed in connection with the following exemplary embodiments.

[0022] 1 is a system diagram according to one embodiment of the present disclosure. System 100 may be implemented in a hearing aid. In an exemplary embodiment, system 100 is implemented in one or both earpieces of a hearing aid device. System 100 may be implemented as an integrated circuit. System 100 may be implemented as an IC or SoC.

[0023] The system 100 receives an incoming signal 110 and provides an output signal 190. The incoming signal 110 may comprise acoustic signals emanating from multiple sources. The acoustic sources emanating the acoustic signal 110 may comprise ambient noise, human voices, alarm sounds, etc. Each acoustic source may emit sound at a different volume relative to the other sources. Thus, the incoming signal 110 may be a collection of different sounds arriving at the system 100 at different volumes.

[0024] The front-end receiver 120 may comprise one or more modules configured to convert the incoming acoustic signal 110 to a digital signal using an analog-to-digital converter (ADC). The front-end receiver 120 may also receive signals from one or more microphones at one or more earphones. In an embodiment, the signal received at one earphone is transmitted to the other earphone using a low-latency protocol such as near-field magnetic induction for use in signal processing. The output of the front-end receiver 120 is a digital signal 125 representing one or more received audio streams. It should be noted that FIG. 1 illustrates an exemplary embodiment in which the front-end 120 and the controller 130 are separate components. In an embodiment, one or more functions of the front-end 120 may be performed by the controller 130 to omit the front-end 120.

[0025] In the embodiment of FIG. 1, the NNE circuit is interposed between the control unit 130 and the DSP 140. The NNE circuit 150 is therefore in the direct signal processing path, meaning that when that signal path is employed, the audio is processed and improved through a neural network before the same audio is played back. This is in contrast to methods where a neural network is employed outside of the direct signal chain to adjust parameters of the direct signal chain. These methods use the neural network output to improve the audio that is subsequently received, rather than the same audio that has been processed through the neural network. In one embodiment, the NNE circuit is configured to selectively apply a complex ratio mask to the incoming signal of the front-end receiver to obtain a plurality of components, each of the plurality of components corresponding to a voice class or individual speaker, and the NNE circuit is further configured to combine these components into an output signal, the volume of the components being set to obtain a user-controlled signal-to-noise ratio.

[0026] The control unit 130 receives the digital signal 125 from the front-end receiver unit 120. The control unit 130 may comprise one or more processor circuits (herein, processors), memory circuits, and other electronic and software components configured to, among other things, (a) perform the digital signal processing operations necessary to prepare the signal for processing by the neural network engine 150 or the DSP engine 140, and (b) determine the next step in the processing chain from among a number of options. In one embodiment of the present disclosure, the control unit 130 executes decision logic that determines whether to proceed with signal processing through one or both of the DSP unit 140 and the neural network engine (NNE) circuit 150. It should be noted that the front-end 120 comprises one or more processors that convert the incoming signal, while the control unit 130 comprises one or more processors that perform the exemplary tasks disclosed herein, and that these functions may be combined and implemented in the control unit 130.

[0027] The DSP 140 may be configured to apply a set of filters to the incoming audio components. Each filter may isolate a desired frequency range of the incoming signal and apply a non-linear, time-varying gain to each filtered signal. The gain values ​​may be set to achieve dynamic range compression or to identify stationary background noise. The DSP 140 may then recombine the filtered and gain-applied signals to provide an output signal.

[0028] As mentioned above, in one embodiment, the controller performs digital signal processing operations to prepare the signal for processing by one or both of the DSP 140 and the NNE 150. The NNE 150 and the DSP 140 may accept a time-frequency domain signal (e.g., the signal 110) as an input, and the controller 130 may perform a short-time Fourier transform (STFT) of the incoming signal before passing it to the controller. In another example, the controller 130 may perform beamforming of signals received at different microphones to improve sounds coming from a particular direction.

[0029] In an embodiment, the controller 130 continuously determines the next step in the signal chain for processing the received audio data. For example, the controller 130 activates the NNE 150 based on one or more of user-controlled criteria, user-independent criteria, user clinical criteria, accelerometer data, location information, stored data, and calculated metrics characterizing the audio environment, such as signal-to-noise ratio (SNR). If the NNE 150 is not activated, the controller 130 instead passes the signal 135 directly to the DSP 140. In some embodiments, the controller 130 may pass data to both the NNE 150 and the DSP 140 simultaneously, as indicated by the arrow 135.

[0030] The user-controlled criteria (interchangeably, logic or user-defined) may comprise a user input comprising a selection of an operating mode via an application on the user's smartphone or an input on the device (e.g., by tapping on the device). For example, if the user is in a restaurant, the operating mode can be changed to noise cancellation / speech separation by making an appropriate selection on the smartphone. The user-controlled criteria may also comprise a set of user-defined settings or preferences that the user enters through an application (app) or that the device learns over time. For example, the user-controlled logic may comprise a user preference regarding the sounds the user hears (e.g., a new parent may want a baby's cries amplified at all times and a dog owner may want a dog's barking amplified at all times), or the user's general tolerance to background noise. The user clinical criteria may comprise a clinically relevant hearing profile, such as, for example, the user's general degree of hearing loss and ability to understand speech in noise.

[0031] User-controlled logic may also be used in conjunction with or separate from user-independent criteria (or logic). User-independent logic may take into account variables that are independent of the user. For example, user-independent logic may take into account the available power level of the hearing aid (as a function of expected NNE performance demand), the time of day, or the expected duration of NNE operation.

[0032] In some embodiments, acceleration data captured by sensors in the device may assist the controller 130 in determining whether to direct the signal controller output signal 135 to one or both of the DSP 140 and the NNE 150. The motion or acceleration information may guide the controller 130 in determining whether the user is moving or sitting. The acceleration data may be used in combination with other information or may be overwritten with other data. Similarly, data from sensors capturing acceleration may be provided to a neural network as information for inference.

[0033] In other embodiments, the user's location may be used by the control unit 130 to determine whether to engage one or both of the DSP 140 and the NNE circuitry 150. Certain locations may require activation of the NNE circuitry 150. For example, if the user's location indicates high ambient noise (e.g., the user is walking in a park or attending a concert) and there is no in-person conversation, the control unit 130 may activate only the DSP 140. On the other hand, if the user's location suggests that the user is on the move (e.g., taking a car or train) and other indicators suggest human communication, the NNE circuitry 150 may be operated to amplify the human voice over the ambient noise.

[0034] The stored data may also be a factor in the control unit 130 determining the processing path. The stored data may include important characteristics of a user's unique sound, voice, preferences, or commands. The system 100 may optionally include a storage circuit 132 for storing data representing voices that, when detected, may serve as input to the logic of the control unit. The storage circuit 132 may be local, as shown, or remote from the hearing device. The stored data may include a so-called voice registry of known conversation partners. The voice registry may provide the necessary information for the neural network to detect and separate specific voices from background noise. The voice registry may include a classification embedding of each registered voice calculated by an off-device neural network (i.e., large NNE), described herein as a voice signature, and an on-device neural network (i.e., local NNE) may be configured to accept the voice signature as an input to isolate speech that matches the signature.

[0035] In addition to the voice signatures, the system 100 may store different preferences for each voice in the registry 132 so that different speakers elicit different behavior from the device. The NNE 150 can then implement various algorithms to determine which voices to amplify relative to other voices.

[0036] The controller 130 may execute algorithmic logic to select a processing path. The controller 130 may consider the detected SNR and determine whether to engage one or both of the DSP 140 and the NNE 150. In one embodiment, the controller 130 compares the detected SNR value to a threshold to determine which processing path to initiate. The threshold may be one or more of empirically determined, user-independent, or user-controlled. The controller 130 may also consider other user preferences or parameters in determining the threshold, as described above.

[0037] In another embodiment, the control unit 130 can calculate certain metrics to characterize the incoming speech as input for determining the subsequent processing path. These metrics can be calculated based on the received speech signal. For example, the control unit 130 can detect periods of silence, knowing that these periods do not require neural network refinement and therefore should only engage the DSP 140. In a more complex example, the control unit 130 can include a voice activity detector (VAD) 134 to determine the processing path of the speech separation mode. In some embodiments, the VAD can be a much smaller (i.e., much less computationally intensive) neural network in the control unit.

[0038] In an exemplary embodiment, the control unit 130 may receive as input to its calculations the output of the NNE 150 for recently processed speech, as indicated by arrow 151. The NNE 150 may be configured to isolate speech of interest in the presence of background noise, providing the necessary input to robustly estimate the SNR. The control unit 130 may utilize this capability to detect whether the SNR of the incoming signal is high or low enough to affect the processing path. In yet another example, the output of the NNE 150 may be used as the basis for a more robust VAD 134. Speech detection in the presence of noise is computationally intensive. By leveraging the output of the NNE 150, the system 100 may perform this task with minimal computational overhead.

[0039] If the control unit 130 uses the NNE output 151, the control unit 130 can only use the output 151 to affect the signal path of subsequently received audio. When a given audio sample is received at the control unit, the output of the NNE 150 for that sample has not yet been calculated and therefore cannot be used to affect the control unit's decisions for that sample. However, the audio environment less than a second ago can predict the current environment, so the NNE output of previously received audio can be used.

[0040] When the NNE 150 is activated, the control unit may use the NNE output 151 without additional computational cost. In an embodiment, the control unit 130 may activate the NNE 150 for support computations even in modes where the NNE 150 is not the selected signal path. In such modes, the incoming audio signal is passed directly from the control unit 130 to the DSP 140, but data (i.e., voice clips) are additionally passed at less frequent intervals to the NNE 150 for computation. This computation provides an estimate of the SNR of the surrounding environment or detects speech in the presence of noise in substantially real-time. In an exemplary implementation, the control unit 130 may transmit a 16 ms data window once per second for VAD 134 detection at the NNE 150. In some embodiments, the NNE 150 may be used for VAD instead of the control unit 130. In another embodiment, the control unit 130 may dynamically adjust the duration of the voice clip or the frequency with which it communicates the voice clip as a function of the estimated probability of useful computation. For example, if the SNR has fluctuated significantly over recent requests, the control unit 130 may request additional NNE calculations at more frequent intervals.

[0041] The NNE 150 may comprise one or more real and virtual circuits for receiving the controller output signal 135 and providing an improved digital signal 155. In an exemplary embodiment, the NNE 150 uses a neural network algorithm (NN model) to improve the signal by generating a set of intermediate signals. Each intermediate signal is representative of one or more original sound sources that make up the original signal. For example, the incoming signal 110 may comprise two speeches, an alarm, and other background noises. In some embodiments, the NN model running on the NNE 150 may generate a first intermediate signal representing the speech and a second intermediate signal representing the background noise. The NNE 150 may also separate one of the speakers from the other speaker. The NNE 150 may separate the alarm from the remaining background noise to ensure that the user can hear the alarm even when the noise cancellation mode is activated. Different situations may require different intermediate signals, and different embodiments of the present invention may comprise different neural networks with different capabilities that best suit the wearer's needs. In one embodiment, a remote (off-chip) NNE can augment the capabilities of a local (on-chip) NNE.

[0042] As will be described later in relation to Figures 7-10, a neural network, called an artificial neural network (ANN) or simulated neural network (SNN) in the case of artificial neurons, is an interconnected group of natural or artificial neurons that uses a mathematical or computational model for information processing based on the so-called connectionistic approach to computation. In most cases, an ANN is an adaptive system, changing its structure based on external or internal information flowing through the network. A neural network is a nonlinear statistical data modeling or decision-making tool. Such systems may be used to model complex relationships between inputs and outputs and to find patterns in the data. The utility of an artificial neural network model lies in its ability to infer functions from observations and to exploit them. This is achieved by training the model. The model takes representative data as input and iteratively changes the weights of the parameters in the network to optimize the given function. In supervised learning, the model operates on a labeled data set, while in unsupervised learning, the model operates on unlabeled data. These methods can be used in combination. An illustration of an exemplary ANN or NNE is shown in Figure 10.

[0043] According to some of the disclosed principles, a neural network (which may be implemented via a neural network engine) is trained to separate one or more sound sources. In an exemplary embodiment, this is done by supervised learning. As input data, the model receives pairs of audio clips, one of which is a target and the other is a mixture comprising both the target signal and other signals. The training data includes clips of speakers speaking in the absence of background noise as targets, and the clips can then be synthetically mixed with recordings of background noise to form a mixed clip. Through training, the model learns to generate a complex mask for each pair of clips, which, when applied to the mixed clip, returns, on average, a sound that most closely resembles the target clip as measured by a loss function (training aims to minimize the loss over the training dataset). By devising a model that works well on a variety of different clips that are representative of the task at hand, the model learns a function that can generalize to unseen audio data. When applied to data comprising the speaker's speech and background noise, the model is able to estimate a signal that comprises only speech content, or at least substantially speech content.

[0044] To generate a model suitable for in-person processing of audio, the model may be trained to generate outputs based on inputs that represent small samples of audio. The model may process the audio continuously, receiving and processing each sample (or audio clip) so that it can be played before the most recent sample has finished playing.

[0045] As an example, the model operates on 4 ms audio samples. At t=0, the pre-processor starts receiving data from the microphone. At t+4 ms, the control unit (e.g., control unit 130) receives the entire sample and passes it to the NNE 150 for processing. The NNE then calculates an estimate of the 4 ms audio sample (clip) and passes the intermediate signal to the next step in the signal chain. Once the remaining signal processing is complete, playback to the user begins. At t+8 ms, the NNE 150 receives the next 4 ms sample clip from the control unit 130. By the time the playback of the first sample is complete (4 ms after playback begins), the next 4 ms sample clip is ready to be played back so that there are no gaps. For recurrent neural networks, this means that the computation must be completed in less than a sample length, since the computation of subsequent samples depends on an updated launch from the current sample. In other model architectures, this constraint can be circumvented by parallelization (at high computational cost).

[0046] In this example, the model operates on samples of 4 ms audio clips. The sample length can be scaled up or down depending on various parameters. For example, the sample length can be less than 1 ms or 32 ms of data. The longer the sample length, the longer the model must wait to provide a response, and therefore the longer the latency experienced by the user. If the model waits for a full second of audio data, it may provide good background noise suppression, but the user may experience unbearable playback delays. In some embodiments, the model may have a look-ahead function, whereby the model waits to receive more audio before processing, thereby increasing the information available to the model. Extending the above example, the model may wait until t+8 ms (given the 4 ms look-ahead) to start processing the first 4 ms of audio, which may improve the model's performance, but introduce additional latency. In some embodiments, the total latency is kept below 32 ms (or below 20 ms) to prevent echoes that are annoying to the user.

[0047] In certain embodiments, the hearing system may be configured to generate an audible signal approximately 30-35 ms, 20-30 ms, 10-20 ms, 12-8 ms, 10-6 ms, or 8-3 ms from receipt of an incoming audio signal.

[0048] There are many variations of the disclosed training method. For example, the model can be trained to capture multiple audio streams from multiple microphones. The input data can be in the time domain or the time-frequency domain. The loss function can be the mean squared error of the signal or the mean squared error of a complex ideal ratio mask. The input data can include additional sensor data. The input data can include information about the desired target of the neural network, such as in the example where the network is trained to isolate sounds that match a particular sound signature and also receives the signature as input data. The model can also be trained to output each speaker separately or multiple speakers with a single signal. The training target of the model can be sounds with different SNRs (not just speech). The model can also be trained by unsupervised techniques, allowing the use of sounds without a clear target. The training data can be generated synthetically or by recording contemporaneous audio streams in the real world. The above variations are examples to explain the basic concepts and are not exhaustive of the potential variations in model training.

[0049] One exemplary embodiment of the NNE150 includes a recurrent neural network of approximately 40 million units organized into six layers. The network takes an 8 ms clip (or, alternatively, a frame) of audio data as input, and internally converts the chip to a time-frequency representation with a short-time Fourier transform. The network can then generate complex masks that can be applied to the original signal to change the phase and magnitude of each frequency. The network then outputs a clear time-domain audio signal.

[0050] In a further embodiment, the NNE 150 is composed of a convolutional neural network of about 1 million units organized into 13 layers. The first 6 layers correspond to the encoder, where the input is gradually downsampled along the frequency axis via strided 1D convolutions. A gated recurrent unit (GRU) layer is applied at the bottleneck layer to aggregate the temporal context. The decoder includes 6 layers, where the input from the bottleneck is gradually upsampled by transposed convolutions. The network takes as input a time-domain signal with speech and noise (split into 8 ms clips that are fed to the model in real time) and outputs a corresponding clear time-domain signal.

[0051] The NNE 150 then recombines the intermediate signals to generate a new signal. In some embodiments, the signals are recombined in a manner that maximizes the SNR by retaining only the signals (or signal components) that comprise the speech of interest. For example, the modified signal may contain only the voice of the target speaker. In another embodiment, the recombination is performed with a target SNR as the preferred SNR, the preference being determined by user-based and user-independent criteria. As used herein, SNR refers to the ratio of the power of the intermediate signals in the combined signal, each being an estimate of a particular sound source in the original signal, recognizing that such estimates are approximate.

[0052] The user-based criteria may comprise a user input in an application on a smartphone connected to the hearing aid via wireless communication. For example, the user may have the ability to slide or dial up and down the amount of desired background noise that is translated into the target SNR of the model. In another example, the user may have a preferred background noise level stored as a setting in the application, such that when the user selects noise cancellation, the desired SNR is already known as a predefined value. In another embodiment, the SNR may be determined as a function of clinical criteria. Here, the SNR is set to achieve clarity and comfort for the user based on the user's stored hearing profile while retaining a constant amount of ambient noise. In the case of multiple intermediate signals (i.e. multiple speakers), the logic described above is extended such that each subject is adjusted to achieve the desired SNR. The optimal SNR for two contemporaneous speakers may be different, given the constraint that the noise may be constant between the two. User-based criteria (i.e. user-defined criteria or user-controlled criteria) are further described in connection with FIG. 3B.

[0053] Once processed, the signal components (i.e., intermediate signals) are recombined by selecting the degree of amplification (i.e., gain) to be applied to each signal. The challenge in setting the gain is to ensure that the audio is mixed in a way that achieves the target SNR without causing the gain to vary too much. For example, if the SNR was targeted for every 4 millisecond sample of audio, the results would be meaningless because the SNR of the incoming signal measured over such a short sample would be highly unstable and the gain applied to each signal could change dramatically every 4 milliseconds. Thus, the NNE 150 may consider a slower moving average (or, in other words, may evaluate the relative loudness over a longer time window) to determine the SNR and may respond differently to changes in the volume of the background noise versus changes in the volume of the speaker.

[0054] To optimize the voice quality, user-independent criteria can be used. The user-independent criteria may comprise algorithms known to generally achieve a desired user experience. For example, in the absence of personalized settings, noise cancellation can target an SNR that generally leads to improved intelligibility for hearing impaired people. In an exemplary embodiment, the SNR can be dynamically set based on the performance of the NN model.

[0055] Another important user-independence in the recombination of intermediate signals is the estimation performance of the model. Even the best-trained models struggle in extremely low SNR (when the noise is much louder than the speech) as people with normal hearing do, because the noise completely masks the speech signal. Thus, in an exemplary embodiment, the SNR measurement is useful as an indicator of when the model is about to fail, allowing the system to fail gracefully rather than playing back estimated speech that would inevitably sound unintelligible and unnatural. In one embodiment, the model simply plays nothing. In another embodiment, the model can default back to the original signal. In yet another embodiment, the model may mix the target estimate with the original signal, or mix back some noise estimate that is the difference between the original signal and the speech estimate.

[0056] In some embodiments, the neural network model can use other measures of its performance as input to the recombination algorithm. Certain intermediate metrics calculated by the neural network may serve as proxies for the model's reliability and can be leveraged to monitor possible model failure. In one embodiment, the neural network can estimate the phase of the target signal using Gumbel softmax, and the pre-threshold value can be used as a frame-by-frame measure of model reliability. The processor can include other algorithms specifically tailored to measure the quality of the model output. Some examples are metrics commonly used in speech improvement research, such as PESQ and STOI, while other examples may be ones specifically developed for this purpose, such as lightweight neural networks trained simply to assess the quality of intelligible speech output.

[0057] In an exemplary embodiment, the NNE 150 combines a target SNR with a marginal SNR, where the target SNR is generated based on user input (such as the user adjusting the desired levels of background noise and speech in the app) and the marginal SNR represents the maximum achievable SNR that the model estimates it can achieve while still meeting certain estimated performance requirements. Thus, a user may want to set the noise reduction parameters to maximum and zero out background noise when there is overwhelming background noise, but the incoming SNR may be too challenging for the model to handle, so the model cannot successfully improve the incoming speech. In this case, the marginal SNR is determined to be the incoming SNR and the speech is played back unchanged. This may be more desirable than playing incomprehensible estimated speech.)

[0058] The NNE circuitry 150 can be updated via wireless communication with the processing device or the cloud. In a preferred embodiment, an application on the user's smartphone can connect to the cloud and download the updated model (retrained for better performance), which can then be transmitted to the device via wireless protocol. In another embodiment, the model is retrained on the smartphone using user-specific data collected by recording audio with the device. Once retrained, the updated model is transmitted to the hearing device.

[0059] In some embodiments, the NNE 150 may be executed on a remote device that communicates with the hearing aid. For example, the NNE 150 may be executed on a smart device (e.g., a smartphone) that communicates with the hearing aid. The hearing aid and the smart device may communicate over Bluetooth Low Energy (BTE). In yet another embodiment, some or all of the NNE 150 may be executed on an auxiliary device that communicates with the hearing aid. The auxiliary device may comprise any device capable of communicating with one or more servers capable of executing the machine language algorithms disclosed herein.

[0060] The DSP 140 comprises hardware, software, and combinations of hardware and software (firmware) for applying digital signal processing to the incoming frequency bands. In some embodiments, an important objective of the DSP processing is to improve the audibility and intelligibility of the incoming signal to the hearing aid wearer, taking into account the user's hearing loss. Traditionally, this is done by compensating for reduced volume sensitivity at certain frequencies, reduced dynamic range, and increased sensitivity to background noise. The DSP 140 can implement various digital signal processing algorithms to achieve dynamic range compression, amplification, and frequency adjustment (applying differential amplification to different frequency bands). The digital signal processing may comprise these traditional algorithms or may comprise additional processing functions (e.g., stationary noise reduction algorithms) configured to reduce background noise. In some embodiments, the DSP 140 can apply a predetermined gain to the incoming signal (e.g., the controller output signal 135 or the improved digital signal 155). The applied gain may be linear or non-linear and may be configured to improve the amplification of one frequency signal band relative to the other bands.

[0061] In an exemplary embodiment, the DSP 140 can pass the incoming signal through a filter bank. The filter bank splits the incoming signal into different frequency bands and applies a gain. The gain can be linear or non-linear for each frequency band or grouping of frequencies. The groupings of frequencies are often called channels. In a preferred embodiment, the specific parameters of the filters, particularly the gain, are user specific and are configured such that the final signal applies more amplification to frequencies where the user is more hard of hearing. The gain can be set to apply more amplification to quieter sounds than to relatively louder sounds, thereby compressing the dynamic range of the signal. In this embodiment, the parameters are set as a function of the user's hearing profile, including but not limited to audiograms. The process of tailoring the parameters applied by the DSP processor to a particular individual can be done either by the individual themselves through a fitting process in an app, or by a professional who can program the device through software connected to the device by wireless connection.

[0062] In another embodiment, the filters and gains are set by analyzing the incoming signal in the time-frequency domain. In some embodiments, the signal is received in this form so no STFT is required in the DSP 140, but in other embodiments, the processor receives the signal in the time domain and then applies the STFT. In some embodiments, an algorithm can be applied to different frequency bands or groups of frequency bands to analyze their content and set the gains accordingly. As an example, such an algorithm can be applied to identify which frequencies contain stationary noise and these frequencies can be attenuated (receive lower gain) to improve the SNR of the reproduced signal. After frequency gains have been applied to the different frequency bands, the bands can be recombined into one signal.

[0063] The output 145 of the DSP 140 is directed to a back-end / output processor 160. The back-end processing circuitry 160 may comprise one or more circuits for converting the processed signal band 145 into an audible signal in the time domain. As an example, the back-end processor 160 may comprise a digital-to-analog (DAC) converter (not shown) that converts the amplified digital signal into an analog signal. The DAC can then send the analog signal to a driver and one or more diaphragm-type speakers (not shown) to present the processed and amplified sound to a user. The speakers (not shown) may further comprise means for adjusting the output volume.

[0064] As mentioned above, the DSP 140 can receive signal data from either the control unit 130 or the NNE 150. This means that the signal may pass through the NNE 150 (receiving corresponding computational costs and associated improvements) or pass directly through the DSP 140. In either case, the DSP 140 may be engaged. If the NNE 150 is engaged, an additional step in the signal processing chain increases the system's power consumption and the time required for computation. The additional processing may result in additional latency for the end user.

[0065] In one implementation, the system 100 of FIG. 1 is formed on an IC. The IC may define a SoC. The integrated circuit may further include a speaker and a driver for the speaker. In the latter embodiment, the integrated circuit 100 may include one or more communication circuits to enable communication between the circuit 100 and one or more external devices that support the NNE 150. Such communication may include, for example, Bluetooth (BT) and Bluetooth Low Energy (BLE) or other short-range wireless technology range technologies.

[0066] As previously mentioned, one of the major obstacles to placing a neural network in a signal path is the power consumption required to run the neural network relative to the battery available for such processing. Accordingly, certain embodiments of the present invention must achieve high efficiency, as measured in operations per milliwatt, in the neural network circuitry in order to achieve good performance while maintaining long battery life.

[0067] In an exemplary embodiment, by slightly shortening the runtime or increasing the size of the battery, approximately 10 milliwatt hours of this battery can be freed up for neural network processing. Batteries found in conventional rechargeable hearing aids and headphones have a typical capacity of approximately 300 milliwatt hours. To use the speech improvement features and lead an active, social life, a user would ideally need access to 10 hours of neural network processing, which means that the neural network circuitry can consume only 1 milliwatt of additional power when it is active. Thus, achieving chip performance of 2-3 billion operations per milliwatt would result in a neural network computation budget of 2-3 billion operations per second, which is sufficient for speech separation. In other embodiments, a larger computational budget can be allocated to the neural network by reducing the total execution time (thereby allocating more battery budget to the neural network) or by reducing the execution time of the neural network (thereby increasing the neural network's budget per second).

[0068] To achieve efficient signal processing, the DSP 140 and the NNE 150 may be located in separate cores on the chip with different architectures suited to their respective tasks. For example, the neural network circuit may be configured for low precision numeric processing with an 8-bit (or less) arithmetic logic unit. It may also be configured for efficient data movement, such that all data required for the calculations is stored within the SOC. In some embodiments, this neural network core may also be configured such that the same processor used to run the neural network can be used for more traditional DSP operations, such as 24-bit operations. Thus, in some embodiments, the DSP 140 and the NNE 150 may run on the same processor.

[0069] FIG. 2 illustrates a schematic diagram of an exemplary front-end receiver 200 according to one embodiment of the present disclosure. In FIG. 2, incoming sound, which may be a combination of voice and ambient noise, is received at microphones 214 and 224. The microphones 214, 224 correspond to separate devices on the left and right sides of the user's head and receive incoming sound, identified as 210 and 220, respectively. In some embodiments, each device comprises multiple microphones. The microphones 214, 224 direct the received signals 210, 220 to ADCs 218, 228, respectively. The ADCs 218, 228 convert the received time-varying signals 210, 220 to corresponding digital representatives 219, 229. Once digitized, the signals 219, 229 are passed to the controller 130 of the respective device. In some embodiments, they are further passed to the controller of the opposite device, allowing processing of the binaural input data.

[0070] FIG. 3A is a schematic diagram of an exemplary system according to one embodiment of the present disclosure. Specifically, FIG. 3A illustrates an exemplary decision-making process that may be implemented in the control system. The control unit 300 may function as a signal processor that performs certain transformations and calculations on the incoming signal (e.g., 110 or 125, FIG. 1) to bring the incoming signal into the form required for processing and select the next processing step. In some embodiments, the control unit 300 may function as a selector switch that optimizes user selection, preferences, and power consumption. In an embodiment, the control system 300 may determine when to engage a larger NNE based on user preferences to amplify the sound of the user preferences.

[0071] The control system 300 of FIG. 3A can be implemented in a hearing aid or headphones. The control unit may be integrated into the hearing aid as hardware, software, or a combination of hardware and software. The control system 300 comprises a processor circuit 330 that receives an audio signal 325. The audio signal may be digital (e.g., 125, FIG. 1) or may be a time-varying signal (e.g., 110, FIG. 1). If the signal is time-varying, an additional ADC (not shown) may be used. As discussed in relation to FIG. 1, the digital audio signal may comprise multiple components, including one or more audio signals and ambient or background noise.

[0072] The processor 330 can receive user input from user controls 310. The user input may comprise user preferences dialed into the system from an auxiliary device such as a smartphone (see, e.g., FIG. 3B). A particular user preference may provide amplification parameters or preferences regarding the relative amplification of different sounds, which may determine the SNR. For example, a user may prefer amplification of voice over other environmental sounds. The user preferences may be obtained through a graphic user interface (GUI) implemented by an app on the user's auxiliary device, such as a smartphone. The user controls may be wirelessly delivered to the process circuitry 330. The user controls 310 may comprise a mode selection 312, a directionality selection 314, a source selection 316, and a target volume 318. These exemplary embodiments are described below with reference to FIG. 3B.

[0073] In an exemplary embodiment, system 300 may optionally include a module (not shown) for receiving and implementing a so-called wake word. A wake word may be one or more special words designated to wake up the device when spoken. Wake words are also known as hot words or trigger words. Processor 330 may have a designated wake word that may be utilized by a user to wake up NNE 350. The wake-up may override processor 330 and decision logic 335 and direct incoming audio to NNE 350. This is indicated by arrow 331.

[0074] The decision logic 335 is shown separately but may optionally be integrated with the processor circuitry 330. The decision logic 335 determines when to engage the NNE 350 and the degree of such engagement. The decision logic 335 may apply decision considerations provided by the user, the NNE, or a combination of both. The decision logic 335 may optionally consider input of the power indicator 305 indicating the available battery level. The decision logic 335 may also utilize such considerations to determine the degree of engagement of the NNE. The decision logic 335 determines whether to engage the NNE 350 (or a portion thereof), the DSP 340, or both. Once selected, the DSP 340 filters the incoming signal 325 into a myriad of different frequency bands. The processor 330 and the decision logic 335 may collectively determine when to engage the NNE 350. For example, the processor 330 may use proprietary logic in combination with user input to determine that the receive frequency band 325 comprises only background noise and not engage the NNE 350.

[0075] The received frequency band may comprise as many as 400 or more bands. The DSP 340 then assigns a different gain to each frequency band. The gain may be linear or non-linear. In one embodiment, the DSP 340 sets an ideal gain for each frequency, which significantly reduces noise.

[0076] FIG. 3B illustrates an exemplary graphic user interface (GUI) according to one embodiment of the present disclosure. The GUI may be implemented as an app on a smart device. The GUI allows user preferences to be communicated to the hearing aid device. Speech volume and background noise may be configured to allow the user to input preferences for speech and noise amplification, respectively. Directivity is an additional input that allows the user to increase the relative volume of noise coming from one direction (usually the front, but in other embodiments, the user may be able to select a different direction) relative to the user. Detected speakers allow the user to select a particular speaker whose voice will be amplified (compared to other voices that may be treated as noise). Mode selection 312 allows the user to select the operating mode of the device (exemplified by activation of a conversation mode). In some embodiments, selectable modes may include a conversation mode, an ambient mode, and an automatic mode. If the ambient mode is selected, the NNE 150 may be disengaged. Other modes, such as a voice mode, may indicate that noise cancellation is desired. The automatic mode may indicate that the processor 330 should make a best prediction of when to turn on the NNE 150 to match user preferences (e.g., when the user is engaged in a conversation and there is background noise).

[0077] Each of the total volume, speech volume, background noise, and directionality may have a dial or slider on the user's device to implement the user's particular preferences. Additional controls may also be included to accommodate one or more sound categories or sources. In some embodiments, the dials on the device act as volume controls for configured sound classes, such as speech and background noise. Turning the dials may raise or lower a user-defined SNR target for recombining the output of the neural network. In some embodiments, one device may have a dial for ambient volume control, while the other device may have a dial to change the level of background noise. In some embodiments, a single dial may adjust the SNR by dynamically adjusting either the speech volume or the noise volume based on the starting SNR or the incoming volume. For example, the SNR may be increased by initially gradually decreasing the volume of the background noise in the output signal, but once the background noise is completely removed, further improvements in the SNR may be achieved by increasing the volume of the speech signal (as the speech signal is still competing with the sounds entering the hearing device's surrounding ears). In some embodiments, the physical dials may be specifically configured in the smartphone app settings to assign different behaviors.

[0078] FIG. 3B shows speech volume, background noise level control, and mode switch. These parameters (along with or in combination with other parameters) can be used to determine the user's desired level of noise cancellation. Referring to FIG. 3A, the user's desired level of noise cancellation may be communicated to the NNE 350 via the processor 330 or may be directly input to the NNE 350 (not shown). When engaged, the NNE 350 can identify different sound sources and separate the incoming signals accordingly. Given the user-preferred level of noise cancellation, the NNE 350 can then apply appropriate amplification gains to the target sounds and noises.

[0079] In one embodiment, source selection 316 allows a user to pre-identify specific voices and match the identified voices with known individuals. Source selection 316 may be performed optionally. NNE 350, or a subset thereof, may be performed to allow a user to perform source selection. Upon matching the incoming frequency band with an identified individual, system 300 may perform steps to separate and amplify the individual's voice from ambient noise. Identified voices may include voices of caregivers, children, and family members. Other sounds, including alarms or emergency sirens, may also be identified to be easily separated and selectively amplified by the user or by system 300. In one embodiment, source selection 316 allows a user to identify one or more groups of sounds for amplification (or non-amplification).

[0080] Fig. 4 shows a signal processing system according to another embodiment of the present disclosure. The system of Fig. 4 can be implemented in a hearing device according to the disclosed principles. In Fig. 4, a front-end receiver 420 is shown, which combines incoming signals from different microphones into one digital signal, as described in relation to Fig. 2. The control system 430 includes a user control 434, an SNR detector 432, and decision logic 436.

[0081] The decision logic 436 communicates with both the DSP 440 and the NNE 450 as described in connection with FIG. 3A. In FIG. 4, the NNE 450 provides additional feedback to the decision logic 436, as indicated by arrow 451. In some embodiments, the NNE 450 measures an estimated SNR of the incoming signal, which can then serve as an input to the logic 436. If the SNR is extremely high, the NNE 450 may no longer be necessary. If the SNR is exceptionally low such that no voice is detected, the NNE 450 may be useless. In some embodiments, intermittently sending data to the NNE 450 provides a way to measure characteristics of the voice signal without constantly consuming power.

[0082] The example NNE 450 of FIG. 4 includes example modules for source separation 452, relative gain 454, recombiner 456, and performance monitoring 458. When activated, source separation 452 receives the incoming audio signal on a frame-by-frame basis. The audio can be received in the time domain or the time-frequency domain. For example, the frames are 10, 14, 16, or 20 milliseconds long. In some embodiments, the frames may be less than 1 millisecond or longer than 30 milliseconds. Each frame is processed through a neural network, which outputs one or more complex masks that can be used to separate one or more audio sources. By applying these masks, the source separation module 452 can filter each frame down to the audio source. Noise can be found by generating a mask for the noise or by subtracting all other separated sources from the original signal so that the noise is the remainder.

[0083] The relative gain module receives the user's auditory preferences from user control 434 and applies one or more relative gains to each frame received from source separation 452. The gains applied to different frequency bands in NNE 450 may be non-linear (compared to the gains applied in DSP 440). In this implementation, different gains may be applied on a source and frame-by-frame basis.

[0084] FIG. 5A illustrates the interaction between user preferences and the nonlinear gains applied by an exemplary NNE according to one embodiment of the present disclosure. In FIG. 5A, incoming sound in the form of a digitized signal 500 is directed to the NNE 510. The source separation 452 splits the incoming sound into different data streams, for example as a function of the respective sound source. This data is then directed as different bands to the relative gain filter 454, which applies different gains based on the user preferences, as indicated by arrow 435. The user preferences 540 determine the optimal combination (or optimal weights) of the various sound sources. The recombiner 456 then combines the differently weighted frequency bands to form a combined signal 580.

[0085] 4, the NNE 450 directs the recombined audio stream to the DSP 440 for further processing. Thus, according to one embodiment, the components of the NNE 450 estimate an ideal ratio mask that separates the audio and noise signals, apply a differential gain to each of the identified audio and noise signals, and combine the differentially amplified signals into one data stream.

[0086] A performance monitor module 458 may be optionally used. In one embodiment, the performance monitor module 458 examines the output signal of the NNE 450 to determine whether the output signal is within the hearing requirement criteria. If the output signal does not meet the requirements, the performance monitor module 458 may signal the decision logic 436 to divert the incoming signal directly to the DSP 440. This is shown by arrow 451. Otherwise, the NNE output may be directed to the DSP 440, as shown by arrow 459. In another embodiment, the performance monitor 458 may function as an ingress to the relative gain 454 and may limit the aggressiveness of the noise suppression if the performance monitor 458 detects an error in the source separation 452.

[0087] The DSP 440 includes, among other things, a filter bank 442 that separates the incoming signal into different frequency bands and a non-linear gain filter 444 that applies a gain to each band. In one embodiment, each filter identifies noise components in each different band and applies a noise cancellation gain to cancel the noise components.

[0088] An active noise cancellation (ANC) 425 is placed in the signal path between the front-end receiver 420 and the back-end receiver 460. ANC can be used optionally. The ANC 425 may comprise a processing circuit configured to receive an ADC signal from the hearing aid microphone and process the signal to improve the signal-to-noise ratio (SNR). Conventional ANC techniques can be used for noise cancellation. The input to the ANC 425 may be the incoming signal 421, optionally the control signal output 431, or both. The ANC process may be implemented for each unit of the hearing aid device to address the noise uncertainty associated with each unit. In an embodiment of the present disclosure, the ANC 425 may remain engaged even without a user control input 434 or without the engagement of the DSP or the engagement of the NNE. Considering the latency of the audio processing through the neural net and the low latency requirement of the ANC, the ANC is applied to the entire incoming signal (including both speech and noise components) and then the system plays back the speech after the processing is completed.

[0089] The back-end processor 460, as well as optional processor circuitry 462, includes a speaker 464. The speaker 464 may include a conventional hearing aid speaker that converts the processed digital signal into an audible signal.

[0090] FIG. 5B is an illustration of an exemplary NNE circuit logic implemented according to one embodiment of the disclosure. The logic may be implemented in an NNE engine circuit 550. A received audio signal is shown as an input 530. The received audio signal is directed to a neural network (NN) model 532. The NN model 532 may comprise an exemplary algorithm for separating sound sources or improving SNR according to the disclosed embodiments. The NN model 532 may comprise hardware, software, or a combination of hardware and software. The NN model 532 receives user preferences in the form of user controls 531, for example as described in connection with FIG. 3B. The output of the NN model 532 (NN output signal 533) is directed to a performance measurement unit 534. The performance measurement 534 implements a metric used to predict the performance of the neural network or to predict errors. These predictions may further be used as input to a recombiner 536, which attempts to optimize how the model's outputs are recombined to form a final signal. The recombiner 536 optimally recombines the outputs of the NN models 532 taking into account both the user preferences expressed from the user controls 531 and the outputs of the performance measures 534 .

[0091] In an exemplary embodiment, a performance measurement unit 534 receives the output signal 533 in successive frames and determines the SNR of each frame. The measurement unit then estimates the average SNR of the environment, which can be used to predict the model error (since model error generally increases with more difficult input SNR). The recombiner 536 also receives user preferences from the user control 531. Given the user preferences and the estimated SNR, the recombiner 536 then determines a set of relative gains to be applied to the signal 533 and communicates the gain values ​​to the recombiner 536. In an exemplary embodiment, the recombiner attempts to set the gains to best match the user preferences while maintaining the total error below a certain threshold.

[0092] The recombiner 536 applies the gain values ​​to the NN output signal 533 to obtain the output 538 signal. In one embodiment, multiple gain values ​​are communicated to the recombiner 536. Each gain value corresponds to an intermediate signal, which corresponds to an audio source. The recombiner 536 multiplies each gain value by a corresponding intermediate signal and combines the results to generate the output 538.

[0093] The following examples illustrate certain non-exhaustive implementations of the disclosed principles.

[0094] Example 1 - The average SNR value of signal 533 is below the threshold at which speech can be reliably separated (audible speech threshold). In this example, neural network processing is ineffective, regardless of user preferences or system capabilities. In this case, performance measurement unit 534 may set the gain so that the incoming signal is not altered, or relay a signal to control unit 130 as shown in Figure 1 to temporarily turn off neural network processing in order to conserve battery power.

[0095] Example 2 - The average SNR value of signal 533 is above the audible speech threshold and user preferences are applied. In this example, since the SNR value of signal 533 is above the audible speech threshold, the recombiner 536 can determine an appropriate gain. The gain may be determined as a function of the user preferences and the error of the estimated model. The performance measurement unit 534 then determines the gain that most closely approximates the SNR desired by the user while maintaining the error of the model as heard by the user below a certain threshold.

[0096] Example 3 - The average SNR value of the signal 533 is above the audible speech threshold and the recombiner 536 is aware of the user preferences. The recombiner 536 may ignore the user preferences in favor of estimating and applying a different set of relative gains. This may be because it understands that a higher quality sound may be obtained by applying a different gain criterion. In this example, the recombiner 536 substitutes its own criteria for providing an audible output signal 538 that may or may not exceed the user's SNR preferences. In this way, the system works with the NNE circuitry in the signal path to provide an audible signal in substantially real-time while gracefully dealing with the limitations of deep learning models in real-world environments.

[0097] FIG. 5C illustrates a schematic diagram of an exemplary architecture for engaging an NNE circuit according to an embodiment of the present disclosure. The architecture of FIG. 5C may be implemented in an NNE circuit. In FIG. 5C, an incoming signal 550 is received at an NN model 556. User preferences in the form of user controls 552 and a target source 554 are also provided to the NN model 556. The target source 654 may comprise one or more identified sources, for example, a pre-identified and stored voice of a known speaker or the user's own voice.

[0098] User preferences 652 may also be used to set a user's ideal SNR 662. The ideal SNR 562 may define a threshold SNR value that corresponds to the user's personal preferences and speech impairments. For example, the ideal SNR 562 may target an output SNR of 10db because that is a balance conveyed in the smartphone's user controls, or simply because the user's hearing profile is such that 10db is the minimum SNR at which the person can still reliably keep up with speech without effort.

[0099] The NN model 556 outputs a signal to a performance measurement unit 558. A general description of the performance measurement unit was provided in connection with FIG. 5B and will not be repeated here. In FIG. 5C, the performance measurement unit 558 identifies an intermediate signal 560, which may include, for example, a target frequency band and a noise band. The recombiner 590 may include SNR optimization logic 564. The optimization logic 564 receives the user's ideal SNR 562 as well as the output from the performance measurement unit 558 and determines whether to apply or deviate from the user preference (i.e., the ideal SNR 562). As a result, a set of gain values ​​568 is determined that are then respectively applied to the intermediate signals 560 to provide the output signal 570. It should be noted that in the exemplary embodiment of FIG. 5C, the recombiner 590 also applies the optimization logic 564 to determine the gain values ​​568.

[0100] In an exemplary embodiment, the performance measure 558 outputs a marginal SNR, which is the output SNR that keeps the audible distortion caused by model errors below a certain threshold. The SNR optimization logic then compares the marginal SNR to the ideal SNR determined based on user preferences, and takes the lower of the two. The gain is then set to target the SNR determined by this function.

[0101] Example 4 - In this example, compliance with the user preferred SNR 562 may require an output signal with an SNR of approximately 10db. The SNR optimization logic 564 can compare this value to the available system bandwidth and impose a -5db limit on the output signal 570. A gain value is then determined based on the -5db SNR. In this way, the SNR optimization logic 564 acts as an SNR limiter.

[0102] Thus, in accordance with certain disclosed principles, the NN model may be run on a small number of speech frames, for example, once per second, to obtain a preliminary SNR value. The frequency and duration of the speech frame tests can be varied.

[0103] FIG. 6 is a flow diagram illustrating an exemplary activation / deactivation of NNE circuitry according to one embodiment of the disclosure. Such a flow would be performed in the controller 130 of FIG. 1. In one embodiment, the exemplary process is directed to minimizing system power consumption while improving user experience. The disclosed process may be implemented in hardware, software, or a combination of hardware and software. The disclosed process may be implemented in various parts of the systems disclosed herein. For example, certain steps may be implemented in the front-end receiver, other steps may be implemented in the controller, and still other steps may be implemented in the NNE and DSP circuitry.

[0104] In one embodiment, the system monitors the incoming sound without continuously waking up the NNE circuitry, which may be accomplished by layering logic such that more computationally intensive tasks (i.e., power-intensive calculations) are performed only when necessary.

[0105] Referring to FIG. 6, in step 602, the system detects incoming sound. Step 602 can be implemented in the control unit with relatively low computational cost. Conventional sound detection mechanisms can be used in step 602. Once sound is detected, the system determines whether the detected sound exceeds a predefined threshold. This is shown in step 604. If the threshold is not met, the system returns to step 602 and continues to detect incoming sound. Steps 602 and 604 may operate continuously or may be performed intermittently. These steps may be performed in the front-end receiver unit or elsewhere in the system.

[0106] Sound detection can be performed on one or both sides of the hearing aid. Sound detection may be performed by infrequently analyzing speech frames in a low power mode. If the detected sound level exceeds a predefined threshold, the VAD may be activated in step 606. In step 608, the VAD determines whether the detected speech is continuous. If the detected speech is not continuous, then the process returns to step 602. If the detected speech is continuous, then in step 610, the sampling frequency of the incoming speech may be increased. Once activated, the logic can search for sustained speech through more frequent sampling of the incoming speech.

[0107] At step 612, the system engages the NNE circuitry to further process the incoming audio signal. When engaging the NNE circuitry, the system may consider several competing interests. For example, the system may consider the user's input, the ability of the NNE to provide a meaningful SNR (i.e., the performance limits of the NNE), and power availability. In an embodiment, if continued speech is detected, then the entire NNE circuitry may be engaged to analyze the incoming audio while still leaving the output to the user unmodified. This allows the device to analyze the SNR of the incoming audio and determine whether it is preferable to activate the NNE.

[0108] If the NNE is activated, the output is optionally modified according to the user's settings and the audio stream is delivered to the user in step 614. Additionally, the NNE may use the output of the same model to analyze the SNR of the incoming audio stream or audio clip and inform whether the NNE should remain activated.

[0109] In step 618, the control unit, having received the SNR feedback from the NNE, determines whether the SNR exceeds the NNE's limit for providing audible speech. For example, if the SNR of the incoming voice is very high (conversation in a quiet room), no NNE processing is required. Therefore, the system looks at a threshold SNR level set by the user or the device itself (e.g., if an automatic mode is selected). If the SNR is high enough that the NNE cannot provide audible speech even when fully engaged, the system can refuse to filter, as described above. If the NSR level does not exceed the NNE's limit, then the algorithm can process the incoming signal at a level determined by the system or the user (i.e., select the lower level of the target SNR or the NNE's limit SNR). This step is shown as step 620 in FIG. 6. The process may then return to step 602.

[0110] FIG. 7 illustrates a block diagram of an SOC package according to an exemplary embodiment. In FIG. 7, the SOC 702 includes one or more central processing unit (CPU) cores 720, an input / output (I / O) interface 740, and a memory controller 742. The various components of the SOC package 702 may be optionally coupled to interconnects or buses as described herein with reference to other figures. The SOC package 702 may also include components as described with reference to the hearing aid systems of FIGS. 1-6. Additionally, each component of the SOC package 720 may include one or more other components, for example, as described with reference to FIG. 2 or FIG. 3. In one embodiment, the SOC package 702 (and its components) are provided on one or more integrated circuit (IC) dies, which are packaged in a single semiconductor device, for example. The single semiconductor device may be configured to be used as a hearing aid, an amplification system, or a hearing device used in the human ear canal.

[0111] As shown in FIG. 7, the SOC package 702 is coupled to the memory 760 via the memory controller 742. In an embodiment, the memory 760 (or a portion thereof) can be integrated on the SOC package 702. The I / O interface 740 may be coupled to one or more I / O devices 770, for example, via an interconnect and / or bus as discussed herein. The I / O devices 770 may include means for communicating with the SOC 702. In an exemplary embodiment, the I / O interface 740 wirelessly communicates with the I / O devices 770. The SOC package 702 may include, for example, hardware, software, and logic for implementing the embodiments of FIG. 1 and FIG. 4. The implementation may communicate with auxiliary devices, for example, the I / O devices 770. The I / O devices 770 may include additional communication capabilities, for example, cellular or WiFi, for accessing the NNE.

[0112] 8 is a block diagram of an exemplary auxiliary processing system 800 that may be used in connection with the disclosed principles. In various embodiments, system 800 includes one or more processors 802 and one or more graphics processors 808 and may be a single-processor desktop system, a multi-processor workstation system, or a server system having multiple processors 802 or processor cores 807. In one embodiment, system 800 is a processing platform integrated within a system-on-chip (SoC or SOC) integrated circuit for use in mobile, handheld, or embedded devices.

[0113] An embodiment of the system 800 can include or be incorporated into a server-based smart device platform or an online server accessible to the Internet. In some embodiments, the system 800 is a mobile phone, a smartphone, a tablet computing device, or a mobile Internet device. The data processing system 800 may also include a connection with or be integrated within a wearable device, such as a smart watch wearable device, a smart eyewear device (e.g., faceworn glasses), an augmented reality device, or a virtual reality device. In some embodiments, the data processing system 800 is a television or set-top box device having one or more processors 802 and a graphical interface generated by one or more graphics processors 808.

[0114] In some embodiments, the one or more processors 802 each include one or more processor cores 807 for processing instructions that, when executed, perform operations of the system and user software. In some embodiments, each of the one or more processor cores 807 is configured to process a particular instruction set 809. In some embodiments, the instruction set 809 may facilitate operations via Complex Instruction Set Computing (CISC), Reduced Instruction Set Computing (RISC), or Very Long Instruction Words (VLIW). The multiple processor cores 807 may each process a different instruction set 809, which may include instructions that facilitate emulation of other instruction sets. The processor cores 807 may include other processing devices, such as digital signal processors (DSPs).

[0115] In some embodiments, the processor 802 includes a cache memory 804. Depending on the architecture, the processor 802 may have a single internal cache or multiple levels of internal cache. In some embodiments, the cache memory is shared among various components of the processor 802. In some embodiments, the processor 802 also uses an external cache (e.g., a level-3 (L3) cache or a last level cache (LLC)) (not shown), which may be shared among the processor cores 807 using known cache coherency techniques. A register file 806 is further included in the processor 802 and may include different types of registers (e.g., integer registers, floating point registers, status registers, and instruction pointer registers) for storing different types of data. Some registers may be general purpose registers and other registers may be specific to the design of the processor 802.

[0116] In some embodiments, processor 802 is coupled to processor bus 88 for carrying communication signals, such as address, data, or control signals, between processor 802 and other components in system 800. In one embodiment, system 800 uses an exemplary "hub" system architecture that includes a memory controller hub 816 and an input / output (I / O) controller hub 830. Memory controller hub 816 facilitates communication between memory devices and other components of system 800, while I / O controller hub (ICH) 830 provides connectivity to I / O devices via a local I / O bus. In one embodiment, logic for memory controller hub 816 is integrated within the processor.

[0117] The memory device 820 may be a dynamic random access memory (DRAM) device, a static random access memory (SRAM) device, a flash memory device, a phase change memory device, or other memory device with suitable performance to function as a process memory. In one embodiment, the memory device 820 can operate as a system memory for the system 800 to store data 822 and instructions 821 for use by the one or more processors 802 when executing applications or processes. The memory controller hub 816 can also couple to an optional external graphics processor 812 and communicate with one or more graphics processors 808 in the processor 802 to perform graphics and media operations.

[0118] In some embodiments, the ICH 830 allows peripherals to connect to the memory device 820 and the processor 802 via a high-speed I / O bus. The I / O peripherals include, but are not limited to, an audio controller 846, a firmware interface 828, a wireless transceiver 826 (e.g., Wi-Fi, Bluetooth), a data storage device 824 (e.g., hard disk drive, flash memory, etc.), and a legacy I / O controller 840 for coupling legacy (e.g., Personal System 2 (PS / 2)) devices to the system. One or more universal serial bus (USB) controllers 842 connect input devices such as a keyboard and mouse 844 combination. A network controller 834 may also be connected to the ICH 830. In some embodiments, a high-performance network controller (not shown) is coupled to the processor bus 88. It will be appreciated that the illustrated system 800 is illustrative and not limiting, as other types of data processing systems in different configurations may also be used. For example, I / O controller hub 830 may be integrated within one or more processors 802, or, alternatively, memory controller hub 816 and I / O controller hub 830 may be integrated into a separate external graphics processor, such as external graphics processor 812.

[0119] 9 is a generalized diagram of a machine learning software stack 900. A machine learning application 1102 may be configured to train a neural network using a training dataset or to use a trained deep neural network to perform machine intelligence related to the disclosed principles. The machine learning application 902 may include neural network training and inference functions and / or specialized software that can be used to train the neural network prior to deployment to a hearing device. The machine learning application 902 may perform any type of machine intelligence including, but not limited to, image recognition, mapping and localization, autonomous navigation, speech synthesis, medical image processing, or language translation.

[0120] Hardware acceleration of the machine learning application 902 can be enabled through the machine learning framework 904. The machine learning framework 904 can provide a library of machine learning primitives. Machine learning primitives are basic operations typically performed by machine learning algorithms. Without the machine learning framework 904, developers of machine learning algorithms would need to create and optimize the main computational logic associated with the machine learning algorithm and reoptimize the computational logic each time a new parallel processor is developed. Instead, machine learning applications can be configured to perform the required computations using primitives provided by the machine learning framework 904. Exemplary primitives include tensor convolutions, activation functions, and pooling, which are computational operations performed while training a convolutional neural network (CNN). The machine learning framework 904 can also provide primitives for implementing basic linear algebra subprograms performed by many machine learning algorithms, such as matrix and vector operations.

[0121] The machine learning framework 904 can process input data received from the machine learning application 902 and generate appropriate inputs to the computing framework 906. The computing framework 906 can abstract the underlying instructions provided to the GPGPU driver 908 to enable the machine learning framework 904 to take advantage of hardware acceleration via the GPGPU hardware 910 without requiring the machine learning framework 904 to be familiar with the architecture of the GPGPU hardware 910. Additionally, the computing framework 1106 can enable hardware acceleration of the machine learning framework 904 across different types and generations of GPGPU hardware 910.

[0122] The computational architecture provided by the embodiments described herein can be configured to perform a type of parallel processing that is particularly suitable for training and deploying neural networks for machine learning implementation on hearing devices.Neural networks can be generalized as networks of functions with graph relationships.As known in the art, there are various types of neural network implementations used in machine learning.One exemplary type of neural network is the feedforward network, as mentioned above.

[0123] A second exemplary type of neural network is a CNN. A CNN is a specialized feed-forward neural network for processing data with a known lattice-like topology, such as image data. Thus, CNNs are commonly used for computing vision and image recognition applications, but also for other types of pattern recognition, such as inference, speech, and language processing. The nodes of a CNN input layer are organized into a set of filters (feature detectors inspired by the receptive fields found in the retina), and the output of each set of filters is propagated to the nodes of subsequent layers of the network. A CNN computation involves applying a convolution operation to each filter to generate the output of that filter. A convolution is a type of specialized mathematical operation performed by two functions to generate a third function that is a modification of one of the two original functions. In convolutional network terminology, the first function of the convolution is called the input, and the second function is called the convolution kernel. The output is called a feature map. For example, the input to a convolutional layer can be a multidimensional array of data that defines the various color components of the incoming image. The convolution kernel can be a multidimensional array of parameters, which are adapted by a neural network training process.

[0124] Recurrent neural networks (RNNs) are a family of feed-forward neural networks that contain feedback connections between layers. RNNs allow for the modeling of sequential data by sharing parameter data across different parts of the neural network. The architecture of an RNN contains cycles. A cycle represents the effect that a variable's current value has on its own value at a future time point, as at least a portion of the output data from the RNN is used as feedback to process subsequent inputs in the sequence. This feature makes RNNs particularly useful for auditory processing due to the variable nature in which auditory data can be constructed.

[0125] The figures described herein present exemplary feedforward, CNN, and RNN networks and explain the general process for training and deploying each of these types of networks, respectively. It will be understood that these descriptions are exemplary and non-limiting with respect to any certain embodiments described herein, and that the illustrated concepts may be generally applied to deep neural networks and machine learning techniques in general.

[0126] The exemplary neural networks described above can be used to perform deep learning to implement one or more of the disclosed principles. Deep learning is machine learning that uses deep neural networks. Deep neural networks used in deep learning are artificial neural networks that are composed of multiple hidden layers, as opposed to shallow neural networks that only have a single hidden layer. Deeper neural networks generally require more computational effort to train. However, the increased number of hidden layers in the network allows for multi-stage pattern recognition, resulting in reduced output error compared to shallow machine learning techniques.

[0127] Deep neural networks used in deep learning typically comprise a front-end network performing feature recognition coupled to a back-end network representing a mathematical model capable of performing operations (e.g., object classification, noise and / or speech recognition, etc.) based on feature representations provided to the model. Deep learning makes it possible to perform machine learning without the need to perform manual feature engineering for the model. Instead, deep neural networks can learn features based on statistical structures or correlations in the input data. The learned features can be provided to a mathematical model that can map the detected features to an output. The mathematical model used by the network is generally specialized for the particular task being performed, and different models are used to perform different tasks.

[0128] Once a neural network is structured, a learning model can be applied to the network to train it to perform a specific task. The learning model describes how to adjust the weights in the model to reduce the network's output error. Backpropagation of error is a commonly used method for training neural networks. An input vector is presented to the network for processing. The output of the network is compared to the desired output using a loss function, and an error value is calculated for each neuron in the output layer. The error values ​​are propagated backwards until each neuron has an associated error value that roughly represents its contribution to the original output. An algorithm such as the stochastic gradient descent algorithm is then used to learn from these errors and update the neural network weights.

[0129] FIG. 10 illustrates the training and deployment of a deep neural network according to an embodiment of the present disclosure. Once a given auditory network is structured for a task, the neural network can be trained using a training dataset 1002. Various training frameworks have been developed to allow hardware acceleration of the training process. For example, the machine learning framework 904 of FIG. 9 can be configured as a training framework 1004. The training framework 1004 can be hooked to an untrained neural net 1006 and can train the untrained neural net using parallel processing resources described herein to generate a trained neural net 1008. To start the training process, initial weights (e.g., amplification gains corresponding to sound sources) can be selected randomly or by pre-training using a deep belief network. Then, a training cycle is performed in either a supervised or unsupervised manner.

[0130] Supervised learning is a learning method in which training is performed as a mediated operation, such as when the training data set 1002 comprises inputs paired with desired outputs for the inputs, or when the training data set includes inputs with known outputs and the neural network's output is manually graded. The network processes the inputs and compares the resulting output to a set of expected or desired outputs. Errors are then propagated through the system. The training framework 1004 can be adjusted to adjust the weights that control the untrained neural network 1006. The training framework 1004 can provide tools to monitor how well the untrained neural network 1006 is converging towards a model suitable for generating correct answers based on known input data. The training process occurs iteratively such that the network weights are adjusted to improve the output generated by the auditory neural network. The training process can continue until the neural network reaches a statistically desired accuracy associated with the trained neural network 1208. This determination can be made by technical and hearing experts or can be performed at the machine level. The trained neural network 1008 can then be deployed to perform any number of machine learning operations.

[0131] Unsupervised learning is an exemplary learning method in which the network attempts to train itself using unlabeled data. Thus, in the case of unsupervised learning, the training dataset 1002 includes input data without associated output data. The untrained neural network 1006 can learn groupings within the unlabeled inputs and determine how individual inputs relate to the entire dataset. Unsupervised training can be used to generate self-organizing maps, which are a type of trained neural network 1007 that can perform operations useful for reducing the dimensionality of data. Unsupervised training can also be used to perform anomaly detection, which can identify data points within the input dataset that deviate from normal patterns in the data.

[0132] Variations of supervised and unsupervised learning can also be employed. Semi-supervised learning is a technique in which the training data set 1002 contains a mixture of labeled and unlabeled data of the same distribution. Incremental learning is a variant of supervised learning in which input data is continually used to further train the model. Incremental learning allows the trained neural network 1008 to adapt to new data 1012 without forgetting the knowledge implanted in the network during initial training. All of the above training can be performed in conjunction with hearing professionals, physicians, or technicians.

[0133] The training process of neural networks, especially deep ones, whether supervised or unsupervised, can be computationally too intensive for a single computation node. Instead of using a single computation node, a distributed network of computation nodes can be used to accelerate the training process.

[0134] Example 1 is directed to an apparatus for improving an incoming audio signal, the apparatus comprising: a controller that receives the incoming signal and provides a controller output signal; a neural network engine (NNE) circuit in communication with the controller, the NNE circuit being activateable by the controller and configured to generate an NNE output signal from the controller output signal; and a digital signal processing (DSP) circuit that receives one or more of the controller output signal or the NNE circuit output signal and thereby generates a processed signal, the controller determining a processing path of the controller output signal through one of the DSP circuit or the NNE circuit as a function of one or more of predetermined parameters, characteristics of the incoming signal, and feedback of the NNE circuit.

[0135] Example 2 is directed to the apparatus of Example 1, where the predetermined parameters comprise a user-defined characteristic and a user-independent characteristic.

[0136] Example 3 is directed to the apparatus of example 2, wherein the user-defined characteristics further comprise one or more of a user signal-to-noise ratio (U-SNR) threshold and natural speaker identification information.

[0137] Example 4 is directed to the apparatus of example 2, where the user-independent characteristic comprises one or more of an available power level and a system signal-to-noise ratio (S-SNR) threshold.

[0138] Example 5 is directed to the apparatus of example 1, where the characteristic of the incoming signal comprises detectable speech or detectable silence.

[0139] Example 6 is directed to the apparatus of example 5, wherein the controller disengages at least one of the DSP or the NNE upon detecting silence, silence being defined by a noise level below a predetermined threshold.

[0140] Example 7 is directed to the apparatus of example 1, where the feedback of the NNE circuit comprises a detected SNR value.

[0141] Example 8 is directed to the apparatus of example 1, where the feedback of the NNE circuitry comprises an indication of voice detection in the NNE circuitry.

[0142] Example 9 is directed to the apparatus of example 1, wherein the controller is configured to send an audio clip to the NNE circuitry and receive feedback of the NNE circuitry.

[0143] Example 10 is directed to the apparatus of example 9, in which the audio clip defines a portion of the incoming signal and is transmitted intermittently from the control unit.

[0144] Example 11 is directed to the apparatus of Example 9, wherein the audio clips have a predetermined length and are transmitted at predetermined intervals and at a frequency, the frequency of transmission being determined as a function of a feedback signal of the NNE circuit.

[0145] Example 12 is directed to the apparatus of example 1, wherein the controller determines a processing path for the controller output signal substantially in real time.

[0146] Example 13 is directed to the device of Example 1, in which the controller, the DSP, and the NNE are integrated on a system on chip (SOC).

[0147] Example 14 is directed to the device of Example 1, in which the controller, DSP, and NNE are integrated into a hearing aid configured to fit to be worn on a human ear.

[0148] Example 15 is directed to the device of example 1, further comprising an active noise cancellation (ANC) circuit for processing the controller output signal.

[0149] Example 16 is directed to a method of improving the quality of an incoming audio signal comprising: receiving the incoming signal at a control unit and providing a control unit output signal; activating a neural network engine (NNE) and processing the control unit output signal to generate an NNE output signal and an NNE feedback signal; and activating a digital signal processing (DSP) circuit to receive one or more of the control unit output signal and the NNE circuit output signal and generate a processed signal, wherein the control unit determines a processing path of the control unit output signal through one of the DSP circuit or the NNE circuit as a function of one or more of predetermined parameters, characteristics of the incoming signal, and the NNE circuit feedback.

[0150] Example 17 is directed to the method of example 16, wherein the predetermined parameters comprise a user-defined characteristic and a user-independent characteristic.

[0151] Example 18 is directed to the method of Example 17, wherein the user-defined characteristics further comprise one or more of a user signal-to-noise ratio (U-SNR) threshold and natural speaker identification information.

[0152] Example 19 is directed to the method of example 17, wherein the user-independent characteristics further comprise one or more of an available power level and a system signal-to-noise ratio (S-SNR) threshold.

[0153] Example 20 is directed to the method of example 16, wherein the characteristic of the incoming signal comprises a detectable sound or detectable silence.

[0154] Example 21 is directed to the method of Example 20, further comprising disengaging the DSP and the NNE upon detecting silence in the control unit.

[0155] Example 22 is directed to the method of example 16, further comprising detecting an SNR value, the NNE providing the detected SNR value as a feedback signal for the NNE circuit.

[0156] Example 23 is directed to the method of example 16, wherein the NNE feedback signal further comprises an indication of voice detection at the NNE.

[0157] Example 24 is directed to the method of Example 16, further comprising transmitting an audio clip from the controller to the NNE prior to receiving the NNE feedback signal.

[0158] Example 25 is directed to the method of example 24, wherein the audio clip defines a portion of the incoming signal and is transmitted intermittently.

[0159] Example 26 is directed to the method of example 24, wherein the audio clips have a predetermined length and are transmitted at predetermined intervals and at a frequency, the frequency of transmission being determined as a function of a feedback signal of the NNE circuit.

[0160] Example 27 is directed to the method of example 16, further comprising determining the processing path in real time with the controller.

[0161] Example 28 is directed to the method of Example 16, further comprising integrating the controller, the DSP, and the NNE onto a system on a chip (SOC).

[0162] Example 29 is directed to the method of Example 16, further comprising integrating the controller, the DSP, and the NNE into a hearing aid configured to fit a human ear.

[0163] Example 30 is directed to the method of Example 16, further comprising engaging an active noise cancellation (ANC) circuit when processing the controller output signal via the NNE circuit.

[0164] Example 31 is directed to at least one non-transitory machine-readable medium comprising instructions that, when executed by computing hardware including a processor circuit coupled to a memory circuit, cause the computing hardware to receive an incoming signal at a control unit and provide a control unit output signal, activate a neural network engine (NNE) and process the control unit output signal to generate an NNE output signal and an NNE feedback signal, and activate a digital signal processing (DSP) circuit to receive one or more of the control unit output signal and the NNE circuit output signal and generate a processed signal, wherein the control unit determines a processing path of the control unit output signal through one of the DSP circuit or the NNE circuit as a function of one or more of predetermined parameters, characteristics of the incoming signal, and the NNE circuit feedback.

[0165] Example 32 is directed to the medium of example 31, wherein the predetermined parameters comprise a user-defined characteristic and a user-independent characteristic.

[0166] Example 33 is directed to the medium of Example 32, wherein the user-defined characteristics further comprise one or more of a user signal-to-noise ratio (U-SNR) threshold and natural speaker identification information.

[0167] Example 34 is directed to the medium of example 32, wherein the user-independent characteristic further comprises one or more of an available power level and a system signal-to-noise ratio (S-SNR) threshold.

[0168] Example 35 is directed to the medium of example 31, where the characteristic of the incoming signal comprises detectable speech or detectable silence.

[0169] Example 36 is directed to the medium of example 35, wherein the instructions further cause the computing hardware to disengage the DSP and the NNE upon detecting silence in the control unit.

[0170] Example 37 is directed to the medium of example 31, wherein the instructions further cause the computing hardware to detect an SNR value and an NNE and provide the detected SNR value as a feedback signal for the NNE circuit.

[0171] Example 38 is directed to the medium of example 31, wherein the NNE feedback signal further comprises an indication of voice detection in the NNE.

[0172] Example 39 is directed to the medium of example 31, wherein the instructions further cause the computing hardware to transmit an audio clip from the control unit to the NNE before receiving the NNE feedback signal.

[0173] Example 40 is directed to the medium of example 39 in which the audio clip defines a portion of the incoming signal and is transmitted intermittently.

[0174] Example 41 is directed to the medium of example 39, in which the audio clips have a predetermined length and are transmitted at predetermined intervals and at a frequency, the frequency of transmission being determined as a function of a feedback signal of the NNE circuit.

[0175] Example 42 is directed to the medium of example 31, wherein the instructions cause the computing hardware to further determine the processing path in real time at the controller.

[0176] Example 43 is directed to the medium of Example 31, in which the controller, the DSP, and the NNE are integrated into a hearing aid configured to fit a human ear.

[0177] Example 44 is directed to a hearing system for improving an incoming audio signal, comprising: a front-end receiver for receiving one or more incoming audio signals, at least one of the incoming audio signals having a plurality of signal components, each signal component corresponding to a respective signal source; a controller in communication with the front-end receiver, the controller receiving an input signal from the front-end receiver and providing a controller output signal, and selectively providing an output signal to at least one of a first or second signal processing path; a neural network engine (NNE) circuit in communication with the controller to define a portion of a first signal processing path, the NNE circuit being activateable by the controller and configured to generate an NNE output signal from the controller output signal; and a digital signal processing (DSP) circuit forming part of the first and second signal processing paths, the DSP circuit receiving one or more of the controller output signals or the NNE circuit output signals and thereby generating a processed signal, wherein the front-end receiver, the controller, the NNE circuit, and the DSP circuit are formed on an integrated circuit (IC).

[0178] Example 45 is directed to the hearing system of Example 44, further comprising a back-end receiver that receives the output signal from the DSP and forms an audible signal.

[0179] Example 46 defines one of a hearing aid, headphones, or faceworn glasses, where the audible signal is directed to the hearing system of Example 45, where the audible signal is formed less than 32 milliseconds after receiving the incoming signal.

[0180] Example 47 is directed to the hearing system of Example 44, wherein the IC comprises a system on a chip (SOC).

[0181] Example 48 is directed to the hearing system of Example 47, further comprising a housing that receives the SOC and a power source.

[0182] Example 49 is directed to the hearing system of Example 44, wherein the controller determines a processing path of the controller output signal as a function of feedback of the NNE circuit.

[0183] Example 50 is directed to the hearing system of Example 44, wherein the controller determines a processing path of the controller output signal as a function of one or more of predetermined parameters, characteristics of the incoming signal, and feedback of the NNE circuitry.

[0184] Example 51 is directed to the hearing system of example 44, further comprising a wireless communication system.

[0185] Example 52 is directed to the hearing system of example 44, in which the NNE circuitry adjusts the relative volume of the components of the incoming signal and the DSP circuitry applies a frequency and time varying gain to the received signal.

[0186] Example 53 is directed to the hearing system of example 52, wherein the components of the incoming signal further comprise at least speech and noise, and the speech volume is increased relative to the noise volume.

[0187] Example 54 is directed to the hearing system of Example 44, where the front-end receiver processes an incoming signal to provide an input signal to the control unit, the incoming signal including one or more of a speech component and a noise component.

[0188] Example 55 is an NNE circuit selectively applying a ratio mask to an incoming signal at the front-end receiver to obtain a plurality of components, each of the plurality of components corresponding to a sound class, to be directed to the hearing system of example 52.

[0189] Example 56 is directed to the hearing system of Example 44, wherein the NNE circuit is configured to selectively apply a complex ratio mask to the control unit output signal to obtain a plurality of signal components, each of the plurality of signal components corresponding to a voice class or an individual speaker, and the NNE circuit is further configured to combine the plurality of components into an output signal, the volume of each component being adjusted relative to at least one other component in accordance with a predetermined user-controlled signal-to-noise ratio.

[0190] Example 57 is directed to the hearing system of example 56, wherein the signal components further comprise speech and noise, and the output signal comprises an increased speech volume relative to the noise volume.

[0191] Example 58 is directed to the hearing system of Example 56, wherein the signal components further comprise a user's speech and a plurality of other sound sources, and the output signal comprises the user's speech attenuated relative to the other sound sources.

[0192] Example 59 is directed to the hearing system of Example 56, wherein the NNE circuitry is further configured to set the volume of each of the different sound sources as a function of a user-controlled parameter.

[0193] Example 60 is directed to the hearing system of Example 44, where the second signal processing path excludes signal processing through the NNE.

[0194] Example 61 is directed to the hearing system of Example 44, wherein the NNE circuitry is further configured to perform one or more DSP functions.

[0195] Example 62 is directed to a method of improving quality of an incoming audio signal comprising: receiving one or more incoming audio signals at a front-end receiver, at least one of the incoming audio signals having a plurality of signal components, each signal component corresponding to a respective signal source; receiving an input signal from the front-end receiver at a controller and providing a controller output signal, the controller selectively providing an output signal to at least one of a first or second signal processing path; generating an NNE output signal from the controller output signal at a neural network engine (NNE) circuit activateable by the controller, the NNE defining at least a portion of the first signal processing path; and generating a processed signal from the controller output signal or the NNE circuit output signal at a digital signal processing (DSP) circuit, the DSP defining at least a portion of the first and second signal processing paths; wherein the front-end receiver, the controller, the NNE circuit, and the DSP circuit are formed on an integrated circuit (IC).

[0196] Example 63 is directed to the method of example 62, further comprising forming an output signal from the processed signal at a back-end receiver.

[0197] Example 64 is directed to the method of example 63, further comprising forming the output signal less than 32 milliseconds after receiving the incoming signal.

[0198] Example 65 is directed to the method of example 63, wherein the hearing system defines one of a hearing aid, headphones, or faceworn glasses.

[0199] Example 66 is directed to the method of example 62, wherein the IC comprises a system on a chip (SOC).

[0200] Example 67 is directed to the method of example 66, further comprising a housing that receives the SOC and the power source.

[0201] Example 68 is directed to the method of Example 62, further comprising determining a processing path of the controller output signal as a function of feedback of the NNE circuit.

[0202] Example 69 is directed to the method of Example 62, further comprising determining a processing path of the controller output signal as a function of one or more of a predetermined parameter, a characteristic of the incoming signal, and a feedback of the NNE circuit.

[0203] Example 70 is directed to the method of Example 62, further comprising processing an incoming signal having one or more of a speech component and a noise component at a front-end receiver to provide an input signal to a controller.

[0204] Example 71 is directed to the method of example 70, in which the NNE circuit selectively applies a ratio mask to the incoming signal at the front-end receiver to obtain a plurality of components, each of the plurality of components corresponding to a voice class.

[0205] Example 72 is directed to the method system of Example 62, further comprising: applying a complex ratio mask to the controller output signal in the NNE circuit to obtain a plurality of signal components, each of the plurality of signal components corresponding to a voice class or an individual speaker; and combining the plurality of components into an output signal in the NNE circuit, wherein a volume of each component is adjusted relative to at least one other component according to a predetermined user-controlled signal-to-noise ratio.

[0206] Example 73 is directed to the method of example 72, wherein the signal components further comprise speech and noise, and the output signal comprises an increased speech volume relative to the noise volume.

[0207] Example 74 is directed to the method of example 72, wherein the signal components further comprise a user's speech and a plurality of other sound sources, and the output signal comprises the user's speech attenuated relative to the other sound sources.

[0208] Example 75 is directed to the method of Example 72, wherein the NNE circuitry is further configured to set a volume for each of the different sound sources as a function of a user-controlled parameter.

[0209] Example 76 is directed to the method of example 62, wherein the signal processing through the first signal processing path excludes signal processing through the NNE.

[0210] Example 77, when executed by computing hardware including a processor circuit coupled to a memory circuit, includes causing the computing hardware to: receive one or more incoming audio signals at a front end receiver, at least one of the incoming audio signals having a plurality of signal components, each signal component corresponding to a respective signal source; receive an input signal from the front end receiver and provide a controller output signal, the controller selectively providing an output signal to at least one of a first signal processing path or a second signal processing path; and operate a neural network engine (N) activateable by the controller. At least one non-transitory machine-readable medium comprising instructions for generating, in a front-end receiver (NE) circuit, an NNE output signal from a controller output signal, the NNE defining at least a portion of a first signal processing path, and generating, in a digital signal processing (DSP) circuit, a processed signal from the controller output signal or the NNE circuit output signal, the DSP defining at least a portion of the first and second signal processing paths, wherein the front-end receiver, controller, NNE circuit, and DSP circuit are formed on an integrated circuit (IC).

[0211] Example 78 is directed to the medium of example 77, wherein the instructions further cause the computing hardware to form an output signal from the processed signal at a back-end receiver.

[0212] Example 79 is directed to the medium of example 78, wherein the instructions further cause the computing hardware to form the output signal in less than 32 milliseconds after receiving the incoming signal.

[0213] Example 80 is directed to the medium of example 78, where the hearing system defines one of a hearing aid, headphones, or facework glasses.

[0214] Example 81 is directed to the medium of example 77, wherein the IC comprises a system on a chip (SOC).

[0215] Example 82 is directed to the medium of example 77, wherein the instructions further cause the computing hardware to determine a processing path for the controller output signal as a function of feedback of the NNE circuit.

[0216] Example 83 is directed to the medium of example 77, wherein the instructions further cause the computing hardware to determine a processing path for the controller output signal as a function of one or more of predetermined parameters, characteristics of the incoming signal, and feedback of the NNE circuitry.

[0217] Example 84 is directed to the medium of example 77, wherein the instructions further cause the computing hardware to process an incoming signal having one or more of a speech component and a noise component at a front end receiver and provide the incoming signal to a controller.

[0218] Example 85 is configured such that the NNE circuitry is configured to selectively apply a ratio mask to an incoming signal at the front-end receiver to obtain a plurality of components, each of the plurality of components being directed to a medium of example 84 corresponding to a voice class.

[0219] Example 86 is directed to the medium of Example 77, wherein the instructions further cause the computing hardware to apply a complex ratio mask to the controller output signal in the NNE circuit to obtain a plurality of signal components, each of the plurality of signal components corresponding to a voice class or an individual speaker, and combine the plurality of components into an output signal in the NNE circuit, wherein the volume of each component is adjusted relative to at least one other component in accordance with a predetermined user-controlled signal-to-noise ratio.

[0220] Example 87 is directed to the medium of example 86, where the signal components further comprise speech and noise, and the output signal comprises an increased speech volume relative to the noise volume.

[0221] Example 88 is directed to the medium of example 84, wherein the signal components further comprise a user's speech and a plurality of other sound sources, and the output signal comprises the user's speech attenuated relative to the other sound sources.

[0222] Example 89 is directed to the medium of example 84, wherein the instructions further cause the computing hardware to set a volume for each of the different sound sources as a function of a user-controlled parameter.

[0223] Example 90 is directed to the medium of example 77, wherein the signal processing through the first signal processing path excludes the signal processing through the NNE.

[0224] Example 91 is directed to an ear-worn hearing system for improving an incoming audio signal, the system comprising a neural network engine (NNE) circuit configured to improve successive received signal samples and then output a continuous audio signal based on the improved signal samples.

[0225] Example 92 is directed to the hearing system of example 91, wherein the audible signal is generated in about 32 milliseconds or less from receipt of the received signal.

[0226] Example 93 is directed to the hearing system of Example 91, wherein the audible signal is generated in about 10 milliseconds or less from receipt of the received signal.

[0227] Example 94 is directed to the hearing system of Example 91, where the audible signal is generated approximately 10-20 ms, 12-8 ms, 10-6 ms, or 8-3 ms from receipt of the incoming audio signal.

[0228] Example 95 is directed to the hearing system of example 92, wherein the neural network performs at least 1 billion operations per second.

[0229] Example 96 is directed to the hearing system of Example 95, wherein the NNE circuitry is configured to process audio signals with an associated power consumption of about 2 milliwatts or less.

[0230] Example 97 is directed to the hearing system of Example 96, wherein the NNE circuit is formed on a system on chip (SOC) and further comprises a plurality of non-transient executable logic for performing signal processing operations at a plurality of levels of precision.

[0231] Example 98 is directed to the hearing system of example 91, in which the neural network improves the audio signal by estimating a complex ratio mask for each signal sample to obtain desired signal components.

[0232] Example 99 is directed to the hearing system of example 98, wherein the desired signal component is speech.

[0233] Example 100 is directed to the hearing system of example 99, where the desired signal components are one or more recognized speakers.

[0234] Example 101 is directed to the hearing system of example 98, wherein the improved audio signal exhibits reduced background noise, the background noise being user configurable.

[0235] Example 102 is directed to the hearing system of example 101, further comprising a physical control switch accessible on the hearing system for adjusting a background noise level.

[0236] Example 103 is directed to the hearing system of example 101, further comprising a logic control switch accessible via the auxiliary device for adjusting a background noise level.

[0237] Example 104 is directed to an ear-worn hearing system for improving an incoming audio signal, the ear-worn hearing system comprising a neural network engine (NNE) circuit configured to improve the audibility of the received signal and provide an improved continuous output signal, and a control dial for adjusting background noise by operating at least one NNE circuit configuration in response to user input.

[0238] Example 105 is directed to the hearing system of example 104, wherein the control dial comprises an adjustable physical dial.

[0239] Example 106 is directed to the hearing system of example 104, in which the control dial affects a signal-to-noise ratio (SNR) of the continuous output signal.

[0240] Example 107 is directed to the hearing system of example 104, in which the control dial exclusively affects the noise component of the incoming sound.

[0241] Example 108 is an apparatus for improving audibility of an audio signal, comprising: a neural network engine (NNE) circuit configured to receive one or more input audio signals and output one or more intermediate signals, each intermediate signal further comprising an audio signal corresponding to one or more sound sources; and a sound mixer circuit configured to receive the one or more intermediate signals, assign a gain to each intermediate signal, and recombine the one or more intermediate signals to form a new output signal directed to the apparatus, wherein the gains assigned to the one or more intermediate signals are set to achieve a target signal-to-noise ratio (SNR), the SNR being determined as a function of at least one user-specific criterion and at least one user-independent criterion.

[0242] Example 109 is directed to the apparatus of example 108, where the user-specific criteria comprises loudness targets for particular desired signal and noise sound classes, or a desired ratio of loudness between desired sound classes and SNR.

[0243] Example 110 is directed to the apparatus of example 109, in which the volume of a desired voice class is user controlled.

[0244] Example 111 is directed to the apparatus of example 108, in which the number and configuration of intermediate signals output by the neural network are configurable according to user-specific selection criteria.

[0245] Example 112 is directed to the apparatus of example 109, where the user specific criteria further comprises a desired amplification of one or more natural speakers.

[0246] Example 113 is directed to the apparatus of example 109, wherein the user-independent criterion further comprises an estimated SNR of a recently received and processed input audio signal.

[0247] Example 114 is directed to the apparatus of example 109, wherein the user-independent criterion further comprises an estimation error of the neural network.

[0248] Example 115 is directed to the apparatus of example 114, wherein the sound mixer circuit steps recombine one or more intermediate signals to form a new output signal based on the network's prediction error.

[0249] Example 116 is directed to the apparatus of example 108, in which the target SNR is determined as the lower of a user desired SNR or an SNR based on a neural network estimation error.

[0250] In various embodiments, the operations discussed herein may be implemented as hardware (e.g., logic circuitry), software, firmware, or a combination thereof, e.g., with reference to the figures described herein, and may be provided as a computer program product, e.g., comprising a tangible (e.g., non-transitory) machine-readable or computer-readable medium having stored thereon instructions (or software procedures) used to program a computer to perform the processes discussed herein. The machine-readable medium may comprise a storage device, such as those discussed in connection with the drawings of this application.

[0251] Furthermore, such computer-readable media may be downloaded as a computer program product, where the program may be transferred from a remote computer (e.g., a server) to a requesting computer (e.g., a client) by data signals provided on a carrier wave or other propagation medium over a communications link (e.g., a bus, modem, or network connection).

[0252] References herein to "one embodiment" or "an embodiment" mean that at least the embodiment includes a particular feature, structure, and / or characteristic described in connection with the embodiment. The appearances of the phrase "in one embodiment" in various places in the specification may or may not all refer to the same embodiment.

[0253] Also, in the specification and claims, the terms "coupled" and "connected," along with their derivatives, may be used. In some embodiments, "connected" may be used to indicate that two or more elements are in direct physical or electrical contact with each other. "Coupled" may mean that two or more elements are in direct physical or electrical contact. However, "coupled" may also mean that two or more elements still cooperate or interact with each other even if they are not in direct contact with each other.

[0254] Thus, although the embodiments have been described in language specific to structural features and / or methodological acts, it is to be understood that the claimed subject matter may not be limited to the particular features or acts described. Rather, the particular features and acts are disclosed as sample forms of implementing the claimed subject matter.

Claims

1. 1. An apparatus for improving an incoming audio signal, comprising: a controller receiving the incoming signal and providing a controller output signal; a neural network engine (NNE) circuit in communication with the controller, the NNE circuit being activatable by the controller and configured to generate an NNE output signal from the controller output signal; a digital signal processing (DSP) circuit that receives one or more of the controller output signal or the NNE circuit output signal and generates a processed signal therefrom; Equipped with the control unit determines a processing path of a control unit output signal through the DSP circuit or one of the NNE circuits as a function of one or more of predetermined parameters, characteristics of an incoming signal, and feedback of an NNE circuit; Device.

2. The apparatus of claim 1 , wherein the predetermined parameters comprise user-defined and user-independent characteristics.

3. The apparatus of claim 2 , wherein the user-defined characteristics further comprise one or more of a user signal-to-noise ratio (U-SNR) threshold and natural speaker identification information.

4. The apparatus of claim 2 , wherein the user-independent characteristics comprise one or more of an available power level and a system signal-to-noise ratio (S-SNR) threshold.

5. The apparatus of claim 1 , wherein the characteristic of the incoming signal comprises detectable speech or detectable silence.

6. The apparatus of claim 5 , wherein the controller disengages at least one of the DSP or the NNE upon detecting silence, silence being defined by a noise level below a predetermined threshold.

7. The apparatus of claim 1 , wherein the NNE circuit feedback comprises a detected SNR value.

8. The apparatus of claim 1 , wherein the feedback of the NNE circuit comprises an indication of voice detection in the NNE circuit.

9. The apparatus of claim 1 , wherein the control unit is configured to send audio clips to the NNE circuitry and receive feedback from the NNE circuitry.

10. The apparatus of claim 9 , wherein the audio clip defines a portion of the incoming signal and is transmitted intermittently from the control unit.

11. 10. The apparatus of claim 9, wherein the audio clips have a predetermined length and are transmitted at predetermined intervals and at a frequency, the frequency of transmission being determined as a function of a feedback signal of the NNE circuit.

12. The apparatus of claim 1 , wherein the controller determines a processing path for the controller output signal substantially in real time.

13. The apparatus of claim 1 , wherein the controller, DSP, and NNE are integrated on a system-on-chip (SOC).

14. The device of claim 1 , wherein the controller, DSP, and NNE are integrated into a hearing aid configured to fit to be worn on a human ear.

15. The apparatus of claim 1 , further comprising an active noise cancellation (ANC) circuit for processing the controller output signal.

16. 1. A method for improving the quality of an incoming audio signal, comprising: receiving an incoming signal at a controller and providing a controller output signal; initiating a neural network engine (NNE) and processing the controller output signal to generate an NNE output signal and an NNE feedback signal; receiving one or more of the controller output signal and the NNE circuit output signal and initiating a digital signal processing (DSP) circuit to generate a processed signal; Equipped with the control unit determines a processing path of the control unit output signal through one of the DSP circuit or the NNE circuit as a function of one or more of predetermined parameters, characteristics of an incoming signal, and feedback of an NNE circuit; method.

17. The method of claim 16 , wherein the predetermined parameters comprise user-defined and user-independent characteristics.

18. 20. The method of claim 17, wherein the user-defined characteristics further comprise one or more of a user signal-to-noise ratio (U-SNR) threshold and natural speaker identification information.

19. 20. The method of claim 17, wherein the user-independent characteristics further comprise one or more of an available power level and a system signal-to-noise ratio (S-SNR) threshold.

20. The method of claim 16 , wherein the characteristic of the incoming signal comprises a detectable sound or detectable silence.

21. 21. The method of claim 20, further comprising disengaging the DSP and the NNE upon detecting silence in the control unit.

22. 17. The method of claim 16, further comprising detecting an SNR value and providing the SNR value at which the NNE is detected as a feedback signal for the NNE circuit.

23. The method of claim 16 , wherein the NNE feedback signal further comprises an indication of speech detection in the NNE.

24. The method of claim 16 , further comprising sending an audio clip from the control unit to the NNE prior to receiving the NNE feedback signal.

25. 25. The method of claim 24, wherein the audio clip defines a portion of the incoming signal and is transmitted intermittently.

26. 25. The method of claim 24, wherein the audio clips have a predetermined length and are transmitted at predetermined intervals and at a frequency, the frequency of transmission being determined as a function of a feedback signal of the NNE circuit.

27. The method of claim 16 , further comprising determining a processing path in real time at the control unit.

28. The method of claim 16 , further comprising integrating the controller, DSP, and NNE onto a system on a chip (SOC).

29. 17. The method of claim 16, further comprising integrating the controller, DSP, and NNE into a hearing aid configured to fit a human ear.

30. 17. The method of claim 16, further comprising engaging an active noise cancellation (ANC) circuit when processing the controller output signal through the NNE circuit.

31. When executed by computing hardware including a processor circuit coupled to a memory circuit, the computing hardware receiving an incoming signal at a controller and providing a controller output signal; invoking a neural network engine (NNE) to process the controller output signal to generate an NNE output signal and an NNE feedback signal; receiving one or more of the controller output signal and the NNE circuit output signal and initiating a digital signal processing (DSP) circuit to generate a processed signal; At least one non-transitory machine-readable medium comprising instructions for performing: the control unit determines a processing path of the control unit output signal through one of the DSP circuit or the NNE circuit as a function of one or more of predetermined parameters, characteristics of an incoming signal, and feedback of an NNE circuit; Non-transitory machine-readable media.

32. 32. The medium of claim 31, wherein the predetermined parameters comprise user-defined characteristics and user-independent characteristics.

33. 33. The medium of claim 32, wherein the user-defined characteristics further comprise one or more of a user signal-to-noise ratio (U-SNR) threshold and natural speaker identification information.

34. 33. The medium of claim 32, wherein the user-independent characteristics further comprise one or more of an available power level and a system signal-to-noise ratio (S-SNR) threshold.

35. 32. The medium of claim 31, wherein the characteristic of the incoming signal comprises detectable speech or detectable silence.

36. 36. The medium of claim 35, wherein the instructions further cause the computing hardware to disengage the DSP and the NNE upon detecting silence in the control unit.

37. 32. The medium of claim 31, wherein the instructions further cause the computing hardware to detect an SNR value and the NNE and provide the detected SNR value as a feedback signal for the NNE circuit.

38. 32. The medium of claim 31, wherein the NNE feedback signal further comprises an indication of voice detection in the NNE.

39. 32. The medium of claim 31, wherein the instructions further cause the computing hardware to send an audio clip from the control unit to the NNE before receiving the NNE feedback signal.

40. 40. The medium of claim 39, wherein the audio clip defines a portion of an incoming signal and is transmitted intermittently.

41. 40. The medium of claim 39, wherein the audio clips have a predetermined length and are transmitted at predetermined intervals and at a frequency, the frequency of transmission being determined as a function of a feedback signal of the NNE circuit.

42. 32. The medium of claim 31, wherein the instructions further cause the computing hardware to determine a processing path in real time at the controller.

43. 32. The medium of claim 31, wherein the controller, DSP, and NNE are integrated into a hearing aid configured to fit the human ear.

44. 1. A hearing system for improving an incoming audio signal, comprising: a front-end receiver for receiving one or more incoming audio signals, at least one of the incoming audio signals having a plurality of signal components, each signal component corresponding to a respective signal source; a controller in communication with the front-end receiver, the controller receiving an input signal from the front-end receiver and providing a controller output signal, the controller output signal selectively provided to at least one of a first or a second signal processing path; a neural network engine (NNE) circuit in communication with the controller to define a portion of a first signal processing path, the NNE circuit being activateable by the controller and configured to generate an NNE output signal from the controller output signal; a digital signal processing (DSP) circuit forming part of the first and second signal processing paths, the DSP circuit receiving one or more of a controller output signal or the NNE circuit output signal and generating a processed signal therefrom; Equipped with The front-end receiver, the control unit, the NNE circuit, and the DSP circuit are formed on an integrated circuit (IC). Hearing system.

45. 45. The hearing system of claim 44, further comprising a back-end receiver that receives an output signal from the DSP and forms an audible signal.

46. 46. ​​The hearing system of claim 45, defining one of a hearing aid, headphones, or faceworn glasses, wherein the audible signal is formed less than 32 milliseconds after receiving an incoming signal.

47. 45. The hearing system of claim 44, wherein the IC comprises a system on a chip (SOC).

48. 48. The hearing system of claim 47, further comprising a housing that receives the SOC and a power source.

49. 45. The hearing system of claim 44, wherein the control unit determines the processing path of the control unit output signal as a function of feedback of an NNE circuit.

50. 45. The hearing system of claim 44, wherein the control unit determines a processing path of the control unit output signal as a function of one or more of predetermined parameters, characteristics of the incoming signal, and feedback of an NNE circuit.

51. 45. The hearing system of claim 44, further comprising a wireless communication system.

52. 45. The hearing system of claim 44, wherein the NNE circuit adjusts the relative volume of components of the incoming signal and the DSP circuit applies a frequency and time varying gain to the received signal.

53. 53. The hearing system of claim 52, wherein the components of the incoming signal further comprise at least speech and noise, and wherein the speech volume is increased relative to the noise volume.

54. 45. The hearing system of claim 44, wherein the front-end receiver processes an incoming signal to provide an input signal to the control unit, the incoming signal including one or more of a speech component and a noise component.

55. 53. The hearing system of claim 52, wherein the NNE circuit selectively applies a ratio mask to the incoming signal of the front-end receiver to obtain a plurality of components, each of the plurality of components corresponding to a sound class.

56. 45. The hearing system of claim 44, wherein the NNE circuit is configured to selectively apply a complex ratio mask to the control unit output signal to obtain a plurality of signal components, each of the plurality of signal components corresponding to a voice class or an individual speaker, and the NNE circuit is further configured to combine the plurality of components into an output signal, the volume of each component being adjusted relative to at least one other component in accordance with a predetermined user-controlled signal-to-noise ratio.

57. 57. The hearing system of claim 56, wherein the signal components further comprise speech and noise, and the output signal comprises an increased speech volume relative to a noise volume.

58. 57. The hearing system of claim 56, wherein the signal components further comprise a user's speech and a plurality of other sound sources, and the output signal comprises a user's speech reduced relative to the other sound sources.

59. 57. The hearing system of claim 56, wherein the NNE circuitry is further configured to set the volume of each of the different sound sources as a function of a user-controlled parameter.

60. 45. The hearing system of claim 44, wherein the second signal processing path excludes signal processing through the NNE.

61. 45. The hearing system of claim 44, wherein the NNE circuitry is further configured to perform one or more of the DSP functions.

62. 1. A method for improving the quality of an incoming audio signal, comprising: receiving one or more incoming audio signals at a front-end receiver, at least one of the incoming audio signals having a plurality of signal components, each signal component corresponding to a respective signal source; receiving an input signal from the front-end receiver at a controller and providing a controller output signal, the controller selectively providing the output signal to at least one of a first or a second signal processing path; generating an NNE output signal from the controller output signal in a neural network engine (NNE) circuit activatable by the controller, the NNE defining at least a portion of the first signal processing path; generating a processed signal from the controller output signal or the NNE circuit output signal in a digital signal processing (DSP) circuit, the DSP defining at least a portion of the first and second signal processing paths; Equipped with The front-end receiver, the control unit, the NNE circuit, and the DSP circuit are formed on an integrated circuit (IC). method.

63. 63. The method of claim 62, further comprising forming an output signal from the processed signal at a back-end receive portion.

64. 64. The method of claim 63, further comprising forming the output signal in less than 32 milliseconds after receiving the incoming signal.

65. 64. The method of claim 63, wherein the hearing system defines one of a hearing aid, headphones, or faceworn glasses.

66. 63. The method of claim 62, wherein the IC comprises a system on a chip (SOC).

67. 67. The method of claim 66, further comprising a housing that receives the SOC and a power source.

68. 63. The method of claim 62, further comprising determining the processing path of the controller output signal as a function of NNE circuit feedback.

69. 63. The method of claim 62, further comprising determining the processing path of the controller output signal as a function of one or more of predetermined parameters, characteristics of an incoming signal, and feedback of an NNE circuit.

70. 63. The method of claim 62, further comprising processing an incoming signal having one or more of a speech component and a noise component at the front-end receiver to provide an input signal to the control unit.

71. 71. The method of claim 70, wherein the NNE circuit selectively applies a ratio mask to the incoming signal at the front-end receiver to obtain a plurality of components, each of the plurality of components corresponding to a voice class.

72. 63. The method system of claim 62, further comprising: applying a complex ratio mask to the control unit output signal in the NNE circuit to obtain a plurality of signal components, each of the plurality of signal components corresponding to a voice class or an individual speaker; and combining the plurality of components into an output signal in the NNE circuit, wherein the volume of each component is adjusted relative to at least one other component according to a predetermined user-controlled signal-to-noise ratio.

73. 73. The method of claim 72, wherein the signal components further comprise speech and noise, and the output signal comprises an increased speech volume relative to a noise volume.

74. 73. The method of claim 72, wherein the signal components further comprise a user's speech and a plurality of other sound sources, and the output signal comprises a user's speech reduced relative to the other sound sources.

75. 73. The method of claim 72, wherein the NNE circuitry is further configured to set the volume of each of the different sound sources as a function of a user-controlled parameter.

76. 63. The method of claim 62, wherein signal processing through the first signal processing path excludes signal processing through the NNE.

77. When executed by computing hardware including a processor circuit coupled to a memory circuit, the computing hardware receiving one or more incoming audio signals at a front-end receiver, at least one of the incoming audio signals having a plurality of signal components, each signal component corresponding to a respective signal source; receiving an input signal from the front-end receiver and providing a controller output signal, the controller selectively providing the output signal to at least one of a first signal processing path or a second signal processing path; generating, in a neural network engine (NNE) circuit activatable by the controller, an NNE output signal from the controller output signal, the NNE defining at least a portion of the first signal processing path; generating, in a digital signal processing (DSP) circuit, a processed signal from the controller output signal or the NNE circuit output signal, the DSP defining at least a portion of first and second signal processing paths; At least one non-transitory machine-readable medium comprising instructions for performing: The front-end receiver, the control unit, the NNE circuit, and the DSP circuit are formed on an integrated circuit (IC). At least one non-transitory machine-readable medium.

78. 78. The medium of claim 77, wherein the instructions further cause the computing hardware to form an output signal from the processed signal at a back-end receive portion.

79. 80. The medium of claim 78, wherein the instructions further cause the computing hardware to form the output signal in less than 32 milliseconds after receiving the incoming signal.

80. 80. The media of claim 78, wherein the hearing system defines one of a hearing aid, headphones, or facework glasses.

81. 80. The medium of claim 77, wherein the IC comprises a system on a chip (SOC).

82. 78. The medium of claim 77, wherein the instructions further cause the computing hardware to determine a processing path for the controller output signal as a function of NNE circuit feedback.

83. 78. The medium of claim 77, wherein the instructions further cause the computing hardware to determine a processing path of the controller output signal as a function of one or more of predetermined parameters, characteristics of an incoming signal, and feedback of an NNE circuit.

84. 80. The medium of claim 77, wherein the instructions further cause the computing hardware to process an incoming signal having one or more of a speech component and a noise component at the front end receiver and provide an incoming signal to the controller.

85. 85. The medium of claim 84, wherein the NNE circuitry is configured to selectively apply a ratio mask to an incoming signal at the front-end receiver to obtain a plurality of components, each of the plurality of components corresponding to a voice class.

86. 78. The medium of claim 77, wherein the instructions further cause the computing hardware to apply a complex ratio mask to the controller output signal in the NNE circuit to obtain a plurality of signal components, each of the plurality of signal components corresponding to a voice class or an individual speaker, and combine the plurality of components into an output signal in the NNE circuit, the volume of each component being adjusted relative to at least one other component in accordance with a predetermined user-controlled signal-to-noise ratio.

87. 87. The medium of claim 86, wherein the signal components further comprise speech and noise, and the output signal comprises an increased speech volume relative to a noise volume.

88. 85. The medium of claim 84, wherein the signal components further comprise a user's speech and a plurality of other sound sources, and the output signal comprises a user's speech reduced relative to the other sound sources.

89. 85. The medium of claim 84, wherein the instructions further cause the computing hardware to set a volume for each of the different sound sources as a function of a user-controlled parameter.

90. 80. The medium of claim 77, wherein the signal processing through the first signal processing path excludes signal processing through the NNE.

91. 1. An ear-worn hearing system for improving an incoming audio signal, comprising: A hearing system comprising a neural network engine (NNE) circuit configured to improve successively received signal samples and then output a successive audible signal based on the improved signal samples.

92. 92. The hearing system of claim 91, wherein the audible signal is generated within about 32 milliseconds or less of receipt of the received signal.

93. 92. The hearing system of claim 91, wherein the audible signal is generated within about 10 milliseconds or less of receipt of the received signal.

94. 92. The hearing system of claim 91, wherein the audible signal is generated approximately 10-20 ms, 12-8 ms, 10-6 ms, or 8-3 ms from receipt of the incoming audio signal.

95. 93. The hearing system of claim 92, wherein the neural network performs at least 1 billion operations per second.

96. 96. The hearing system of claim 95, wherein the NNE circuitry is configured to process audio signals with an associated power consumption of about 2 milliwatts or less.

97. 97. The hearing system of claim 96, wherein the NNE circuit is formed on a system on chip (SOC) and further comprises a plurality of non-transient executable logic for performing signal processing operations at a plurality of levels of precision.

98. 92. The hearing system of claim 91, wherein the neural network improves the audio signal by estimating a complex ratio mask for each signal sample to obtain desired signal components.

99. 99. The hearing system of claim 98, wherein the desired signal component is speech.

100. 100. The hearing system of claim 99, wherein the desired signal components are one or more recognized speakers.

101. 99. The hearing system of claim 98, wherein the improved audio signal exhibits reduced background noise, said background noise being user configurable.

102. 102. The hearing system of claim 101, further comprising a physical control switch accessible on the hearing system for adjusting a background noise level.

103. 102. The hearing system of claim 101, further comprising a logically controlled switch accessible via the auxiliary device for adjusting the background noise level.

104. 1. An ear-worn hearing system for improving an incoming audio signal, comprising: a neural network engine (NNE) circuit configured to improve the audibility of a received signal and to provide an improved continuous output signal; a control dial for adjusting the background noise by operating at least one NNE circuitry in response to user input; An ear-worn hearing system comprising:

105. 105. The hearing system of claim 104, wherein the control dial comprises an adjustable physical dial.

106. 105. The hearing system of claim 104, wherein the control dial affects a signal-to-noise ratio (SNR) of the continuous output signal.

107. 105. The hearing system of claim 104, wherein the control dial exclusively affects the noise component of the incoming sound.

108. 1. An apparatus for improving the audibility of an audio signal, comprising: a neural network engine (NNE) circuit configured to receive one or more input audio signals and output one or more intermediate signals, each intermediate signal further comprising an audio signal corresponding to one or more audio sources; a sound mixer circuit configured to receive the one or more intermediate signals, assign a gain to each intermediate signal, and recombine the one or more intermediate signals to form a new output signal; Equipped with a gain assigned to the one or more intermediate signals is set to achieve a target signal-to-noise ratio (SNR), the SNR being determined as a function of at least one user-specific criterion and at least one user-independent criterion; Device.

109. 109. The apparatus of claim 108, wherein the user-specific criteria comprises loudness targets for particular desired signal and noise sound classes or a desired ratio of loudness between desired sound classes and SNR.

110. 110. The apparatus of claim 109, wherein the volume of the desired voice class is user controlled.

111. 109. The apparatus of claim 108, wherein the number and composition of the intermediate signals output by the neural network are configurable according to user-specific selection criteria.

112. 110. The apparatus of claim 109, wherein the user specific criteria further comprises a desired amplification of one or more natural speakers.

113. 110. The apparatus of claim 109, wherein the user-independent criterion further comprises the estimated SNR of a recently received and processed input speech signal.

114. 110. The apparatus of claim 109, wherein the user-independent criteria further comprises the estimation error of the neural network.

115. 115. The apparatus of claim 114, wherein the sound mixer circuit step recombines the one or more intermediate signals to form a new output signal based on a prediction error of the network.

116. 109. The apparatus of claim 108, wherein the target SNR is determined as the lower of a desired SNR of the user or the SNR based on the estimation error of the neural network.

Images