Ear-worn device with parallel neural network

Abstract

An apparatus (e.g., an ear-worn device such as a hearing aid) may include a neural network circuit and a control circuit. The neural network circuit may be configured to implement a neural network system including at least a first neural network and a second neural network operating in parallel. The control circuit may be configured to control the neural network system to receive a first input signal, process the first input signal using a first neural network to produce a first output, and process the first input signal using a second neural network to produce a second output, combine the first output and the second output, and output the combined output. The one or more states of the first neural network are reset, and the one or more states of the second neural network are reset at a different time than the one or more states of the first neural network are reset.

Description

Technical Field

[0001] This disclosure also relates to an apparatus that uses a neural network to process signals, such as an ear-worn device. Background Technology

[0002] Hearing aids and other ear-worn devices are used to help people with hearing difficulties hear better. Typically, hearing aids amplify the received sound. Some hearing aids attempt to remove ambient noise from the incoming sound. Summary of the Invention

[0003] A recurrent neural network (RNN) is a neural network in which the result of processing at one time step can affect processing at subsequent time steps. Therefore, an RNN has a "state" that can persist from one time step to another, representing contextual information derived from the analysis of previous inputs. It should be understood that other types of neural networks can also be stateful (i.e., have and use states), and can also utilize the methods described herein.

[0004] An RNN may include an input layer, one or more RNN layers, and an output layer. Each RNN layer may include an input node, an output node, and a state. In some embodiments, there may be one state, which may be referred to as a "hidden state." In some embodiments, such as Long Short-Term Memory (LSTM) type RNNs, there may be two states, which may be referred to as "hidden states" and "cell states." At each time step, the state from the previous time step may be concatenated with the input of the current time step.

[0005] Recurrent neural networks (RNNs) and other stateful neural networks can have drawbacks. Over time, the states in a neural network may drift and acquire values ​​that were never reached during training. Therefore, the performance of the neural network may degrade over time. Resetting some states of the neural network (i.e., one or more states) can prevent long-term performance degradation. However, doing so can have a drawback: the performance of the neural network may immediately decrease after resetting the states because the network is operating without contextual information gained from processing previous inputs. Therefore, while resetting the states can address the problem of states drifting to values ​​never reached during training, it introduces another problem.

[0006] The inventors discovered that to address these two problems with stateful neural networks, multiple neural networks can operate in parallel on the same input, but their states may be reset at time points offset from each other. In this way, both neural networks can avoid long-term degradation, and at any given time point, at least one neural network may have already established state information helpful in calculating its output. The outputs of the neural networks can be combined, allowing the combined outputs of the parallel neural networks to more fully utilize the predictions of the neural network whose state is at an optimal processing point. Therefore, the neural network system can reduce its dependence on neural networks whose states are not at an optimal point. In other words, the output of one neural network may have a larger weight than the output of another neural network. For example, the weight of the most recently reset neural network may be lower than the weight of another neural network. Parallel neural networks may have different architectures and different weights, or they may have the same architecture and weights. Due to the staggered state reset times, even parallel neural networks with the same architecture and weights may have different states.

[0007] Generally, one or more states in a neural network may be reset. In some embodiments, all states in the neural network may be reset. In some embodiments, only certain types of states in the neural network may be reset. For example, in an LSTM neural network, in some embodiments, only the unit states may be reset without resetting the hidden states. In some embodiments, only certain states of one or more layers of the neural network may be reset, and states from different layers or groups of layers may be reset at different times. As a specific example, all states of one layer (e.g., layer 1) may be reset at one time, then all states of different layers (e.g., layer 2) may be reset at another time, and so on. Therefore, as described herein, resetting one or more states in a neural network can mean resetting all states in the neural network, resetting only certain states in the neural network (e.g., a certain type or more types of states), resetting all states of one or more layers in the neural network (e.g., one layer), and / or resetting certain states of one or more layers in the neural network (e.g., one layer) (e.g., a certain type or more types of states).

[0008] As described in this article, resetting a state may refer to actively changing the value in the state to 0, or actively changing the value in the state to a value other than zero. Furthermore, as described in this article, resetting a state may refer to actively changing the value in the state immediately or over a finite period of time. In the latter case, the reset may be smooth, causing the value in the state to decay to zero or a different value over time.

[0009] Traditional ear-worn devices (such as hearing aids, cochlear implants, headphones, etc.) receive an input acoustic signal, amplify the signal, and output the signal to the wearer. Hearing aid performance can be improved by utilizing neural networks, for example, to reduce noise in the audio signal. In some embodiments, parallel RNNs, such as those described herein, can be implemented in ear-worn devices such as hearing aids. The input to the neural network can be based on audio received by one or more microphones on the hearing aid, and the outputs of the parallel neural network can be combined in a weighted manner to produce an output that is played back to the wearer by the hearing aid's receiver. However, while some embodiments of the technology described herein may relate to hearing aids or other ear-worn devices (such as cochlear implants and headphones), this should be understood as non-limiting, and it should be understood that any device implementing a stateful neural network can also utilize this technology. Attached Figure Description

[0010] Various aspects and embodiments of this application will be described with reference to the following accompanying drawings. It should be understood that these drawings are not necessarily drawn to scale. Items appearing in multiple drawings are denoted by the same reference numerals in all the drawings in which they appear.

[0011] Figure 1 These are schematic diagrams of neural network systems according to certain embodiments described herein;

[0012] Figure 2 The timeline and weighting scheme of two neural networks according to certain embodiments described herein are shown;

[0013] Figure 3 The timeline and weighting scheme of two neural networks according to certain embodiments described herein are shown;

[0014] Figure 4 Schemes for processing data fragments according to certain embodiments described herein are illustrated;

[0015] Figure 5 Graphs of example weighting schemes and reset schemes according to certain embodiments described herein are shown;

[0016] Figure 6 Graphs of example weighting schemes and reset schemes according to certain embodiments described herein are shown;

[0017] Figure 7 A block diagram of an ear-worn device according to certain embodiments described herein is shown;

[0018] Figure 8 A perspective view of a hearing aid according to certain embodiments described herein is shown;

[0019] Figure 9The process of using multiple parallel neural networks to process data is illustrated according to certain embodiments described herein;

[0020] Figure 10 The process of processing data using multiple parallel neural networks according to certain embodiments described herein is illustrated; and

[0021] Figure 11 Weighting schemes for a first neural network and a second neural network according to certain embodiments described herein are illustrated. Detailed Implementation

[0022] The aspects and embodiments described above, as well as additional aspects and embodiments, will be further described below. These aspects and / or embodiments may be used alone, together, or in any combination of two or more, as this disclosure is not limited in this respect.

[0023] Figure 1 This is a schematic diagram of a neural network (NN) system 100 according to certain embodiments described herein. The neural network system 100 may be implemented, for example, by neural network circuitry (e.g., neural network circuitry 720), and may be part of a device such as a hearing aid or other ear-worn device. Figure 1 As shown, the neural network system 100 includes two neural networks 104 and 106 for parallel processing and a combiner 110. While two neural networks are shown, the neural network system 100 is not limited to this and may include more than two neural networks, such as three, four, or more. Neural networks 104 and 106 can both be recurrent neural networks or another type of stateful neural network. Neural networks 104 and 106 can be trained to reduce noise in audio signals. Noise reduction may include reducing background noise and / or interfering speech (e.g., speech from another conversation). Typically, in some embodiments, each of neural networks 104 and 106 may have a different architecture and different weights. Alternatively, each of neural networks 104 and 106 may have the same architecture and the same weights. Therefore, Figure 1 The neural networks 104 and 106 in the examples can be the same, that is, they can have the same architecture and the same set of weights.

[0024] Figure 1 The diagram shows that each of neural networks 104 and 106 includes n layers. Although Figure 1 At least 5 layers in each of neural networks 104 and 106 are shown, but it should be understood that fewer than 5 layers (e.g., 4, 3, 2 or 1 layers), 5 layers or more can be used.

[0025] As shown in the figure, neural network 106 can be configured to receive input 102. For example, in the context of a hearing aid, input 102 can be an audio signal. In some embodiments, input 102 can be processed before being input to neural network system 100. For example, in the context of a hearing aid, the audio signal can undergo analog processing (e.g., pre-amplification and / or filtering) and digital processing (e.g., wind noise reduction, beamforming, anti-feedback, Fourier transform, and calibration). The audio signal can also be decomposed into segments (e.g., as referenced). Figure 4 (As described above), input 102 can be a segment of an input signal. Neural network 106 can be configured to process input 102 sequentially through each of its layers. As shown, the k-th layer of both neural network 104 and neural network 106 can be configured to receive input from the (k-1)-th layer of neural network 106. Neural network 104 can be configured (using its k-th layer) to process this input to produce output 112, and neural network 106 can be configured (using its k-th layer) to process this input to produce output 114. Combiner 110 can be configured to combine outputs 112 and 114 to produce output 108, which can then be fed into the (k+1)-th layer of neural network 106. Neural network 106 can be configured to complete the processing with its remaining n layers and produce output 116.

[0026] Typically, one or more layers of neural network 104 can be configured to operate in parallel with one or more layers of neural network 106. When each layer is configured to receive and process the same input, a layer of one neural network can be considered configured to operate in parallel with a layer of another neural network. For example, in Figure 1 In this example, the k-th layer of neural network 104 and the k-th layer of neural network 106 are configured to receive and process the same input (output from the (k-1)-th layer in neural network 106). Therefore, the k-th layer in neural network 104 and the k-th layer in neural network 106 can be considered to be configured to operate in parallel. Generally, when at least one layer in each neural network is configured to receive and process the same input, the two neural networks can be considered to be configured to operate in parallel. Therefore, in Figure 1In this context, neural network 104 can be considered to be configured to operate in parallel with neural network 106. It should be understood that any layer can be configured to operate in parallel. For example, the first layer of neural networks 104 and 106 can be configured to operate in parallel, and combiner 110 can be configured to combine the outputs from the first layers of the two neural networks. It should also be understood that more than one layer can be configured to operate in parallel. For example, the first and second layers of neural networks 104 and 106 can be configured to operate in parallel. In some embodiments, when multiple layers are configured to operate in parallel, only the output from the last parallel layer can be combined. For example, if the first and second layers are configured to operate in parallel, the output from the first layer of neural network 104 will enter the second layer in neural network 104, the output from the first layer of neural network 106 will enter the second layer in neural network 106, and the outputs of the two second layers will be combined by combiner 110. Alternatively, the outputs of each pair of parallel layers can be combined. For example, if layers 1 and 2 are configured for parallel operation, the outputs of the two layers 1 will be combined, and this combined output will be fed into the two layers 2, and the outputs of the two layers 2 will also be combined. It should also be understood that the layers configured for parallel operation may change. For example, as will be referred to... Figure 3 The first layer can be configured for parallel operation, then the second layer can be configured for parallel operation, and so on.

[0027] As can be understood from the above, a complete neural network (e.g., neural network 106) can be configured to operate (i.e., configured to process input 102), while fewer than a complete neural network (e.g., fewer than all layers of neural network 104, such as one layer in neural network 104) can be configured to operate in parallel. If one or more layers receive input 102 itself or the result of another layer processing that input, then that one or more layers can be considered configured to process input 102. In some embodiments, when two neural networks operate in parallel, one neural network can be configured to have more operating layers than the other neural network. Figure 3Example schemes of parallel operation of layers in different neural networks are illustrated. In extreme cases, all layers of neural networks 104 and 106 (e.g., a complete neural network) can operate in parallel. In these embodiments, combiner 110 may combine only the final output of each neural network (e.g., the output from the nth layer) to produce output 116 of neural network system 100, or the outputs of each pair of corresponding layers may be combined by combiner 110 before being fed to the next layer in each neural network. Regarding the term "complete" neural network, a "complete" neural network can be considered to include n layers if at some point each of the n layers is configured to operate (e.g., receive and process input). For example, if neural network system 100 is allowed to operate for a long time, and n different layers of neural network 104 operate during this period, then a complete neural network 104 can be said to include n layers. As another example, if the layers of neural network 104 are configured to operate according to a specific sequence or period, and n layers operate within one such period, then a complete neural network 104 can be said to include n layers. As another example, if neural networks 104 and 106 are known to be the same, then the layers in neural network 104 can be the same as the layers in neural network 106.

[0028] In some embodiments, the output 116 of the neural network system 100 can be an output used by the neural network system 100 to reduce noise components of a signal to obtain an enhanced output. For example, the neural network system output 116 can be a mask that, when multiplied by the input 102, leaves only the speech portion of the audio signal. In some embodiments, the neural network system output 116 can be an enhanced output. For example, in the context of a hearing aid, the neural network system output 116 can be the speech portion of the input 102.

[0029] Combiner 110 can combine the outputs 112 and 114 of neural networks 104 and 106 in any suitable manner. In some embodiments, the outputs of neural networks 104 and 106 can be combined according to a weighting scheme. For example, combiner 110 can multiply output 112 by a first weight, multiply output 114 by a second weight (the second weight may be different from the first weight), and add the two products together. Example weighting schemes are further described herein.

[0030] Each of neural networks 104 and 106 can be a separate neural network. As mentioned above, each of neural networks 104 and 106 can have a different architecture and different weights. Alternatively, each of neural networks 104 and 106 can have the same architecture and the same weights. Therefore, each of neural networks 104 and 106 can have the same architecture and the same set of weights. The state of each of neural networks 104 and 106 can change as the neural network runs and can differ from each other, for example, due to staggered reset times. In some embodiments, using the same weights for each neural network can save memory (e.g., in devices such as hearing aids that implement neural networks).

[0031] As described herein, the output of a neural network should be understood to include the output of any layer within the neural network. Therefore, the neural network output described herein can be the output of the first layer, an intermediate layer, the last layer, or any combination of multiple layers. Similarly, processing input through a neural network should be understood to include processing input through any layer of the neural network.

[0032] When implemented by a neural network circuit (e.g., neural network circuit 720), in some embodiments, the state of the neural network layers configured to operate may be stored in the memory of the neural network circuit, while the state of layers not configured to operate may not be stored in memory. For example, although neural network 104 in Figure 1 The diagram is shown as having n layers, but since only the k-th layer of the neural network 104 is configured to operate, only the state of the k-th layer can actually be stored in memory.

[0033] Figure 2 The timeline and weighting scheme of two neural networks according to certain embodiments described herein are shown. Figure 2 A first timeline 202 for a first neural network (e.g., neural network 104) and a second timeline 204 for a second neural network (e.g., neural network 106) are shown. Although Figure 2 The timeline for two neural networks is shown, but it should be understood that more than one neural network may operate in parallel. Certain states (i.e., one or more states) in each neural network can be reset after a period of time, occurring at reset 208. For example, in a neural network that only includes hidden states, in some embodiments, all hidden states can be reset. As another example, in a neural network that includes both hidden states and unit states, in some embodiments, all unit states can be reset, but the hidden states are not. As yet another example, the states of one layer of a neural network can be reset at each reset 208, while states from different layers can be reset at different reset 208s. Figure 2In the example, each neural network resets one or more of its states after 10 seconds, as shown by the time between reset 208 in each timetable 202 and 204. It should be understood that other times between resets can be used. As a non-limiting example, the neural network can be reset after 1 second, 5 seconds, 20 seconds, 30 seconds, 45 seconds, 60 seconds, 2 minutes, 5 minutes, 10 minutes, 15 minutes, 30 minutes, 45 minutes, 1 hour, 2 hours, 3 hours, or any other suitable time. For example, the time between resets may be between, or approximately equal to, one of the following endpoints: 1 second and 10 seconds; 1 second and 20 seconds; 1 second and 30 seconds; 1 second and 45 seconds; 1 second and 60 seconds; 1 second and 2 minutes; 1 second and 5 minutes; 1 second and 10 minutes; 1 second and 15 minutes; 1 second and 30 minutes; 1 second and 45 minutes; 1 second and 1 hour; 1 second and 2 hours; 1 second and 3 hours; 5 seconds and 10 seconds; 5 seconds and 20 seconds; 5 seconds and 30 seconds; 5 seconds and 45 seconds; 5 seconds and 60 seconds; 5 seconds and 2 minutes; 5 seconds The following are examples of non-limiting examples: 5 minutes; 5 seconds and 10 minutes; 5 seconds and 15 minutes; 5 seconds and 30 minutes; 5 seconds and 45 minutes; 5 seconds and 1 hour; 5 seconds and 2 hours; 5 seconds and 3 hours; 10 seconds and 20 seconds; 10 seconds and 30 seconds; 10 seconds and 45 seconds; 10 seconds and 60 seconds; 10 seconds and 2 minutes; 10 seconds and 5 minutes; 10 seconds and 10 minutes; 10 seconds and 15 minutes; 10 seconds and 30 minutes; 10 seconds and 45 minutes; 10 seconds and 1 hour; 10 seconds and 2 hours; and 10 seconds and 3 hours. The time between resetting 208 may be related to the length of time typically required for the neural network's performance to drop below a performance threshold due to the aforementioned state drift.

[0034] exist Figure 2 In the example, the resets 208 of the two neural networks do not occur simultaneously; instead, the reset 208 of the first timetable 202 is offset from the reset 208 of the second timetable 204. By offsetting the resets 208 of the first and second neural networks, one neural network can be run while the other is reset and warmed up. Specifically, as... Figure 2As further illustrated, each neural network has a warm-up period 210, which is a short period after reset 208 during which the neural network is on but its weights are at 0%. The warm-up period 210 can be helpful because for a brief period after state reset 208, the neural network may operate without contextual information gained from processing previous inputs, potentially leading to performance degradation. The warm-up period 210 provides the neural network with the opportunity to gain the benefit of contextual information before it is used to produce output (i.e., before its weights increase above 0). Therefore, staggered resets can enable the warm-up period 210. Staggered resets also allow for diversity in neural network processing, permitting different states. However, in some embodiments, the resets 208 of two neural networks may occur simultaneously.

[0035] The outputs of a neural network can be weighted before being combined. As mentioned above, the output of a neural network should be understood to include the outputs of any layer within the network. Therefore, refer to... Figure 2 The output of the described neural network can be the output of the first layer, a middle layer, the last layer, or any combination of multiple layers. In extreme cases, the output may be the output of the entire neural network. Furthermore, the reference to weighting applied to a neural network should be understood as an average weight applied to the output of the neural network, with the meaning just provided. Figure 2 An example weighting scheme 206 is shown, illustrating the weighting ratios (ranging from 0 at the bottom to 1 at the top) applied as a function of time to the first neural network on the left, and the weighting ratios (ranging from 1 at the bottom to 0 at the top) applied to the second neural network on the right. It should be understood that in Figure 2 In the example, at any given time, the weight of each neural network is 1 minus the weight applied to the other neural network. Therefore, when the weight of the first neural network is 1, the weight of the second neural network is 0, and vice versa. Between the minimum and maximum values, the weighting scheme 206 can follow a linear function, allowing the weights to take values ​​between 0 and 1 (e.g., 0.5).

[0036] Typically, the weighting scheme can be a linear piecewise function, a smoothing function (such as a sine function), or another function that transfers weights between neural networks. Therefore, the weighting scheme can be a dynamic weighting scheme, such that the first weight at the output of the first neural network and the second weight at the output of the second neural network change over a period of time. Such a dynamic weighting scheme can include variable weights corresponding to different reset sequences, for example, first and second variable weights that can be transferred from values ​​between 0 and 1.

[0037] In some embodiments, the controller (e.g., implemented by control circuitry 726) can dynamically compute the weights of each neural network. In such embodiments, using a linear function as the weighting scheme may help reduce the computational complexity of the controller. For example, the controller may only need to compute the weights of one neural network, and then obtain the weights of another neural network by subtracting the first weight from 1. However, in some embodiments, each neural network may have an independent scheme, and the controller can compute the weights of each neural network independently. In some embodiments, the weights may be stored in memory, and the controller may not need to dynamically compute the weights.

[0038] Some of the weighting schemes described in this article (e.g., in) Figure 2 , Figure 3 , Figure 5 and Figure 6 The weights used may depend at least in part on the length of time since the last reset. The weights may transition from low to high after a reset (in some embodiments, after a shutdown period and / or warm-up period), and then from high to low before the next reset.

[0039] In some embodiments, weights can be determined based on a confidence metric associated with the output of each neural network, such that when two neural networks are combined, the neural network with higher quality / confidence is weighted more heavily. For example, a confidence metric might be inter-frame coherence, as the coherence of neural network outputs is often closely related to the confidence of the neural network. A neural network with insufficient confidence may have very different outputs between frames.

[0040] In some embodiments, the reset may not occur at fixed time intervals, but rather within a period during which the neural network can reset with minimal impact. For example, in some embodiments, one of the neural networks may switch and reset during a silent period. In some embodiments, the silent period can be used for the reset, and only one neural network may be allowed to be used. In other words, the output of the neural network can be used even shortly after the reset, because the silent period may continue after the reset, and the neural network is unlikely to introduce artifacts during the silent period. The silent period may exclude neither speech nor noise above a threshold (e.g., 20 dB, 40 dB, or 60 dB).

[0041] In some embodiments, a reset can occur only when certain conditions are met. By monitoring the quality of the neural network output using metrics, the controller can determine whether a reset should occur. Once a metric (e.g., inter-frame coherence) exceeds a certain threshold, a reset can be initiated, starting with the initialization of the second neural network. Once the second neural network has completed its warm-up period, weights can begin to shift to incorporate more of the second neural network's output into the combined output.

[0042] Figure 3 The timeline and weighting scheme of two neural networks according to certain embodiments described herein are illustrated. As described above, a complete neural network (e.g., neural network 106) can be configured to operate (i.e., configured to process input 102), while fewer than one complete neural network (e.g., fewer than all layers of neural network 104, such as one layer of neural network 104) can be configured to operate in parallel. In some embodiments, when the two neural networks operate in parallel, one can be configured to have more layers of operation than the other. Figure 3 yes Figure 2 One version shows a more detailed timeline of this neural network. Figure 3 A first timeline 302 for a first neural network (e.g., neural network 104) and a second timeline 304 for a second neural network (e.g., neural network 106) are shown. At any given time, only one layer of the first neural network is operating, while all layers of the second neural network are always running. Before a layer of the second neural network is reset, the corresponding layer of the first neural network is turned on, and the outputs of the two corresponding layers are combined according to a weighting scheme 306. The weighting scheme 306 shows the weighting ratios applied as a function of time to the left-hand first neural network (ranging from 0 at the bottom to 1 at the top) and the weighting ratios applied as a function of time to the right-hand second neural network (ranging from 1 at the bottom to 0 at the top). Figure 3 In the example, weighting is applied to any output being combined (which might just be the output of one layer). In other words, a layer might have one or two "versions" operating in parallel. When there is only one version, the output of that layer is simply the output of that layer and is fed to the next layer. When there are two versions, the output of that layer will be a weighted combination of the outputs of those versions, which will be fed to the next layer. While first timetable 302 shows a layer of a first neural network operating in parallel with the second neural network, it should be understood that subsets of these layers can also operate in parallel with the second neural network. Similarly, while second timetable 304 shows a layer being reset at some point, it should be understood that subsets of these layers can be reset at some point. Figure 3 Two parallel neural networks are shown, but in some embodiments, there may be more than two neural networks.

[0043] Figure 4 Schemes for processing data fragments according to certain embodiments described herein are illustrated. Figure 4A timeline 408 for input data, a timeline 402 for processing by a first neural network (e.g., neural network 104), a timeline 404 for processing by a second neural network (e.g., neural network 106), and a timeline 406 for playback are shown. The playback output can be a version of the input data that has already been processed by two neural networks operating in parallel. For example, in the context of a hearing aid, the playback output can be played back by the hearing aid's receiver and can be an audio version that has been processed by two or more noise-reduction neural networks before entering the microphone.

[0044] Input data can be sampled as data segments, each segment having a certain time length (“window”) and starting a certain time after the start of a previous segment (“step”). In some embodiments, the window can be between 1 ms and 10 ms, for example, 4 ms. In some embodiments, the step size can be between 1 ms and 5 ms, for example, 2 ms. In a non-limiting embodiment with a 4 ms window and a 2 ms step size, the latest 4 ms input data can be passed through each neural network every 2 ms. Depending on the window and step size, multiple segments can sample the same input data. For example, in Figure 4 In the examples, segments 1 and 2 both sample the input data from t=0 to t=2ms.

[0045] As shown, after a time corresponding to the computational delay, each neural network produces an output based on the data segment. The output of each neural network can be referred to as a vote. As mentioned above, the output of a neural network should be understood to include the outputs of any layer within the neural network. Therefore, refer to... Figure 4 The output of the neural network can be the output of any combination of the first layer, middle layer, last layer, or multiple layers of the neural network. The processing scheme can wait for each neural network to generate multiple votes for a given segment of input data. For example, the input data segment from t=0 to t=2ms can receive votes 1A and 1B from the first neural network and votes 1B and 2B from the second neural network. The processing scheme can include waiting until the output playback begins, and then performing a second vote (i.e., votes 2A and 2B) on the relevant portions of the input data for each neural network, even if the first vote has already been performed. Therefore, for input data starting at t=0, Figure 4 The total delay is shown from the start of input data at t=0 to the start of playback of the corresponding output.

[0046] The outputs of each neural network can be combined, and the combined outputs can be used to generate the final output for playback. For example, the combined outputs could include averaging relevant votes or using a weighted scheme. Figure 4In the example, a weight w1 / 2 is assigned to each of the two votes from the first neural network, and a weight w2 / 2 is assigned to each of the two votes from the second neural network. In some embodiments, the weights of each neural network may be different; for example, the values ​​of w1 and w2 may be chosen to support one neural network having more votes than the other. In some embodiments, the weights of each neural network may be the same (i.e., w1 = w2).

[0047] Figure 5 Graphs of example weighting schemes and reset schemes according to certain embodiments described herein are shown. Figure 5 The diagram illustrates a first weighting scheme 502 for the output of a first neural network (e.g., neural network 104), a second weighting scheme 504 for the output of a second neural network (e.g., neural network 106), a reset time 506 for the first neural network, and a reset time 508 for the second neural network. As described above, the output of a neural network should be understood to include the output of any layer of the neural network. Therefore, refer to... Figure 5 The described neural network output can be the output of the first layer, a middle layer, the last layer, or any combination of multiple layers. Furthermore, the reference to weighting applied to a neural network should be understood as an average weighting applied to the neural network output.

[0048] As shown in the first weighting scheme 502, the first neural network can be weighted to 1 at a certain time, then converted from 1 to 0 during a transition time, and then reset at a reset time 506. After the reset time 506 of the first neural network, the first neural network can continue to operate, but is in a warm-up period 510. After the warm-up period of the first neural network, as shown in the first weighting scheme 502, the weights of the first neural network begin to transition from 0 to 1. While the first neural network is in the warm-up period, the weights of the second neural network are 1. The second neural network can follow the opposite weighting and reset scheme, as shown in the second weighting scheme 504 and reset time 508. Although Figure 5 A preheating period of approximately 0.45 seconds is shown for 510, but some embodiments may have preheating periods of other suitable lengths, such as between 0.1 and 2.0 seconds.

[0049] Figure 6 Graphs of example weighting schemes and reset schemes according to certain embodiments described herein are shown. Figure 6 The diagram illustrates a first weighting scheme 602 for the output of a first neural network (e.g., neural network 104), a second weighting scheme 604 for the output of a second neural network (e.g., neural network 106), a reset time 606 for the first neural network, and a reset time 608 for the second neural network. As described above, the output of a neural network should be understood to include the output of any layer of the neural network. Therefore, referring to... Figure 6 The described neural network output can be the output of the first layer, a middle layer, the last layer, or any combination of multiple layers. Furthermore, the reference to weighting applied to a neural network should be understood as an average weighting applied to the neural network output.

[0050] As shown in the first weighting scheme 602, the first neural network can be weighted to 1 at a certain time, and then converted from 1 to 0 during a transition time. During the shutdown period 612, the first neural network is turned off. At the reset time 606, the first neural network is turned on and reset, and then remains on but in a warm-up period 610. After the warm-up period 610 of the first neural network, as shown in the first weighting scheme 602, the weights of the first neural network begin to transition from 0 to 1. While the first neural network is in the warm-up period, the weights of the second neural network are 1. The second neural network can follow the opposite weighting and reset scheme, as shown in the second weighting scheme 604 and reset time 608. Although Figure 6 The diagram shows a shutdown period 612 and a warm-up period 610, each approximately 0.74 seconds long, but some embodiments may have shutdown and warm-up periods of other suitable lengths (e.g., between 0.1 and 2.0 seconds).

[0051] Therefore, in some embodiments, a neural network can operate independently for a period of time. For example, in Figure 6 In this configuration, the second neural network operates independently during the shutdown period 612. In some embodiments, the first neural network may operate independently, and then the second neural network may begin operation after a certain period. During this time, the second neural network may warm up and begin establishing meaningful states. After the warm-up period, the output of the second neural network can be utilized. After another period, the first neural network may begin to reset, wherein the first neural network terminates its operation and is then reset. After a certain period, the first neural network warms up, and then after the warm-up period, the output of the first neural network can be utilized.

[0052] As described above, in some embodiments, two complete neural networks can operate in parallel, while in other embodiments, a complete neural network can operate in parallel with one layer or a subset of layers of another neural network. Assuming two complete neural networks operate in parallel, in... Figure 5 In the example, each neural network operates 100% of the time. Figure 6 In the example, the two neural networks run simultaneously for approximately 70.48% of the time. However, other percentages are possible.

[0053] Let P be the power consumed by the complete neural network during operation. The inventors have realized that when two such neural networks (e.g., two identical neural networks) run at 100% time, as... Figure 5 As shown, the power consumption may always be 2P. (As indicated...) Figure 6 As shown, reducing the time that two neural networks run simultaneously may help reduce power consumption. Figure 6 In this case, power consumption is 2P only for 70.48% of the time. Alternatively, the two weighting schemes may not be opposites. For example, with... Figure 6 In contrast, if the weighting scheme of the first neural network can be maintained at 1 for a longer period and / or the warm-up period of the first neural network can be shorter, the amount of time that the two neural networks can run simultaneously can be shorter, and the power consumption can be reduced.

[0054] Suppose a complete neural network operates in parallel with one layer or a subset of layers of a second neural network; the power consumption may be lower. For example, if the second neural network operates at most one layer of its n layers at any given time, the power consumption at that time might only increase to (1+1) / n )× P .

[0055] Generally, it should be understood that neural networks such as RNNs can have multiple layers, each containing multiple states. For example, a network with n layers, each containing m states, may have a total of n*m states. For two parallel neural networks, there may be a total of 2*n*m states. In some embodiments, resetting the states of two neural networks at different times may mean that all n*m states in one neural network are reset simultaneously (“first time”), and all n*m states in the other neural network are reset simultaneously (second time), but the first and second resets are different. In some embodiments, even the states of a single neural network can be reset at different times. For example, all 2*n*m states in two neural network systems can be reset at different times.

[0056] Further description of neural networks, training neural networks, and implementing neural networks in hearing aids can be found in U.S. Patent No. 11,812,225, entitled “Method, Apparatus and System for Neural Network Hearing Aid,” published November 7, 2023, the entire contents of which are incorporated herein by reference.

[0057] Figure 7A block diagram of an ear-worn device 702 according to certain embodiments described herein is shown. The ear-worn device 702 can be any type of ear-worn device (e.g., a hearing aid, cochlear implant, headphones, etc.) and can be any such ear-worn device described herein. The ear-worn device 702 includes one or more microphones 714, processing circuitry 716, noise reduction circuitry 718 including neural network circuitry 720, processing circuitry 730, a receiver 722, communication circuitry 724, control circuitry 726, and a battery 728. It should be understood that the ear-worn device 702 may include more elements than shown.

[0058] One or more microphones 714 may be configured to receive sound and convert it into analog electrical signals. In some embodiments, processing circuitry 716 may include analog processing circuitry. The analog processing circuitry may be configured to perform analog processing on the audio signals received from the microphones 714. For example, the analog processing circuitry may be configured to perform one or more of analog preamplification, analog filtering, and analog-to-digital conversion. Therefore, the analog processing circuitry may be configured to generate analog processed audio signals based on the audio signals received from the microphones 714. The analog processed audio signals may include multiple individual signals, each being an analog processed version of one of the audio signals received from the microphones 714. As described herein, the analog processing circuitry may include analog-to-digital conversion circuitry, and the analog processed signals may be digital signals converted from analog to digital by the analog-to-digital conversion circuitry.

[0059] In some embodiments, processing circuitry 716 may include digital processing circuitry. The digital processing circuitry may be configured to perform digital processing on analog processed audio signals received from analog processing circuitry. For example, the digital processing circuitry may be configured to perform one or more of wind noise reduction, input calibration, and anti-feedback processing. Therefore, the digital processing circuitry may be configured to generate digitally processed audio signals from the analog processed audio signals. The digitally processed audio signals may include multiple separate signals, each a digitally processed version of one of the analog processed audio signals.

[0060] In some embodiments, the processing circuitry 716 may include beamforming circuitry. The beamforming circuitry may be configured to generate one or more beamformed audio signals from two or more digitally processed audio signals. The beamformed audio signals may include one or more individual signals, each of which is a beamformed version of one or more digitally processed audio signals.

[0061] The noise reduction circuit 718 includes a neural network circuit 720. The neural network circuit 720 can be configured to implement one or more trained neural network layers to perform noise reduction. In some embodiments, the neural network circuit 720 can be configured to implement multiple neural networks (e.g., recurrent neural networks or other stateful neural networks) operating in parallel (e.g., neural network system 100). Although the neural network circuit 720 can receive audio signals that have been received by one or more microphones 714 and processed (e.g., by processing circuit 716), this can still be referred to herein as neural network circuit 720 reducing noise in audio signals received by one or more microphones 714.

[0062] The processing circuit 730 can be configured to perform further processing on the output of the noise reduction circuit 718. For example, the processing circuit 730 may include digital processing circuitry configured to perform one or more of wide dynamic range compression and output calibration.

[0063] Receiver 722 can be configured to play back the output of processing circuitry 730 as sound to the user's ear. Receiver 722 can also be configured to perform digital-to-analog conversion before playback.

[0064] The communication circuit 724 can be configured to communicate with other devices via a wireless connection (such as Bluetooth, WiFi, LTE, or NFMI). The control circuit 726 can be configured to control the operation of the processing circuit 716, the noise reduction circuit 718 (including the neural network circuit 720), the processing circuit 730, the communication circuit 724, and the receiver 722.

[0065] The neural network circuit 720 may include circuitry configured to perform operations required to compute the output of a neural network layer. One such operation may be matrix-vector multiplication. In some embodiments, the neural network circuit 720 may include multiple identical tiles, each tile including multiple multiplication-accumulation circuits configured to perform intermediate computations of matrix-vector multiplication in parallel, and then compute the results of the intermediate computations into a final result. Each tile may additionally include memory configured to store neural network weights, registers configured to store input activation elements, and routing circuitry configured to facilitate state and data communication between tiles. Other types of circuitry configured to perform processing, such as processing circuitry 716, other portions of noise reduction circuitry 718, and / or processing circuitry 730, may be implemented as digital processing circuitry. In some embodiments, such digital processing circuitry may use a SIMD (Single Instruction Multiple Data) architecture. The ear-worn device 702 may include a chip implementing certain portions of the circuitry. For example, noise reduction circuitry 718 may be implemented (in whole or in part) on a chip. Therefore, the chip may include the tiles and digital processing circuitry described above. In some embodiments, for models with up to 10M 8-bit weights, when operating on time-series data at 100 GOP / s, when the chip uses a supply voltage between 0.5 and 1.8 V, and when the chip operates without idle time, the chip can achieve a power efficiency of 4 GOP / mW measured at 40 degrees Celsius. Further description of a chip for an ear-worn device incorporating neural network circuitry (among other elements in some embodiments) can be found in U.S. Patent No. 11,886,974, entitled “Neural Network Chip for Ear-Worn Device,” published January 30, 2024, the entire contents of which are incorporated herein by reference. In some embodiments, in addition to such a chip including some or all of the noise reduction circuitry, the ear-worn device 702 may also include a digital signal processor configured to perform other operations (e.g., some or all of the processing performed by processing circuitry 716 and / or processing circuitry 730). Therefore, in some embodiments, processing circuitry 716 (or a portion thereof) can be implemented on a single chip (i.e., a single semiconductor die or substrate). In some embodiments, noise reduction circuitry 718 (or a portion thereof) can be implemented on a single chip. In some embodiments, neural network circuitry 720 (or a portion thereof) can be implemented on a single chip. In some embodiments, processing circuitry 716 (or a portion thereof), noise reduction circuitry 718 (or a portion thereof), and processing circuitry 730 (or a portion thereof) can be implemented on a single chip.

[0066] Figure 8A perspective view of a hearing aid 802 according to certain embodiments described herein is shown. Hearing aid 802 can be any ear-worn device or hearing aid described herein. Hearing aid 802 is an in-canal receiver (RIC) type hearing aid (also known as an in-ear receiver (RITE)). However, any other type of hearing aid (e.g., behind-the-ear, in-the-ear, in-the-canal, completely-in-the-canal, open-back, etc.) can also be used. Hearing aid 802 includes a body 811, a receiver cable 813, a receiver 822, and an earpiece 815. The body 811 is coupled to the receiver cable 813, and the receiver cable 813 is coupled to the receiver 822. The earpiece 815 is placed on the receiver 822. The body 811 includes a front microphone 814f, a rear microphone 814b, and a user input device 804. The body 811 also includes... Figure 1 The circuitry not shown herein (e.g., any circuitry described herein other than receiver 822). Microphones 814f and 814b may be the same as microphone 714. Receiver 822 may be the same as receiver 722. When hearing aid 802 is worn, the front microphone 814f may be positioned closer to the front of the wearer, and the rear microphone 814b may be positioned closer to the rear of the wearer. The front microphone 814f and rear microphone 814b may be configured to receive sound signals and generate audio signals based on the sound signals. Any of the two or more microphones described herein may be the front microphone 814f and rear microphone 814b of hearing aid 802. User input device 804 (e.g., a button) may be configured to control certain functions of hearing aid 802, such as volume, activation of neural network-based noise reduction, etc.

[0067] Receiver line 813 can be configured to transmit audio signals from body 811 to receiver 822. Receiver 822 can be configured to receive audio signals (i.e., those audio signals generated by body 811 and transmitted by receiver line 813) and generate sound signals based on the audio signals. Earplug 815 can be configured to fit snugly inside the wearer's ear and guide the sound signals generated by receiver 822 into the wearer's ear canal.

[0068] In some embodiments, the length of the main body 811 may be equal to 2 cm, equal to 5 cm, or between 2 cm and 5 cm. In some embodiments, the weight of the hearing aid 802 may be less than 4.5 grams. In some embodiments, the spacing between the microphones may be equal to 5 mm, equal to 12 mm, or between 5 mm and 12 mm. In some embodiments, the main body 811 may include a battery ( Figure 1 (Not visible in the image), such as a lithium-ion rechargeable button battery.

[0069] Figure 9 and Figure 10Processes 900 and 1000, respectively, using multiple parallel neural networks to process data according to certain embodiments described herein, are illustrated. In some embodiments, processes 900 and 1000 may be performed by an ear-worn device (e.g., ear-worn device 702), which may be a hearing aid. Processes 900 and 1000 may be performed by a neural network system (e.g., neural network system 100), which may be implemented by neural network circuitry (e.g., neural network circuitry 720). A controller, which may be implemented by control circuitry (e.g., control circuitry 726), may control the neural network system to perform processes 900 and 1000. The neural network system may include a first neural network (e.g., neural network 104) and a second neural network (e.g., neural network 106) operating in parallel. However, it should be understood that the neural network system may include two or more neural networks operating in parallel. The neural network may be a recurrent neural network or another stateful neural network. The neural network may be trained to reduce noise in an audio signal. As described above, in some embodiments, a complete neural network (e.g., neural network 106) can be configured to operate (i.e., configured to process input 102), while fewer than one complete neural network (e.g., fewer than all layers of neural network 104, such as one layer in neural network 104) can be configured to operate in parallel. In some embodiments, when two neural networks operate in parallel, one can be configured to have more operating layers than the other. In some embodiments, two complete neural networks can operate in parallel. Figure 11 Weighting scheme 1102 for a first neural network and weighting scheme 1104 for a second neural network according to certain embodiments described herein are illustrated and will be referenced in the description of procedures 900 and 1000. Figure 11 However, it should be understood that Figure 11 The specific weighting schemes shown are presented as examples only and are not limited to weighting schemes that can be used in conjunction with processes 900 and 1000. For example, any weighting scheme described herein can be used in conjunction with processes 900 and 1000.

[0070] Figures 9-10 The description describes certain steps that may occur at a specific time (e.g., time t1 or t2). This could mean that the step occurs at a single moment or over a period of time. When a step occurs over a period of time but is described as occurring at a specific time, that specific time may be the end of that period. For example, resetting one or more states of a neural network could be described as occurring at time t1, but in reality, resetting one or more states may take some time to complete. Time t1 may be the end of the time required for the reset to occur. From another perspective, if a step is described as occurring at a specific time, this might be equivalent to saying that the step is completed at that specific time.

[0071] In step 902, the neural network system resets one or more states of the first neural network. Step 902 can occur at a first moment. As described above, in some embodiments, resetting a state may mean actively changing the value in the state to zero, or actively changing the value of the state to a different value other than zero. Furthermore, resetting one or more states at a first moment may mean actively changing the value in the state immediately or within a finite time period starting from the first moment. In the latter case, the reset may be smooth, such that the value in the state decays to zero or a different value over time. In some embodiments where the first neural network includes only hidden states, all hidden states may be reset. In some embodiments where the first neural network includes both hidden states and unit states, all unit states may be reset, but the hidden states may not be reset. In some embodiments, only one or more states of certain (i.e., one or more) layers of the neural network may be reset. For example, all states of one or more layers of the neural network may be reset, or a subset of states of one layer or a subset of layers of the neural network (e.g., unit states in an LSTM, but not hidden states) may be reset. As a particular example, one or more states of a layer may be reset at a certain time (e.g., as shown in the image). Figure 3 (As shown). In Figure 11 In the middle, the first time can be t1.

[0072] In step 904, the neural network system receives a first input signal. Step 904 may occur after resetting one or more states of the first neural network (i.e., in step 902). Step 904 may also occur at a later time than the first time. For example, the input signal may be input 102. As described above, the input signal may be processed before being received by the neural network system. Figure 11 In this context, the second time point can be t2. In the context of a hearing aid, the first input signal can be an audio signal.

[0073] In step 906, the neural network system processes the first input signal using a first neural network to produce a first output, and processes the first input signal using a second neural network to produce a second output. For example, the first output could be output 112, and the second output could be input 114. The output can be the output of any layer (e.g., one layer) of the neural network. For example, the output can come from the first layer of the neural network, a middle layer of the neural network, the last layer of the neural network, or any combination of multiple layers. Processing the first input signal should be understood to include processing the input by any layer (e.g., one layer) of the neural network.

[0074] As described above, in some embodiments, the neural network system can run the first neural network during a warm-up period after the reset (e.g., warm-up periods 210 and / or 510), during which the first neural network is turned on but the weights used for its output are zero. Therefore, if input is received between steps 902 and 904 and during the warm-up period of the first neural network, the neural network system can process the input with both neural networks, but only use the output of the second neural network. Equivalently, the output from the first neural network may be weighted to 0. During the warm-up period, the weights from the output of the second neural network can be 1. It should be understood that step 906 can also be applied during the warm-up period of the first neural network. In this case, the first weight can be 0 and / or the second weight can be 1. The warm-up period can occur within a time period after t1, where scheme 1102 is 0. Furthermore, as described above, in some embodiments, the neural network system can be configured to turn off the first neural network before the reset time (e.g., during the shutdown period 612).

[0075] In step 908, the neural network system combines the first output and the second output using a first weight for the first output and a second weight for the second output, wherein the second weight is greater than the first weight. For example, combiner 110 can combine the first and second outputs. Figure 11 In the example, the first weight can be w1 (i.e., the weight associated with time t2 on the first weighting scheme 1102), the second weight can be w2 (i.e., the weight associated with time t2 on the second weighting scheme 1104), and w2 is greater than w1. Figure 11 The weights used in weighting schemes 1102 and 1104 may depend at least in part on the amount of time that has elapsed since the last reset. Weights may transition from low to high after a reset (in some embodiments, after a shutdown period and / or warm-up period), and then from high to low before the next reset. However, other weighting schemes may also be used. For example, in some embodiments, weights may be determined based on a confidence metric associated with the output of each neural network, such that when two neural networks are combined, the neural network with higher quality / confidence is weighted more heavily. In such an embodiment, at step 908, the first weight may be less than the second weight because the confidence associated with the first output is less than the confidence associated with the second output. Although Figure 11 The weighted schemes 1102 and 1104 in the model are typically formed by linear piecewise functions, but other smoothing functions, such as sine functions, can also be used.

[0076] In an embodiment where the first output is the output of a layer of a first neural network and the second output is the output of a layer of a second neural network, the neural network system can feed the combined first and second outputs into subsequent layers in the first neural network.

[0077] Furthermore, in some embodiments, the first output of the first neural network may include a combination of multiple outputs of the first neural network, and the second output of the second neural network may include a combination of multiple outputs of the second neural network, for example, referring to... Figure 4 The description of the vote. (See reference...) Figure 4 To be further described, because a neural network can produce one vote and then another vote at a later time, a neural network system can be configured to wait until the neural network produces multiple outputs before determining the first output.

[0078] In some embodiments, process 900 continues to process 1000. Therefore, step 1010 of process 1000 can occur after step 908 of process 900. In step 1010, the neural network system receives a second input signal. Step 1010 can occur after receiving the first audio signal (i.e., in step 904), or after combining the first and second outputs (i.e., in step 908). Step 1010 can occur at a third time later than the second time. Further description of receiving the input signal can be found with reference to step 904. Figure 11 In this context, the third time point can be t3.

[0079] In step 1012, the neural network system processes the second input signal using a first neural network to produce a third output, and processes the second input signal using a second neural network to produce a fourth output. Further description of the input signal processing can be found in step 906.

[0080] In step 1014, the neural network system combines the third output and the fourth output using the third weight of the third output and the fourth weight of the fourth output, where the third weight is greater than the first weight. Further description of the combined output can be found in step 908. Figure 11 In the example, the third weight can be w3 (i.e., the weight associated with time t3 on the first weighting scheme 1102), the fourth weight can be w4 (i.e., the weight associated with time t3 on the second weighting scheme 1104), and w3 is greater than w1. Furthermore, in Figure 11 In the example, the fourth weight is less than the second weight. In some embodiments, in step 1014, the third and fourth weights may be different (e.g., 0.25 and 0.75). In some embodiments, the third and fourth weights may be the same (e.g., 0.5). In some embodiments, in step 1014, the third weight may be less than the fourth weight. In some embodiments, in step 1014, the third weight may be greater than the fourth weight.

[0081] In step 1016, the neural network system resets one or more states of the second neural network. Step 1016 can occur after the second audio signal is received (i.e., in step 1010). Step 1016 can also occur at a fourth time later than the third time. Further description of the state reset can be found in step 902. Figure 11 In the example, the fourth time point could be t4.

[0082] In some embodiments, after step 1016, processes 900 and 1000 can be repeated from step 904, but the roles of the first neural network and the second neural network are interchanged. Therefore, the neural network system can receive a third input signal at a fifth time later than the fourth time; process the third input signal using the first neural network to produce a fifth output, and process the third input signal using the second neural network to produce a sixth output; combine the fifth and sixth outputs using fifth weights for the fifth output and sixth weights for the sixth output, wherein the fifth weight is greater than the sixth weight; receive a fourth input signal at a sixth time later than the fifth time; process the fourth input signal using the first neural network to produce a seventh output, and process the fourth input signal using the second neural network to produce an eighth output; and combine the seventh and eighth outputs using seventh weights for the seventh output and eighth weights for the eighth output, wherein the eighth weight is greater than the sixth weight.

[0083] Furthermore, in an embodiment where one or more states of a layer of the first neural network are reset in step 902, one or more states of different layers of the first neural network may be reset at a later time. This later time may differ from the fourth time to provide interleaved resets of the first and second neural networks.

[0084] As can be understood from the foregoing, typically, the control circuitry (e.g., control circuitry 726) in a device (e.g., an ear-worn device, such as ear-worn device 702) can be configured to reset one or more states of a first neural network at one or more first reset times, and to reset one or more states of a second neural network at one or more second reset times, wherein the one or more second reset times may differ from the one or more first reset times. In other words, the one or more second reset times may be staggered relative to the one or more first reset times.

[0085] It should be understood that while the above description may focus on ear-worn devices (e.g., hearing aids), the described features can be implemented in any type of device using a neural network system. Therefore, any device can have neural network circuitry and control circuitry configured to implement a neural network system comprising at least a first neural network and a second neural network operating in parallel, the control circuitry configured to control the neural network system to receive a first input signal (not necessarily an audio signal), process the first input signal using the first neural network to produce a first output, process the first input signal using the second neural network to produce a second output, combine the first and second outputs, reset one or more states of the first neural network, and reset one or more states of the second neural network at a time different from resetting the one or more states of the first neural network.

[0086] Example 1 relates to an apparatus including a neural network circuit and a control circuit. The neural network circuit is configured to implement a neural network system, the neural network system including at least a first neural network and a second neural network configured to operate in parallel. The control circuit is configured to control the neural network system to: reset one or more states of the first neural network; receive a first audio signal after resetting one or more states of the first neural network; process the first audio signal using the first neural network to produce a first output, and process the first audio signal using the second neural network to produce a second output; and combine the first output and the second output using a first weight for the first output and a second weight for the second output, wherein the second weight is greater than the first weight.

[0087] Example 2 relates to the apparatus of Example 1, wherein the control circuitry is configured to: reset one or more states of a first neural network at one or more first reset times; and reset one or more states of a second neural network at one or more second reset times; wherein the one or more second reset times are different from the one or more first reset times.

[0088] Example 3 relates to the apparatus of Example 1, wherein the neural network is further configured to: receive a second audio signal after combining a first output and a second output; process the second audio signal using a first neural network to produce a third output, and process the second audio signal using a second neural network to produce a fourth output; combine the third output and the fourth output using a third weight for the third output and a fourth weight for the fourth output, wherein the third weight is greater than the first weight; and reset one or more states of the second neural network after receiving the second audio signal.

[0089] Example 4 relates to an apparatus of any one of Examples 1-3, wherein all layers of a first neural network are configured to process a first audio signal, and fewer than all layers of a second neural network are configured to process the first audio signal.

[0090] Example 5 relates to the apparatus of Example 4, wherein the neural network system is configured to reset one or more states of a specific layer of the first neural network when resetting one or more states of the first neural network.

[0091] Example 6 relates to the apparatus of Example 5, wherein the neural network system is further configured to reset one or more states of different layers of the first neural network after resetting one or more states of a specific layer of the first neural network.

[0092] Example 7 relates to the apparatus of Example 6, wherein the time between resetting one or more states of a specific layer of the first neural network and resetting one or more states of a different layer of the first neural network is approximately equal to 1 second, approximately equal to 60 seconds, or between 1 second and 60 seconds.

[0093] Example 8 relates to an apparatus of any one of Examples 5-7, wherein the neural network system is configured to process the first audio information using a layer of the first neural network to produce the first output and to process the first audio data using a layer of the second neural network to produce the second output when processing the first audio signal using a first neural network to produce a first output and processing the first audio signal using a second neural network to produce a second output.

[0094] Example 9 relates to the apparatus of Example 8, wherein the neural network system is further configured to feed the combined first and second outputs to subsequent layers of the first neural network.

[0095] Example 10 relates to an apparatus of any one of Examples 1-9, wherein the neural network system is configured to run the first neural network during a warm-up period after resetting one or more states of the first neural network, and to weight the output of the first neural network to zero during the warm-up period.

[0096] Example 11 relates to an apparatus of any one of Examples 1-10, wherein the neural network system is configured to shut down the first neural network during a shutdown period before resetting one or more states of the first neural network.

[0097] Example 12 relates to an apparatus of any one of Examples 1-11, wherein the weights applied to the output of the first neural network depend at least in part on the amount of time elapsed since one or more states of the first neural network were reset.

[0098] Example 13 relates to the apparatus of Example 12, wherein, after resetting one or more states of the first neural network, the weights applied to the output of the first neural network are switched from low to high, and then from high to low before the next reset of one or more states of the first neural network.

[0099] Example 14 relates to an apparatus of any of Examples 1-13, wherein the neural network circuitry is implemented on a chip.

[0100] Example 15 relates to an apparatus of any one of Examples 1-14, wherein a first output from a first neural network comprises a combination of multiple outputs from the first neural network, and the neural network system is configured to wait until the first neural network produces multiple outputs before determining the first output.

[0101] Example 16 relates to an apparatus of any one of Examples 1-15, wherein the first and second weights are determined according to a weighting scheme comprising a linear piecewise function or a smoothing function.

[0102] Example 17 relates to an apparatus of any one of Examples 1-16, wherein a first neural network and a second neural network are trained to reduce noise in an audio signal.

[0103] Example 18 relates to an apparatus of any one of Examples 1-17, wherein the first neural network and the second neural network include the same architecture and the same weights.

[0104] Example 19 relates to an apparatus of any one of Examples 1-18, wherein the first neural network and the second neural network include recurrent neural networks.

[0105] Example 20 relates to an apparatus of any one of Examples 3-19, wherein the control circuitry is further configured to control a neural network system to: receive a third audio signal after resetting one or more states of a second neural network; process the third audio signal using a first neural network to produce a fifth output, and process the third audio signal using a second neural network to produce a sixth output; combine the fifth and sixth outputs using a fifth weight for the fifth output and a sixth weight for the sixth output, wherein the fifth weight is greater than the sixth weight; receive a fourth audio signal after receiving the third audio signal; process the fourth audio signal using the first neural network to produce a seventh output, and process the fourth audio signal using the second neural network to produce an eighth output; and combine the seventh and eighth outputs using a seventh weight for the seventh output and an eighth weight for the eighth output, wherein the eighth weight is greater than the sixth weight.

[0106] Example 21 relates to a device of any of Examples 1-20, wherein the device includes an ear-worn device.

[0107] Example 22 relates to a device of any of Examples 1-21, wherein the device includes a hearing aid.

[0108] After describing several embodiments of the present technology in detail, various modifications and improvements will readily occur to those skilled in the art. These modifications and improvements are intended to fall within the spirit and scope of the invention. Therefore, the above description is merely illustrative and not intended to be limiting. For example, any of the components described above may include hardware, software, or a combination of hardware and software.

[0109] Unless otherwise expressly stated, the indefinite articles “a” and “an” used in the specification and claims shall be understood as “at least one”.

[0110] The phrase “and / or” as used in this specification and claims should be understood to mean “one or two” of so-called related elements, i.e., elements that exist together in some cases and separately in others. Multiple elements listed with “and / or” should be interpreted in the same way, i.e., “one or more” of so-called related elements. In addition to the elements specifically identified by the “and / or” clause, other elements may optionally exist, whether related to or unrelated to those specifically identified.

[0111] As used in the specification and claims, the phrase "at least one" refers to a list of one or more elements and should be understood to mean at least one element selected from any one or more elements in the list, but does not necessarily include at least one of each element specifically listed in the list, nor exclude any combination of elements in the list. This definition also allows for the optional presence of elements other than those specifically identified in the list of elements referred to by the phrase "at least one," regardless of whether such elements are related to those specifically identified elements.

[0112] In some embodiments, the terms "approximately" and "about" may be used to indicate within ±20% of a target value, within ±10% of a target value in some embodiments, within ±5% of a target value in some embodiments, and within ±2% of a target value in some embodiments. The terms "approximately" and "about" may include the target value.

[0113] Furthermore, the wording and terminology used herein are for descriptive purposes only and should not be considered limiting. The terms “including,” “comprising,” “having,” “containing,” “involving,” and variations thereof, as used herein, are intended to include items listed below and their equivalents, as well as additional items.

[0114] Having described several aspects of at least one embodiment above, various changes, modifications, and improvements will readily occur to those skilled in the art. These changes, modifications, and improvements are intended to serve the purposes of this disclosure. Therefore, the above description and figures are merely examples.

Claims

1. An ear-worn device, comprising: A neural network circuit configured to implement a neural network system comprising at least a first neural network and a second neural network configured to operate in parallel. as well as Control circuitry, the control circuitry being configured to control the neural network system to: Reset one or more states of the first neural network; A first audio signal is received after resetting one or more states of the first neural network; The first audio signal is processed using the first neural network to produce a first output, and the first audio signal is processed using the second neural network to produce a second output; as well as The first output and the second output are combined using a first weight for the first output and a second weight for the second output, wherein the second weight is greater than the first weight.

2. The ear-worn device according to claim 1, wherein, The control circuit is configured as follows: Reset one or more states of the first neural network at one or more first reset times; as well as Reset one or more states of the second neural network at one or more second reset times; Wherein, the one or more second reset times are different from the one or more first reset times.

3. The ear-worn device according to claim 1, characterized in that: The neural network is also configured to: After combining the first output and the second output, a second audio signal is received; The first neural network is used to process the second audio signal to produce a third output, and the second neural network is used to process the second audio signal to produce a fourth output; The third output and the fourth output are combined using a third weight for the third output and a fourth weight for the fourth output, wherein the third weight is greater than the first weight; as well as After receiving the second audio signal, one or more states of the second neural network are reset.

4. The ear-worn device according to claim 3, wherein, The control circuit is also configured to control the neural network system to: A third audio signal is received after resetting one or more states of the second neural network; The third audio signal is processed using the first neural network to produce a fifth output, and the third audio signal is processed using the second neural network to produce a sixth output; The fifth output and the sixth output are combined using a fifth weight for the fifth output and a sixth weight for the sixth output, wherein the fifth weight is greater than the sixth weight; After receiving the third audio signal, a fourth audio signal is received; The fourth audio signal is processed using the first neural network to produce a seventh output, and the fourth audio signal is processed using the second neural network to produce an eighth output; and The seventh output and the eighth output are combined using a seventh weight for the seventh output and an eighth weight for the eighth output, wherein the eighth weight is greater than the sixth weight.

5. The ear-worn device according to claim 1, wherein, All layers of the first neural network are configured to process the first audio signal, and fewer than all layers of the second neural network are configured to process the first audio signal.

6. The ear-worn device according to claim 5, wherein, The neural network system is configured to reset one or more states of a specific layer of the first neural network when resetting one or more states of the first neural network.

7. The ear-worn device according to claim 6, wherein, The neural network system is also configured to reset one or more states of different layers of the first neural network after resetting one or more states of a specific layer of the first neural network.

8. The ear-worn device according to claim 7, wherein, The time between resetting one or more states of a specific layer of the first neural network and resetting one or more states of a different layer of the first neural network is approximately equal to 1 second, approximately equal to 60 seconds, or between 1 second and 60 seconds.

9. The ear-worn device according to claim 6, wherein, The neural network system is configured such that, when the first neural network is used to process the first audio signal to produce the first output and the second neural network is used to process the first audio signal to produce the second output, a layer of the first neural network is used to process the first audio signal to produce the first output, and a layer of the second neural network is used to process the first audio signal to produce the second output.

10. The ear-worn device according to claim 9, wherein, The neural network system is also configured to feed the combined first and second outputs into subsequent layers of the first neural network.

11. The ear-worn device according to claim 1, wherein, The neural network system is configured to run the first neural network during a warm-up period after resetting one or more states of the first neural network, and to weight the output of the first neural network to zero during the warm-up period.

12. The ear-worn device according to claim 1, wherein, The neural network system is configured to shut down the first neural network during a shutdown period before resetting one or more states of the first neural network.

13. The ear-worn device according to claim 1, wherein, The weights applied to the output of the first neural network depend at least in part on the amount of time that has elapsed since one or more states of the first neural network were reset.

14. The ear-worn device according to claim 13, wherein, The weights applied to the output of the first neural network transition from low to high after resetting one or more states of the first neural network, and then transition from high to low before the next reset of one or more states of the first neural network.

15. The ear-worn device according to claim 1, wherein, The neural network circuit is implemented on a chip.

16. The ear-worn device according to claim 1, wherein, The first output from the first neural network includes a combination of multiple outputs from the first neural network, and the neural network system is configured to wait until the first neural network produces the multiple outputs before determining the first output.

17. The ear-worn device according to claim 1, wherein, The first weight and the second weight are determined according to a weighting scheme that includes a linear piecewise function or a smoothing function.

18. The ear-worn device according to claim 1, wherein, The first neural network and the second neural network are trained to reduce noise in the audio signal.

19. The ear-worn device according to claim 1, wherein, The first neural network and the second neural network have the same architecture and the same weights.

20. The ear-worn device according to claim 1, wherein, The first neural network and the second neural network include recurrent neural networks.

Images