Neural network chip for ear-mounted device

Abstract

An ear-mounted device may include a neural network chip. A neural network chip may include a plurality of computing units arranged in a computing unit array, each computing unit including a memory, a plurality of multiply-accumulator circuits (MACs), and a routing circuit. The memory of each computing unit may be configured to store a portion of elements of a matrix A including weights of the neural network. Each computing unit may be configured to receive and store elements of an activation vector X, wherein the activation vector X is derived from an input audio signal. All or a subset of the plurality of computing units may be configured to perform matrix vector multiplication AX by performing multiplication and accumulation sub-operations in parallel between all or a subset of the plurality of computing units. Routing circuitry for computing cells in the array of computing cells may be configured to combine the results of the multiply and accumulate sub-operations.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to a neural network chip for an ear-wearable device, such as a hearing aid. BACKGROUND

[0002] Ear-wearable devices, such as hearing aids, can be used to help people with hearing difficulties to hear better. Typically, a hearing aid amplifies received sound. Some hearing aids attempt to enhance sound, for example, by removing ambient noise from the incoming sound. SUMMARY

[0003] Wearers of ear-wearable devices, e.g. hearing aids or cochlear implants, typically have a hearing deficiency. While conventional ear-wearable devices can be used to amplify sound, they can not be configured to distinguish between target and non-target sound and / or to selectively process components of detected audio. Neural network-based audio enhancement techniques can be employed to address such deficiencies of conventional ear-wearable device technology.

[0004] Deploying audio enhancement techniques can introduce a delay between when a sound is emitted by a sound source and when the enhanced sound is output to a user. For example, such techniques can introduce a delay between when a speaker speaks and when the enhanced speech is heard by a listener. During face-to-face communication, a long delay time can create an echo perception since both the original sound and the enhanced version of the sound are played back to the listener. In addition, a long delay time can interfere with how the listener processes incoming sound due to a disconnection between visual cues, e.g. moving lips, and the arrival of the associated sound.

[0005] Conventional approaches for incorporating neural networks into the signal processor of an ear-wearable device, such as a hearing aid, involve allocating a fixed number of processors to run the neural network. The inventors have recognized that, in order to achieve tolerable delays when implementing neural networks on ear-wearable devices, ear-wearable devices can need to be able to perform billions of operations per second. Conventional approaches for achieving this processing speed can involve increasing the clock frequency of the processors or increasing the total number of processors used to implement the neural network. However, the inventors have recognized drawbacks associated with both of these approaches.

[0006] First, increasing the clock frequency can require increasing the voltage supplied to the processors. This leads to increased power consumption, which shortens the battery life of the device. Power consumption increases because: first, power consumption is proportional to f v 2proportional to the square of the frequency (where f is the clock frequency and v is the voltage). Additionally, the size of the logic elements required to support higher frequencies also increases power consumption. Prospective wearers of such devices can need to frequently replace or recharge batteries to bear the burden of shortened battery life. Furthermore, while increasing the size of the battery can help extend battery life, it can increase the weight of the ear-wearable device, which can cause discomfort to the wearer.

[0007] Second, increasing the number of processors can result in those processors being physically scattered across the chip. This can pose challenges to implementing speech and audio enhancement algorithms, such as, for example, recurrent neural networks (RNNs), and the like. Such algorithms can use recently received information to enhance currently received audio signals. Unlike other neural networks, such as convolutional neural networks, such algorithms can reuse data very little. As a result, executing such algorithms involves constantly reading weights (i.e., parameters of the neural network model) from memory, which incurs power consumption and latency. This problem can be exacerbated when the processors are physically scattered across the chip, as more power can be consumed when moving data across a power-consuming bus between memory and a remote processor.

[0008] Accordingly, the inventors have developed methods and apparatuses that address the above challenges of conventional neural network-based audio enhancement techniques and hearing aid technology. In some embodiments, the methods and apparatuses can include an ear-wearable device (e.g., a hearing aid or cochlear implant) having a neural network chip configured to implement a neural network model for enhancing (e.g., denoising and / or spatial focusing) audio signals. BRIEF DESCRIPTION OF DRAWINGS

[0009] Various aspects and embodiments of the present application will be described with reference to the following drawings. It should be appreciated that the drawings are not necessarily to scale. Items appearing in multiple drawings are indicated by the same reference numeral appearing in all the drawings in which they appear.

[0010] Figure 1 A computing tile is shown in accordance with certain embodiments described herein;

[0011] Figure 2 A biasing circuit is shown in accordance with certain embodiments described herein;

[0012] Figure 3 Circuitry on a neural network chip is shown in accordance with certain embodiments described herein;

[0013] Figure 4shows how a single element X can be simultaneously fanned out from a vector memory to each MAC in a compute unit using a direct parallel path according to certain embodiments described herein (e.g., within a single clock cycle);

[0014] Figure 5 shows an array of compute units according to certain embodiments described herein;

[0015] Figure 6 shows circuitry in a neural network chip according to certain embodiments described herein in more detail;

[0016] Figure 7 shows a process for operating a neural network on a neural network chip according to certain embodiments described herein;

[0017] Figure 8 shows a block diagram of an ear-wearable device according to certain embodiments described herein. DETAILED DESCRIPTION

[0018] In some embodiments, a neural network chip can include substantially identical circuit compute units. Figure 1 A compute unit 100 according to certain embodiments described herein is shown. The compute unit 100 can be one of a plurality of compute units in a neural network chip. Each compute unit 100 of the plurality of compute units includes a memory 128, a processing circuit 130, a routing circuit 108, and a logic circuit 134. The memory 128 includes a vector memory 102 and a weight memory 104. The processing circuit 130 includes a multiplier-accumulator (MAC) circuit 130. An input v_in of the compute unit 100 is coupled to an input of the vector memory 102. An input r_in of the compute unit 100 is coupled to an input of the routing circuit 108. An output r_out of the compute unit 100 is coupled to an output of the routing circuit 108. Outputs of the vector memory 102 and the weight memory 104 are coupled to inputs of the processing circuit 130. An output of the processing circuit 130 is coupled to an input of the routing circuit 108. The logic circuit 134 is coupled to the memory 128 and the processing circuit 130, and the logic circuit 134 is configured to control their operations. As shown, the memory 128 and the processing circuit 130 are locally disposed within each compute unit 100. In some embodiments, this can mean that a distance from the memory 128 of any given compute unit 100 to the processing circuit 130 of that compute unit can be less than a distance from the memory 128 to the processing circuit 130 of another compute unit 100. In some embodiments, the memory 128 and the processing circuit 130 of any given compute unit 100 can be disposed no more than 0.125 mm apart. 2Within an area. In some embodiments, the memory 128 and processing circuitry 130 of any given computing unit 100 can be arranged within an area not exceeding 0.15 mm. 2 Within an area. In some embodiments, the memory 128 and processing circuitry 130 of any given computing unit 100 may be arranged within an area not exceeding 0.175 mm. 2 Within an area. In some embodiments, the memory 128 and processing circuitry 130 of any given computing unit 100 can be arranged within an area not exceeding 0.2 mm. 2 Within an area. In some embodiments, the memory 128 and processing circuitry 130 of any given computing unit 100 can be arranged within an area not exceeding 0.225 mm. 2 Within an area. In some embodiments, the memory 128 and processing circuitry 130 of any given computing unit 100 may be arranged within an area not exceeding 0.25 mm. 2 Within an area. These area values ​​can be based at least in part on the size of the memory 128 within the computing unit 100 and the number of memory 128 entities within the computing unit 128. As will be described below, memory may become inefficient beyond a certain size. The number of memory 128 entities in the computing unit 100 can depend on how many entities the logic circuit 134 can effectively control, as will be described below.

[0019] The weight memory 104 of a specific computing unit 100 can store the weights of a neural network (e.g., weights corresponding to at least a portion of the layers of the neural network). The vector memory 102 of the specific computing unit 100 can store one or more elements of an activation vector. Configuring the memory 128 in parallel with the processing circuitry 130 in this way reduces the power consumption associated with moving data from remote memory to processing circuitry that can be physically distributed throughout a conventional chip. Therefore, the processing circuitry 130 can efficiently retrieve the weights required to perform operations. Thus, the method and apparatus developed by the inventors can avoid the high power consumption associated with frequently moving large amounts of data between remote memory and processing circuitry.

[0020] The neural network chip developed by the inventors achieves a balance between (a) reducing power consumption associated with moving data between processing circuitry and remote memory separate from the processing circuitry, and (b) reducing inefficiencies associated with the size of the on-chip memory. Each computing unit 100 may include one or more memories 128 (e.g., 1, 2, 3, 4, 5, 6, etc.), each memory configured in parallel with one or more instances of processing circuitry 130 (e.g., 1, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 24, 28, 32, etc.). The inventors have recognized that, in some embodiments, it may be advantageous to configure more than one instance of processing circuitry 130 in parallel with each memory 128 to reduce inefficiencies associated with the size of the on-chip memory. For example, larger memories are more efficient than smaller memories, but they occupy more space. Therefore, due to the size limitations of the neural network chip, placing a single instance of processing circuitry 130 together with each relatively large memory may be inefficient. Therefore, placing a limited number of instances of processing circuitry 130 together with each memory 128 (where multiple instances of processing circuitry 130 coupled to memory 128 can be considered "cores") allows for the utilization of the efficiency associated with larger memories without compromising the efficiency associated with parallel configuration of memory 128 and processing circuitry 130, and while adhering to the size limitations of neural network chips. Additionally, in some embodiments, memory 128 may be a single-port memory, meaning that only one address can be read at a time. Single-port memories save space and power compared to, for example, dual-port memories, which can be twice as large and consume more than twice the power. Once memory increases beyond a certain size, the efficiency gains from increasing memory size may become largely negligible. Specifically, for example, there may be a power-area gain per bit when increasing from 32×32 memory to 128×512 memory, but the gain is far less significant when increasing from 128×512 to 128×8192. Therefore, when using one or more (e.g., four) small memories instead of one large memory in computing unit 100, efficiency may decrease slightly, but read bandwidth may increase significantly (by a factor of four in this example) because multiple (four in this example) different addresses may be read at once.

[0021] As shown, each computing unit 100 in the neural network chip further includes logic circuitry 134 for configuring and controlling processing circuitry 130 and memory 102 located on the computing unit 100. Since the logic circuitry 134 does not participate in the computations required to implement the recurrent neural network, the inventors have recognized that minimizing the area and power consumption of the logic circuitry 134 may be beneficial. Therefore, in some embodiments, the computing unit 100 developed by the inventors includes more than one group of logic circuitry 134 for controlling memory 128 and processing circuitry 130. For example, a particular computing unit 100 may include logic circuitry 134 configured to control multiple (e.g., 1, 2, 3, 4, 5, 6, etc.) cores of memory 128 and the processing circuitry 130 associated with that memory 128.

[0022] In some embodiments, the number of computing units 100 in the computing unit array can be between 2-64, 2-32, 2-16, 4-64, 4-32, 4-16, 8-64, 8-32, 8-16, 16-64, or 16-32, or equal to 2-64, 2-32, 2-16, 4-64, 4-32, 4-16, 8-64, 8-32, 8-16, 16-64, or 16-32. For example, there may be 16 computing units 100, which can be arranged in a 4×4 computing unit array. In some embodiments, the number of MAC circuits 106 in the computing unit 100 may be between, or equal to, 16-256, 16-128, 16-64, 32-256, 32-128, 32-64, 64-256, or 64-128. In some embodiments, the number of MAC circuits 106 in the computing unit 100 may be 64. As a non-limiting example, the computing unit 100 may include 64 instances of processing circuitry 130, each instance including MAC circuitry 106. These may be implemented, for example, as four cores, each core including one instance of memory 128 and 16 instances of processing circuitry 130. This computing unit 100 may be configured to compute 64 multiplication and accumulation operations in parallel.

[0023] In operation, the computation unit 100 can be configured to use MAC circuitry 106 to multiply the activation vector elements received from vector memory 102 with weights received from weight memory 104, and add the product to a running sum. The weights in weight memory 104 can be stored continuously on the chip; in other words, these weights do not need to be retrieved from external memory each time a computation with weights is performed. When the device is powered on, they can initially be loaded from external memory (e.g., electrically erasable programmable read-only memory (EEPROM)) in a device in which the chips are arranged (e.g., an ear-worn device). This external memory can be configured to receive updated weights wirelessly (e.g., Bluetooth) and load the updated weights by restarting the device when updates to the weights become available.

[0024] Figure 2 A bias circuit 210 according to certain embodiments described herein is shown. The bias circuit 210 includes a bias memory 212 and a routing circuit 214. The bias circuit 210 has an input v_in coupled to an input of the bias memory 212. The bias circuit 210 has an output r_out coupled to an output of the routing circuit 214. The output of the bias memory 212 is coupled to an input of the routing circuit 214. The bias memory 212 of each bias circuit 210 can be configured to store one or more biases.

[0025] Figure 3 Circuitry on a neural network chip 326 according to certain embodiments described herein is shown. The neural network chip 326 includes... Figure 1 The computing unit 100 shown and Figure 2 Multiple examples of bias circuit 210 are shown. Computational units 100 are electrically arranged in an array 316 of computational units having rows and columns. Fewer bias circuits 210 than computational units 100 may exist, for example, one bias circuit 210 electrically coupled to a row of computational units 100. Figure 3 All the circuits shown can be implemented on a single chip (in other words, a single semiconductor substrate / wafer).

[0026] The computation units 100 of the neural network chip 326 can be configured to combine operations with each other to implement a recurrent neural network. The recurrent neural network may include one or more layers. In some embodiments, implementing a recurrent neural network may include computing one or more matrix-vector operations (e.g., multiplications) for each of the one or more layers of the recurrent neural network. For example, matrix-vector multiplication between activation vectors and the weight matrix of the recurrent neural network may be computed.

[0027] Matrix-vector multiplication can be, for example, AX = Y, where A is a matrix including the weights of the recurrent neural network, X is an activation vector, and Y is the result. The activation vector X can be derived from the input audio signal. For example, the activation vector X used for the first layer can be the result of processing the result of a short-time Fourier transform (STFT) of the digitized audio signal. Each vector Y (i.e., the result of processing the activation vector X using the recurrent neural network with the weights in A) can be an input to subsequent layers (i.e., vector X), or can be used to form the input to subsequent layers (i.e., vector X). As will be described in further detail, matrix-vector multiplication can be decomposed into parallel multiplication and accumulation sub-operations. Therefore, in some embodiments, a subset or all of the computing units 100 of the neural network chip 326 can be combined to compute a particular matrix-vector multiplication of the recurrent neural network. For example, each computing unit 100 in a subset of computing unit array 316 or all computing units 100 in computing unit array 316 may be configured to perform multiplication and accumulation sub-operations in parallel among all the plurality of computing units 100 (using MAC circuit 106), and neural network chip 326 may combine the results of multiplication and accumulation sub-operations to produce the result of matrix-vector multiplication.

[0028] like Figure 3 As shown, each computing unit 100 can be configured to receive and store elements of vector X in vector memory 102 within the computing unit 100. Elements activating vector X can be propagated along columns of computing units 100 in the computing unit array 316 (to input v_in); in other words, each computing unit 100 in a column can receive the same element of X. In some embodiments, chip 326 may not have the ability to transfer different elements of X to different computing units in a column; this lack of flexibility can help reduce the power consumption and / or area of ​​chip 326. Elements of X can be stored near processing circuitry 130 within the computing unit 100, such that weighting requires minimal data movement. Reducing data movement reduces power consumption. The computing unit 100 can then simultaneously fan out a single element X to all MACs 106 within the computing unit 100 for computation within a single clock cycle. Thus, in a computation performed within a single clock cycle, a single element of X can be reused across all MACs 106 within the computing unit 100. Figure 4This illustrates how, according to certain embodiments described herein, a single element X can be simultaneously fanned out from vector memory 102 to each MAC 106 in the compute unit using a direct parallel path (e.g., within a single clock cycle). The inventors have recognized that moving vectors between compute units and individual memories contributes to overall power consumption. Therefore, to reduce total power consumption, instead of retrieving the activation vector for each vector-wise operation (e.g., for each row matrix vector operation), the activation vector can be retrieved from vector memory 320 once and reused. Each bias circuit 210 can be configured to receive a bias and store the bias in bias memory 212.

[0029] As shown, the routing circuit 108 of all computing units 100 in a row and the routing circuit 214 of the row bias circuit 210 can electrically couple all computing units 100 in a row and the row bias circuit 210 together. The routing circuit 108 and the routing circuit 214 can be configured to combine the results of multiplication and accumulation calculations of computing units 100 with the bias.

[0030] The following description details how matrix-vector multiplication plus the calculation of the bias Y=Ax+b can be performed in parallel. The sum of matrix-vector multiplication and the bias b is shown below:

[0031] Figure 5 An array of computing units according to certain embodiments described herein is shown. Figure 5 The computational cell array can be the same as computational cell array 316. Assuming m=256 and n=256, and the computational cell array as follows... Figure 5 As shown. Computation units 0, 4, 8, and 12 can receive elements x1-x64 of the activation vector, while computation units 1, 5, 9, and 13 can receive x65-x128, etc. Bias circuit 0 (Bias 0) can receive biases b1-b64, and bias circuit 1 (Bias 1) can receive biases b65-b128, etc. During the first clock cycle, computation unit 0 (Tile 0) can use its 64 MAC circuits 106 to calculate the following product: a1, 1 x1; a2, 1 x1; ...; a64, 1 x1. It can be understood that each MAC circuit 106 uses the same element of the activation vector (x1 in this case) within a single clock cycle. Within the second clock cycle, computation unit 0 can use its 64 MACs 106 to compute the following product: a1, 2 x2; a2, 2 x2; ...; a64, 2 x2. Within this clock cycle, computation unit 0 can combine these products with the products from the previous clock cycle to produce a1,1. x1+a1,2 x2; a2, 1 x1+a2,2 x2; ...; a64, 1 x1 + a64, 2 x2. After 64 clock cycles, computation unit 0 can calculate the following: a1, 1 x1+a1,2 x2 + ... + a1, 64 x64; a2, 1 x1+a2,2 x² + ... + a², 64 x64; ...; a64, 1 x1+a64,2 x2 + ... + a64, 64 x64. Computation unit 0 may locally store the following weights used in these computations: a1,1; a1,2; ...; a1,64; a2,1; a2,2; ...; a64,64.

[0032] In a similar scenario, after 64 clock cycles, computation unit 1 (Tile 1) can calculate as follows: a1, 65 x65+a1,66 x66+…+a1,128 x128; a2, 65 x65+a2,66 x66+…+a2,128 x128; ...; a64, 65 x65+a64,66 x66+…+a64,128 x128. The results from computation units 0 and 1 can be combined with the results from computation units 2 and 3, as well as the bias element from bias circuit 0, and the same applies to other rows. Therefore, the result of the first row of computation units can be a1, 1 x1 + a1,2 x2 + … + a1, 256 x256 + b1; a2, 1 x1 + a2, 2 x² + … + a², 256 x256 + b2; ...; a64, 1 x1 + a64,2 x2 + … + a64,256 x256 + b64.

[0033] Figure 6 The circuitry in neural network chip 626 (which may be the same as neural network chip 326) is shown in more detail according to certain embodiments described herein. Neural network chip 626 further includes interconnect circuitry 618, multiple instances of vector memory 620, vector memory control circuitry 622, STFT (Short-Time Fourier Transform) and inverse short-time Fourier transform (iSTFT) circuitry 624, and timing circuitry 632. Figure 6 All the circuits shown can be implemented on a single chip (in other words, a single semiconductor substrate / wafer).

[0034] Timing circuit 632 can be configured to control the sequence of operations performed on the chip. STFT and iSTFT circuit 624 can be configured to perform STFT on incoming audio signals (i.e., audio signals from outside the chip) and iSTFT on outgoing audio signals (i.e., audio signals destined for outside the chip). Specifically, STFT and iSTFT circuit 624 can be configured to receive audio signals from external circuitry (such as circuitry configured to process (e.g., amplify and / or filter) and digitize analog audio signals received by a microphone in an ear-worn device) and perform STFT to convert the audio signals from the time domain to the frequency domain. Vector memory control circuit 632 can be configured to control the writing of data received from STFT and iSTFT circuit 624 into vector memory 620.

[0035] Interconnect circuitry 618 can be configured to interface between vector memory 620, bias circuitry 210, and computing units 100 in computing unit array 316. Therefore, the vector memory 102 of computing unit 100 can be configured to receive elements of the activation vector from vector memory 620 via interconnect circuitry 618, and the bias memory 212 of each bias circuitry 210 can be configured to receive one or more biases from vector memory 620 via interconnect circuitry 618. As discussed above, each computing unit 100 in the column can receive the same element of X from vector memory 620. Therefore, all computing units 100 in the column can be coupled to vector memory 620 via a single shared bus 636, as... Figure 6 As shown. In some embodiments, chip 626 may not have the ability to transfer different elements of X to different computing units in the column. It should be understood that vector memory 620 is different from the circuitry in computing unit array 316.

[0036] The results of calculations performed by computation unit 100 and bias circuit 210 can be routed back to vector memory 620 via interconnect circuit 618 for storage and, in some cases, used as input for calculations representing subsequent layers of the recurrent neural network. Data fully processed by the recurrent neural network can be routed from vector memory 620 to STFT and iSTFT circuits 624 under the control of vector memory control circuit 612, where iSTFT can be performed to convert the data from the frequency domain to the time domain. The resulting signal can then be routed to an output signal generator (such as a receiver) for output as sound by an ear-worn device. (In some embodiments, STFT / iSTFT circuit 624 can be implemented off-chip.)

[0037] As in Figure 6 As seen in the examples, in some embodiments, the computing unit array 316 may lack independent connections between adjacent computing units 100 in a column. In other words, there may not be any electrical connection that connects only two computing units in a column. Instead, all computing units 100 in a column can be electrically connected via a shared bus 636. In some embodiments, computing units 100 in a column may not have the ability to output data (e.g., the result of a calculation) to another computing unit 100 in the same column. This may be due to the lack of independent connections between adjacent computing units 100 in a column. In some embodiments, computing units 100 may use routing circuitry 108 to output data (e.g., the result of a calculation) only to another data unit in the same row. While these features potentially reduce flexibility, they can help reduce the power consumption and / or area of ​​chip 626.

[0038] Chips (e.g., chip 326 and / or chip 626) can be configured to deactivate a subset of the compute units in the array to a low-power state during computations that do not require the full computational power of the compute unit array (e.g., based on the size of the input matrices and vectors (i.e., when performing matrix-vector multiplications of relatively small matrices and / or vectors)). For example, a 4x4 compute unit array can enable a 2x2 compute unit subset while deactivating the other 12 compute units to a low-power state, resulting in a 75% power saving compared to operating the full 4x4 array. Deactivating compute units can include, for example, stopping the clock signaling to the compute unit and / or putting the memory in the compute unit to sleep or power off. This can be particularly attractive for signaling links where the overhead of neural network-based speech enhancement is not always necessary, but rather a small network needs to be running continuously. For example, a signaling link could include estimating the signal-to-noise ratio (SNR) of the incoming audio and subsequently turning the entire network model on and off based on the estimated SNR. In this configuration, the SNR model can run on one computing unit of the chip, while other computing units can remain dormant, thus saving power. In some embodiments, the chip can be configured to select the number of circuit computing units to be performed based on the target denoising amount to be provided by the neural network; for example, more computing units can be used for more denoising, and fewer computing units can be used for less denoising. The target denoising amount can be based on user input.

[0039] Typically, neural network chips can be configured to run different neural networks (besides a neural network used for denoising) using different subsets of computational units. For a given neural network, the size of the subset can depend on the size of the neural network (e.g., how many weights it uses). Some neural networks can use all the computational units in the array, while others can use none. For example, a neural network chip might run a neural network to check if speech matches the speech signature of a target speaker, and / or run a neural network to estimate the direction of a given isolated speech.

[0040] One of the primary metrics used to measure performance is operations per power (Ops / Power) (e.g., trillion operations per watt (TOPs / Watt) or billion operations per milliwatt (GOPs / milliwatt)). In some embodiments, a chip can achieve an efficiency of at least approximately 4 billion operations per milliwatt. A traditional tensor processing unit (TPU) can have a power efficiency of 4 trillion operations per watt, but this only holds true when the TPU is running at trillions of operations per second (GOPs / sec), compared to the chip's ability to operate at tens of billions of operations per second (GOPs / sec) to as high as approximately 100 billion operations per second (100 GOPs / s). When a TPU is running at less than 100 billion operations per second, the 4 trillion operations per watt efficiency metric may no longer be applicable because the TPU has a large fixed power cost associated with it. This fixed cost will dominate the total power, significantly reducing efficiency. Even “low-power” accelerators are often designed to run on a phone and consume hundreds of milliwatts (mW) to several watts of power. People cannot run small models on those accelerators and cannot achieve power targets such as 1-2 mW.

[0041] As described above, the chip architecture and operation described herein enable low-power operation. In some embodiments, for a model with up to 10M 8-bit weights, and operating at 100 billion operations / second on time-series data, the entire chip can achieve a power efficiency of 4 billion operations / mW and a latency of “operations / operations per second + approximately 10 microseconds”. This can be measured at 40 degrees Celsius, with the chip using a supply voltage between 0.5 and 1.8V, and with the chip performing operations without idle time. In some embodiments, the number of operations per second required to train a recurrent neural network for denoising input audio signals and implemented on the chip results in a power consumption of no more than 2mW. In some embodiments, the number of operations per second required to train a recurrent neural network for denoising input audio signals and implemented on the chip results in a power consumption of no more than 3mW. In some embodiments, the number of operations per second required to train a recurrent neural network for denoising input audio signals and implemented on the chip results in a power consumption of no more than 4mW. Therefore, in some embodiments, the chip can consume no more than 2 mW when running a recurrent neural network trained for denoising input audio signals. In some embodiments, the chip can consume no more than 3 mW when running a recurrent neural network. In some embodiments, the chip can consume no more than 4 mW when running a recurrent neural network. With 16 compute units, each with 64 MACs and a clock frequency of 50 MHz, each of the 16 compute units can perform 64 multiplications and accumulations in a single cycle, so the maximum number of operations per second can be 64. 2 16 50 10 6 = 102.4 billion operations / second (64 2 16 50e6 -> 102.4 GOPs / Sec). As indicated herein, multiplication and accumulation are treated as two separate operations. It should be understood that the total delay between when audio is received by the earpiece and when the device outputs the processed audio, in addition to the chip's computational delay, may include delays caused by sampling the audio for a specific duration. For example, if the sampling window is tsample and the chip delay for processing that audio sample is tchip, then the total delay could be tsample + tchip.

[0042] In some embodiments, the total weighted memory of all on-chip memories may not exceed approximately 17 megabits (Mbits). In some embodiments, the total weighted memory of all on-chip memories may not exceed approximately 20 Mbits. In some embodiments, the total weighted memory of all on-chip memories may not exceed approximately 25 Mbits. In some embodiments, the total weighted memory of all on-chip memories may not exceed approximately 30 Mbits. In some embodiments, the total weighted memory of all on-chip memories may not exceed approximately 40 Mbits.

[0043] In some embodiments, the total weighted memory of all on-chip memories may exceed approximately 1 Mbits. In some embodiments, the total weighted memory of all on-chip memories may exceed approximately 4 Mbits. In some embodiments, the total weighted memory of all on-chip memories may exceed approximately 8 Mbits. In some embodiments, the total weighted memory of all on-chip memories may exceed approximately 12 Mbits.

[0044] In some embodiments, the chip area is approximately 9-14 mm². 2 Or between 9-14 mm 2 Between. In some embodiments, the chip area is approximately 9-10 mm². 2 Or between 9-10 mm 2 Between. In some embodiments, the chip area is approximately 10-11 mm². 2 Or between 10-11 mm 2 Between. In some embodiments, the chip area is approximately 11-12 mm². 2 Or between 11-12 mm2 Between. In some embodiments, the chip area is approximately 12-13 mm². 2 Or between 12-13 mm 2 Between. In some embodiments, the chip area is approximately 13-14 mm². 2 Or between 13-14 mm 2 Between. In such embodiments, the process nodes can be 12 nm, 7 nm, 5 nm, or 3 nm.

[0045] In some embodiments, the chip area is approximately 5-8 mm². 2 Or between 5-8 mm 2 Between. In some embodiments, the chip area is approximately 8-10 mm². 2 Or between 8-10 mm 2 Between. In such embodiments, the process node can be 7 nm, 5 nm, or 3 nm. In some embodiments, the chip area is approximately 6-8 mm². 2 Or between 6-8 mm 2 Between. In such embodiments, the technology process node can be 5 nm or 3 nm. In some embodiments, the chip area is approximately 5-6 mm². 2 Or between 5-6 mm 2 Between. In such embodiments, the technology process node can be 3 nm. In some embodiments, the chip area is approximately equal to or less than 20 mm². 2 .

[0046] In some embodiments, the ear-worn device includes any suitable ear-worn device, such as, for example, a hearing aid, a pair of hearing aids, headphones, a pair of earphones, face-worn glasses, a cochlear implant, or any other suitable ear-worn device. Examples of ear-worn devices are described in U.S. Patent Application Publication No. 20230232169A1, published July 20, 2023, based on a January 14, 2022 application entitled “Method, Apparatus and System for Neural Network Hearing Aid,” which is incorporated herein by reference in its entirety. In some embodiments, the ear-worn device includes components such as, for example, a neural network chip, a digital signal processing circuit (DSP) chip, one or more microphones, and one or more output signal generators, the output signal generators being configured to output sound based on the output of the neural network chip. In some embodiments, the neural network chip described herein may be, respectively, as described in U.S. Patent Application Publication No. 20230232169A1. Figure 1 , Figure 3 A, Figure 4 , Figure 5 A and Figure 5 The implementation of neural network engine (NNE) circuits 150, 350, 450, 510, and 550 shown in B.

[0047] Figure 7 A process 700 for operating a neural network on a neural network chip according to certain embodiments described herein is illustrated. Process 700 is performed by a neural network chip (e.g., neural network chip 326 and / or 626). The neural network chip may include a plurality of computing units (e.g., computing unit 100) arranged in an array of computing units (e.g., computing unit array 316). Each computing unit may include a memory (e.g., memory 128), a multiply-accumulate circuit (MAC) (e.g., MAX circuit 106), and routing circuitry (e.g., routing circuit 108). The neural network chip may be configured to implement a recurrent neural network.

[0048] In step 702, the neural network chip stores the elements of matrix A. For example, the memory of each computing unit can be configured to store a subset of the elements of matrix A. The elements of matrix A can be the weights of a recurrent neural network.

[0049] In step 704, the neural network chip receives and stores elements of vector X. For example, each computing unit may be configured to receive and store elements of vector X. Vector X may be an activation vector derived from the input audio signal to the hearing aid. In some embodiments, all computing units in a column of the computing unit array may be configured to receive the same elements of vector X.

[0050] In step 706, the neural network chip performs multiplication and accumulator sub-operations in parallel. In some embodiments, all or a subset of the computation units may be configured to perform matrix-vector multiplication A by performing multiplication and accumulator sub-operations in parallel. X.

[0051] In step 708, the neural network chip combines the results of multiplication and accumulator sub-operations. For example, the routing circuitry from each corresponding computational unit row can be configured to combine the results of multiplication and accumulator sub-operations from the corresponding computational unit rows.

[0052] Further description of neural network chips can be found in [reference needed]. Figures 1-6 The above was found. As further described above, when running a recurrent neural network, the neural network chip can be configured to achieve a power efficiency of at least 4 GOP / mW when operating at 100 GOP / sec and when the recurrent neural network includes up to 10M 8-bit weights.

[0053] Figure 8 A block diagram of an ear-worn device 802 according to certain embodiments described herein is shown. The ear-worn device 802 can be any type of ear-worn device (e.g., a hearing aid, cochlear implant, headphones, etc.) and can be any such ear-worn device described herein. The ear-worn device 802 includes one or more microphones 814, analog processing circuitry 816, digital processing circuitry 818, neural network circuitry 820, an output signal generator in the form of a receiver 822, communication circuitry 824, control circuitry 826, and a battery 828. It should be understood that the ear-worn device 802 may include more elements than shown.

[0054] One or more microphones 814 can be configured to receive sound and convert it into analog electrical signals. Analog processing circuitry 816 can be configured to receive the analog electrical signals representing the sound and perform various analog processing on them, such as pre-amplification, filtering, and analog-to-digital conversion, thereby generating digital signals. Digital processing circuitry 818 can be configured to receive the digital signals from analog processing circuitry 816 and perform various digital processing on them, such as noise reduction, beamforming, anti-feedback processing, Fourier transform, input calibration, wide dynamic range compression, output calibration, and inverse Fourier transform.

[0055] Neural network circuit 820 can be configured to receive digital signals from digital processing circuit 818 and process these signals using a neural network to perform denoising (e.g., separating speech from noise into separate sub-signals) and / or spatial focusing, as described above. While neural network circuit 820 can receive audio signals that have already been processed (e.g., by analog processing circuit 816 and digital processing circuit 818) after being received by one or more microphones 814, this can still be referred to herein as denoising of audio signals received by one or more microphones 814 by neural network circuit 820. The output of neural network circuit 820 can be routed back to digital processing circuit 818 for further processing. Receiver 822 can be configured to receive the final audio signal and output it as sound to a user.

[0056] In some embodiments, analog processing circuitry 816 may be implemented on a single chip (i.e., a single semiconductor die or substrate). In some embodiments, digital processing circuitry 818 may be implemented on a single chip. In some embodiments, neural network circuitry 820 may be implemented on a single chip. In some embodiments, analog processing circuitry 816 (or a portion thereof) and digital processing circuitry 818 (or a portion thereof) may be implemented on a single chip. In some embodiments, digital processing circuitry 818 (or a portion thereof) and neural network circuitry 820 (or a portion thereof) may be implemented on a single chip. In some embodiments, analog processing circuitry 816 (or a portion thereof), digital processing circuitry 818 (or a portion thereof), and neural network circuitry 820 (or a portion thereof) may be implemented on a single chip. In some embodiments, denoised signals output by neural network circuitry 820 on a single chip may be routed to different chips (e.g., chips including digital processing circuitry 818 and / or analog processing circuitry 816), which may then route them to receiver 822 for output to a user. In some embodiments, receiver 822 may be incorporated into a chip that also incorporates some or all of analog processing circuitry 816, digital processing circuitry 818, and neural network circuitry 820. All the chips described in this article can be found in the ear-worn device 802.

[0057] The neural network circuit 820 can be implemented on any neural network chip described herein (e.g., neural network chip 326 and / or 626). The neural network circuit 820 may include... Figures 1-6 Any or all of the circuits shown.

[0058] The communication circuit 824 can be configured to communicate with other devices via a wireless connection (such as Bluetooth, WiFi, LTE, or near-field magnetic induction (NFMI) connection). The control circuit 826 can be configured to control the operation of the analog processing circuit 816, the digital processing circuit 818, the neural network circuit 820, the communication circuit 824, and the receiver 822.

[0059] In some embodiments, a neural network implemented by neural network circuitry 820 can be trained to perform background noise reduction (also known as denoising). Training such a neural network may include obtaining a noisy speech audio signal and a speech-isolated version of the audio signal (i.e., retaining only speech). In some embodiments, a training mask that produces a speech-isolated audio signal when applied to the noisy speech audio signal can be determined. The training input data may be the noisy speech audio signal, and the training output data may be the mask. By using multiple sets of such training data during neural network training, the neural network can learn how to output a mask of the audio signal such that when the mask is applied (e.g., multiplied or added to) the audio signal, the resulting output audio signal is a speech-isolated version of the audio signal. The neural network weights obtained from such training may be those neural network weights stored in weight memory 104.

[0060] In some embodiments, a neural network implemented by neural network circuitry 820 can be trained to perform spatial focusing. Training such a neural network may include forming multiple audio signals from multiple microphones based on component audio signals arriving from different directions of arrival (DOA). In some embodiments, a training mask may be determined such that when the training mask is applied to one of the multiple audio signals, the result is that each component audio signal is multiplied by a weight corresponding to its direction of arrival from which it originates and then summed together. A particular spatial focusing pattern may indicate the weight associated with each DOA. For example, a spatial focusing pattern may assign higher weights to DOAs in front of the wearer and lower weights to DOAs behind and to the sides of the wearer. The training input data may be multiple audio signals, and the training output data may be a mask. By using multiple sets of such training data during neural network training, the neural network can learn how to output a mask based on multiple audio signals such that when the mask is applied (e.g., multiplied or added to) one of the multiple audio signals, the resulting output audio signal includes each sound component multiplied by a weight corresponding to its DOA from which it originates and then summed together. The neural network weights obtained from such training may be those neural network weights stored in weight memory 104.

[0061] Example 1 relates to an ear-worn device including a neural network chip. The neural network chip includes multiple computing units arranged in an array of computing units. Each computing unit includes a memory, multiple multiply-accumulate circuits (MACs), and routing circuitry. The memory of each computing unit is configured to store a subset of elements of a matrix A, including the weights of the neural network. Each computing unit is configured to receive and store elements of an activation vector X, where the activation vector X is derived from an input audio signal. All or a subset of the multiple computing units are configured to perform matrix-vector multiplication A by performing multiplication and accumulate suboperations in parallel among all or a subset of the multiple computing units. X. The routing circuits of the computing units in the computing unit array are configured to combine the results of multiplication and accumulation suboperations.

[0062] Example 2 relates to the ear-worn device of Example 1, wherein a computing unit among a plurality of computing units is configured to share elements of an activation vector X by pervading computations performed by a plurality of MACs in the computing unit.

[0063] Example 3 relates to the ear-worn device of Example 2, wherein a computing unit among a plurality of computing units is configured to: when computing performed by a plurality of MACs in the computing unit to share elements of the activation vector X, computing performed by all MACs in the computing unit to share elements of the activation vector X.

[0064] Example 4 relates to the ear-worn device of Example 2, wherein the computing units among the plurality of computing units are configured to share the elements of the activation vector X by means of computing performed by the plurality of MACs in the computing units within a single clock cycle when the elements of the activation vector X are shared by means of computing performed by the plurality of MACs in the computing units.

[0065] Example 5 relates to the ear-worn device of Example 2, wherein a computing unit among a plurality of computing units is configured to: when computing performed by a plurality of MACs in the computing unit to share elements of the activation vector X, the computing performed by all MACs in the computing unit within a single clock cycle shares elements of the activation vector X.

[0066] Example 6 relates to an ear-worn device of any of Examples 1-5, wherein a computing unit among a plurality of computing units is configured to fan out a single element of an activation vector X from memory to each of a plurality of MACs in the computing unit.

[0067] Example 7 relates to the ear-worn device of Example 6, wherein a computing unit among a plurality of computing units is configured to fan out a single element of the activation vector X from memory to all MACs in the computing unit when a single element of the activation vector X is fanned out from memory to each of the plurality of MACs in the computing unit.

[0068] Example 8 relates to the ear-worn device of Example 6, wherein a computing unit among a plurality of computing units is configured to simultaneously fan out a single element of the activation vector X from memory to each of the plurality of MACs in the computing unit while fanning out a single element of the activation vector X from memory to each of the plurality of MACs in the computing unit.

[0069] Example 9 relates to the ear-worn device of claim 6, wherein a computing unit among a plurality of computing units is configured to simultaneously fan out a single element of the activation vector X from memory to all MACs of the computing unit while fanning out a single element of the activation vector X from memory to each of the plurality of MACs in the computing unit.

[0070] Example 10 relates to an ear-worn device of any of Examples 2-9, wherein the plurality of MACs in the computing unit includes all the MACs in the computing unit.

[0071] Example 11 relates to an ear-worn device of any of Examples 1-10, wherein a group of computing units in a computing unit array is coupled to a vector memory via a single shared bus.

[0072] Example 12 relates to the ear-worn device of Example 11, wherein each group of computing units comprises computing units in a single column of an array of computing units.

[0073] Example 13 relates to an ear-worn device of any of Examples 1-12, wherein at least one computing unit in the computing unit array is not configured to transmit data to at least one adjacent computing unit in the computing unit array.

[0074] Example 14 relates to the ear-worn device of Example 13, wherein at least one computing unit in the computing unit array is not configured to transmit data to another computing unit in the same column of the computing unit array.

[0075] Example 15 relates to an ear-worn device of any of Examples 13-14, wherein the computing unit array lacks independent connections between adjacent computing units in columns of the computing unit array.

[0076] Example 16 relates to an ear-worn device of any of Examples 1-15, wherein all the memory in the plurality of computing units includes a total of no more than about 40 Mbit of memory for weights of the neural network.

[0077] Example 17 relates to an ear-worn device of any of Examples 1-16, wherein the neural network chip is configured to achieve a power efficiency of at least 4 billion operations per milliwatt when operating at 100 billion operations per second, the neural network includes up to 10M 8-bit weights, performs 100 billion operations per second on time-series data, and the neural network chip operates at 40 degrees Celsius, uses a supply voltage between 0.5 and 1.8V, and performs operations without idle time.

[0078] Example 18 relates to an ear-worn device of any of Examples 1 to 17, wherein the neural network chip has a diameter of less than 20 mm. 2 The area.

[0079] Example 19 relates to an ear-worn device of any of Examples 1-18, wherein the routing circuitry of the computing units in the computing unit array is configured to combine the results of multiplication and accumulation sub-operations from each corresponding row of computing units in the computing unit array when combining the results of multiplication and accumulation sub-operations.

[0080] Example 20 relates to an ear-worn device of any of Examples 1 to 19, wherein each computing unit includes 16 to 128 MACs or equal to 16 to 128 MACs.

[0081] Example 21 relates to an ear-worn device of any of Examples 1-20, wherein the neural network includes a recurrent neural network.

[0082] Example 22 relates to an ear-worn device of any of Examples 1-21, wherein all computing units in a column of an array are configured to receive the same element of X.

[0083] Example 23 relates to an ear-worn device of any of Examples 1-22, wherein the memory and MAC of any given computing unit are arranged in an area no larger than 0.25 mm. 2 Within the area.

[0084] Example 24 relates to an ear-worn device of any of Examples 1-23, wherein the neural network chip further includes a plurality of bias circuits, each bias circuit being electrically coupled to a row of computing units in a plurality of computing units in a computing unit array, and including a bias memory and routing circuitry; each of the plurality of bias circuits is configured to receive one or more biases and store them in the bias memory; and the routing circuitry of the computing units in the corresponding row of computing units in the computing unit array and the routing circuitry of the bias circuitry electrically coupled to the corresponding row of computing units are configured to combine the results of multiplication and accumulator suboperations with one or more biases.

[0085] Example 25 relates to an ear-worn device of any of Examples 1-24, wherein the neural network chip further includes a short-time Fourier transform (STFT) circuit and an inverse short-time Fourier transform (iSTFT) circuit, which are configured to perform STFT on an audio signal from outside the chip and to perform iSTFT on an audio signal destined for outside the chip, respectively.

[0086] Example 26 relates to the ear-worn device of Example 25, wherein the activation vector X of the first layer of the neural network is the result of processing an input audio signal from outside the chip using an STFT circuit.

[0087] Example 27 relates to an ear-worn device of any of Examples 1-26, wherein a neural network is configured to perform denoising on an input audio signal.

[0088] Example 28 relates to an ear-worn device of any of Examples 1-27, wherein the neural network chip is configured to disable a subset of multiple computing units in the array of computing units.

[0089] Example 29 relates to the ear-worn device of Example 28, wherein the neural network chip is configured to deactivate a subset of multiple computing units in the array of computing units based on the size of matrix A and / or the size of activation vector X.

[0090] Example 30 relates to the ear-worn device of Example 28, wherein the neural network chip is further configured to deactivate a subset of multiple computing units in the computing unit array when estimating the signal-to-noise ratio (SNR) of the input audio signal.

[0091] Example 31 relates to the ear-worn device of Example 30, wherein the neural network chip is further configured to utilize a computing unit to estimate the SNR of the input audio signal.

[0092] Example 32 relates to an ear-worn device of any of Examples 28-30, wherein the neural network chip is further configured to select a subset of multiple computing units in the computing unit array to be deactivated based on a target amount of denoising to be provided by the neural network.

[0093] Example 33 relates to an ear-worn device of any of Examples 1-32, wherein a neural network is configured to perform spatial focusing of an input audio signal.

[0094] Example 34 relates to an ear-worn device of any of Examples 31-32, and further includes one or more output signal generators configured to output sound based on the output from a neural network.

[0095] Example 35 relates to an ear-worn device of any of Examples 1-34, further including one or more microphones configured to convert received sound into an input audio signal.

[0096] Example 36 relates to an ear-worn device of any of Examples 1-32, wherein the ear-worn device includes a hearing aid.

[0097] Example 37 relates to a neural network chip including multiple computing units arranged in an array of computing units, each computing unit including a memory, multiple multiply-accumulate circuits (MACs), and routing circuitry. The memory of each computing unit is configured to store a subset of elements of a matrix A, including the weights of the neural network. Each computing unit is configured to receive and store elements of an activation vector X, wherein the activation vector X is derived from an input signal. All or a subset of the multiple computing units are configured to perform matrix-vector multiplication A by performing multiplication and accumulate suboperations in parallel among all or a subset of the multiple computing units. X. The routing circuits of the computing units in the computing unit array are configured to combine the results of multiplication and accumulation suboperations.

[0098] Example 38 relates to the neural network chip of Example 37, wherein the computation units in a plurality of computation units are configured to share elements of the activation vector X by performing computations performed by a plurality of MACs in the computation units.

[0099] Example 39 relates to the neural network chip of Example 38, wherein the computation units in a plurality of computation units are configured to share the elements of the activation vector X among all the computation units when the computation performed by the plurality of MACs in the computation unit shares the elements of the activation vector X.

[0100] Example 40 relates to the neural network chip of Example 38, wherein the computing units among the plurality of computing units are configured to share the elements of the activation vector X among the computing units when the computing units perform the computing within a single clock cycle to share the elements of the activation vector X.

[0101] Example 41 relates to the neural network chip of Example 38, wherein the computing units among a plurality of computing units are configured to share the elements of the activation vector X by all the computing units performing the computation within a single clock cycle when the elements of the activation vector X are shared by the computation performed by the plurality of MACs in the computing unit.

[0102] Example 42 relates to a neural network chip of any of Examples 37-541, wherein a computational unit among a plurality of computational units is configured to fan out a single element of an activation vector X from memory to each of a plurality of MACs in the computational unit.

[0103] Example 43 relates to the neural network chip of Example 42, wherein a computing unit among a plurality of computing units is configured to fan out a single element of the activation vector X from memory to all MACs in the computing unit when fanning out a single element of the activation vector X from memory to each of the plurality of MACs in the computing unit.

[0104] Example 44 relates to the neural network chip of Example 42, wherein a computing unit among a plurality of computing units is configured to simultaneously fan out a single element of the activation vector X from memory to each of the plurality of MACs in the computing unit when a single element of the activation vector X is fanned out from memory to each of the plurality of MACs in the computing unit.

[0105] Example 45 relates to the neural network chip of claim 42, wherein a computing unit among a plurality of computing units is configured to simultaneously fan out a single element of the activation vector X from memory to all MACs of the computing unit while fanning out a single element of the activation vector X from memory to each of the plurality of MACs in the computing unit.

[0106] Example 46 relates to a neural network chip of any of Examples 38 to 45, wherein the plurality of MACs in the computing unit includes all the MACs in the computing unit.

[0107] Example 47 relates to a neural network chip of any of Examples 37 to 46, wherein a group of computing units in a computing unit array is coupled to a vector memory by only a single shared bus.

[0108] Example 48 relates to the neural network chip of Example 47, wherein each group of computing units comprises computing units in a single column of an array of computing units.

[0109] Example 49 relates to a neural network chip of any of Examples 37-48, wherein at least one computing unit in the array of computing units is not configured to transmit data to at least one adjacent computing unit in the array of computing units.

[0110] Example 50 relates to the neural network chip of Example 49, wherein at least one computing unit in the array of computing units is not configured to transmit data to another computing unit in the same column of the array of computing units.

[0111] Example 51 relates to a neural network chip of any of Examples 49-50, wherein the array of computing units lacks independent connections between adjacent computing units in columns of the array of computing units.

[0112] Example 52 relates to a neural network chip of any of Examples 37-51, wherein all the memory in the plurality of computing units includes a total of no more than about 40 Mbit of memory for the weights of the neural network.

[0113] Example 53 relates to a neural network chip of any of Examples 37 to 52, wherein the neural network chip is configured to achieve a power efficiency of at least 4 billion operations per milliwatt when operating at 100 billion operations per second, the neural network includes 8-bit weights up to 10M, performs 100 billion operations per second on time series data, and the neural network chip operates at 40 degrees Celsius, uses a supply voltage between 0.5 and 1.8V, and performs operations without idle time.

[0114] Example 54 relates to a neural network chip of any of Examples 37 to 53, wherein the neural network chip has a diameter of less than 20 mm. 2 The area.

[0115] Example 55 relates to a neural network chip of any of Examples 37-54, wherein the routing circuitry of the computational units in the computational unit array is configured to combine the results of multiplication and accumulator suboperations from each corresponding row of computational units in the computational unit array when combining the results of multiplication and accumulator suboperations.

[0116] Example 56 relates to a neural network chip of any of Examples 37 to 55, wherein each computing unit includes 16 to 128 MACs or equal to 16 to 128 MACs.

[0117] Example 57 relates to a neural network chip of any of Examples 37 to 56, wherein the neural network includes a recurrent neural network.

[0118] Example 58 relates to a neural network chip of any of Examples 37 to 57, wherein all computational units in a column of an array are configured to receive the same element of X.

[0119] Example 59 relates to a neural network chip of any of Examples 37 to 58, wherein the memory and MAC of any given computing unit are arranged in an area no larger than 0.25 mm. 2 Within the area.

[0120] Example 60 relates to a neural network chip of any of Examples 37-59, wherein the neural network chip further includes a plurality of bias circuits, each bias circuit being electrically coupled to a row of computing units in a plurality of computing units in a computing unit array, and including a bias memory and routing circuitry; each of the plurality of bias circuits is configured to receive one or more biases and store them in the bias memory; and the routing circuitry of the computing units in the corresponding row of computing units in the computing unit array and the routing circuitry of the bias circuitry electrically coupled to the corresponding row of computing units are configured to combine the results of multiplication and accumulator suboperations with one or more biases.

[0121] Example 61 relates to a neural network chip of any of Examples 37-60, wherein the neural network chip further includes a short-time Fourier transform (STFT) circuit and an inverse short-time Fourier transform (iSTFT) circuit, which are configured to perform STFT on an audio signal from outside the chip and to perform iSTFT on an audio signal destined for outside the chip, respectively.

[0122] Example 62 relates to the neural network chip of Example 61, wherein the activation vector X of the first layer of the neural network is the result of processing an input signal from outside the chip using an STFT circuit.

[0123] Example 63 relates to a neural network chip of any of Examples 37 to 62, wherein the neural network is configured to denoise an audio signal.

[0124] Example 64 relates to a neural network chip of any of Examples 37 to 63, wherein the neural network chip is configured to disable a subset of a plurality of computing units in an array of computing units.

[0125] Example 65 relates to the neural network chip of Example 64, wherein the neural network chip is configured to deactivate a subset of multiple computing units in an array of computing units based on the size of matrix A and / or the size of activation vector X.

[0126] Example 66 relates to the neural network chip of Example 64, wherein the neural network chip is further configured to deactivate a subset of multiple computing units in the computing unit array when estimating the signal-to-noise ratio (SNR) of the input signal.

[0127] Example 67 relates to the neural network chip of Example 66, wherein the neural network chip is further configured to estimate the SNR of the input signal using a computing unit.

[0128] Example 68 relates to a neural network chip of any of Examples 64-66, wherein the neural network chip is further configured to select a subset of a plurality of computing units in a computing unit array to be deactivated based on a target amount of denoising to be provided by the neural network.

[0129] Example 69 relates to a neural network chip of any of Examples 37-68, wherein the input signal includes an input audio signal.

[0130] Example 70 relates to the neural network chip of Example 69, wherein the neural network is configured to perform spatial focusing of an input audio signal.

[0131] Example 71 relates to an ear-worn device that includes a neural network chip from any of Examples 37-70.

[0132] Example 72 relates to the ear-worn device of Example 71, and further includes one or more output signal generators configured to output sound based on the output from a neural network.

[0133] Example 73 relates to an ear-worn device of any of Examples 71-72, and further includes one or more microphones configured to receive sound for conversion into an input audio signal.

[0134] Example 74 relates to an ear-worn device of any of Examples 71-73, wherein the ear-worn device includes a hearing aid.

[0135] Several embodiments of these technologies have been described in detail, and various modifications and improvements will readily occur to those skilled in the art. Such modifications and improvements are intended to fall within the spirit and scope of the invention. Therefore, the above description is merely exemplary and not intended to be limiting. For example, any of the components described above may include hardware, software, or a combination of hardware and software.

[0136] Unless otherwise expressly stated otherwise, the indefinite articles “a” and “an” as used herein in the specification and claims shall be understood to mean “at least one”.

[0137] As used herein in the specification and claims, the phrase “and / or” should be understood to mean “any one or both” of the elements so combined, i.e., elements that are combined in some cases and separate in others. Multiple elements listed with “and / or” should be interpreted in the same way, i.e., “one or more” of the elements so combined. Optionally, other elements may exist besides those specifically identified by the “and / or” clause, whether related to or unrelated to those specifically identified.

[0138] As used herein in the specification and claims, the phrase "at least one" referring to a list of one or more elements should be understood to mean at least one element selected from any one or more elements in the list, but not necessarily including at least one of every element specifically listed in the list, and does not exclude any combination of elements in the list. This definition also allows for the optional presence of elements, whether related to or unrelated to those specifically specified elements, in addition to those specifically designated in the list of elements referred to by the phrase "at least one".

[0139] In some embodiments, the terms "about" and "approximately" may be used to refer to within ±20% of the target value, within ±10% of the target value in some embodiments, within ±5% of the target value in some embodiments, and within ±2% of the target value in some embodiments. The terms "about" and "approximately" may include the target value.

[0140] Furthermore, the wording and terminology used herein are for descriptive purposes and should not be considered restrictive. The use of “including,” “comprising,” “having,” “containing,” “involving,” and variations thereof is intended to cover the items listed thereafter and their equivalents, as well as other items.

[0141] Several aspects of at least one embodiment have been described above. It should be understood that various changes, modifications, and improvements will readily occur to those skilled in the art. Such changes, modifications, and improvements are intended for the purposes of this disclosure. Therefore, the foregoing description and figures are merely examples.

Claims

1. An ear-worn device, comprising: Neural network chips, including: Multiple computing units are arranged in a computing unit array, each computing unit including a memory, multiple multiply-accumulate circuits (MAC) and routing circuits; in: The memory of each computing unit is configured to store a subset of elements of matrix A, which includes the weights of the neural network; Each computing unit is configured to receive and store elements of an activation vector X, wherein the activation vector X is derived from an input audio signal; All or a subset of the plurality of computing units are configured to perform matrix-vector multiplication A by performing multiplication and accumulation suboperations in parallel among all or a subset of the plurality of computing units. X; and The routing circuits of the computing units in the computing unit array are configured to combine the results of multiplication and accumulation sub-operations.

2. The ear-worn device according to claim 1, wherein, The computational units among the plurality of computational units are configured to share elements of the activation vector X by performing computations performed by the plurality of MACs in the computational units.

3. The ear-worn device according to claim 2, wherein, The computing units in the plurality of computing units are configured to share the elements of the activation vector X by means of computing performed by all MACs in the computing units when the elements of the activation vector X are shared by means of computing performed ...

4. The ear-worn device according to claim 2, wherein, The computing units in the plurality of computing units are configured to share the elements of the activation vector X by means of computing performed by the plurality of MACs in the computing units within a single clock cycle when the elements of the activation vector X are shared by means of computing performed by the plurality of MACs in the computing units.

5. The ear-worn device according to claim 2, wherein, The computing units in the plurality of computing units are configured to share the elements of the activation vector X by means of computing performed by all MACs in the computing units within a single clock cycle when the elements of the activation vector X are shared across computing performed by multiple MACs in the computing unit.

6. The ear-worn device according to any one of claims 1 to 5, wherein, The computing units of the plurality of computing units are configured to fan out a single element of the activation vector X from the memory to each of the plurality of MACs in the computing unit.

7. The ear-worn device according to claim 6, wherein, The computing unit among the plurality of computing units is configured to: when fan-out a single element of the activation vector X from the memory to each of the plurality of MACs in the computing unit, fan out a single element of the activation vector X from the memory to all MACs in the computing unit.

8. The ear-worn device according to claim 6, wherein, The computing unit among the plurality of computing units is configured to: simultaneously fan out a single element of the activation vector X from the memory to each of the plurality of MACs in the computing unit while fanning out a single element of the activation vector X from the memory to each of the plurality of MACs in the computing unit.

9. The ear-worn device according to claim 6, wherein, The computing unit among the plurality of computing units is configured to: while fan out a single element of the activation vector X from the memory to each of the plurality of MACs in the computing unit, simultaneously fan out a single element of the activation vector X from the memory to all MACs in the computing unit.

10. The ear-worn device according to any one of claims 2 to 9, wherein, The multiple MACs in the computing unit include all MACs in the computing unit.

11. The ear-worn device according to any one of claims 1 to 10, wherein, The computing unit groups in the computing unit array are coupled to the vector memory only through a single shared bus.

12. The ear-worn device according to claim 11, wherein, Each of the computing unit groups comprises computing units in a single column of the computing unit array.

13. The ear-worn device according to any one of claims 1 to 12, wherein, At least one computing unit in the computing unit array is not configured to transmit data to at least one adjacent computing unit in the computing unit array.

14. The ear-worn device according to claim 13, wherein, At least one computing unit in the computing unit array is not configured to transmit data to another computing unit in the same column of the computing unit array.

15. The ear-worn device according to any one of claims 13 to 14, wherein, The computing unit array lacks independent connections between adjacent computing units in the columns of the computing unit array.

16. The ear-worn device according to any one of claims 1 to 15, wherein, The total memory in all the plurality of computing units includes no more than about 40 Mbit of memory for the weights of the neural network.

17. The ear-worn device according to any one of claims 1 to 16, wherein: The neural network chip is configured to achieve a power efficiency of at least 4 billion operations per milliwatt when operating at 100 billion operations per second. The neural network includes up to 10M 8-bit weights; Perform 100 billion operations per second on time series data; and The neural network chip operates at 40 degrees Celsius using a power supply voltage between 0.5 and 1.8V, and performs operations without idle time.

18. The ear-worn device according to any one of claims 1 to 17, wherein, The neural network chip has a diameter of less than 20 mm. 2 The area.

19. The ear-worn device according to any one of claims 1 to 18, wherein, The routing circuits of the computing units in the computing unit array are configured to combine the results of the multiplication and accumulation sub-operations from each corresponding row of computing units in the computing unit array when combining the results of the multiplication and accumulation sub-operations.

20. The ear-worn device according to any one of claims 1 to 19, wherein, Each computing unit includes 16-128 MACs or equal to 16-128 MACs.

21. The ear-worn device according to any one of claims 1 to 20, wherein, The neural network includes a recurrent neural network.

22. The ear-worn device according to any one of claims 1 to 21, wherein, All computing units in the columns of the array are configured to receive the same element of X.

23. The ear-worn device according to any one of claims 1 to 22, wherein, The memory and MAC of any given computing unit are arranged in a space no larger than 0.25 mm. 2 Within the area.

24. The ear-worn device according to any one of claims 1 to 23, wherein: The neural network chip also includes multiple bias circuits, each bias circuit being electrically coupled to a row of computing units in the array of computing units, and including a bias memory and a routing circuit. Each of the plurality of bias circuits is configured to receive one or more biases and store them in the bias memory; as well as The routing circuits of the computing units in the corresponding computing unit row of the computing unit array and the routing circuits of the bias circuits electrically coupled to the corresponding computing unit row are configured to combine the results of the multiplication and accumulation sub-operations with one or more biases.

25. The ear-worn device according to any one of claims 1 to 24, wherein, The neural network chip also includes a short-time Fourier transform (STFT) circuit and an inverse short-time Fourier transform (iSTFT) circuit, which are configured to perform STFT on audio signals from outside the chip and iSTFT on audio signals destined for outside the chip, respectively.

26. The ear-worn device according to claim 25, wherein, The activation vector X used for the first layer of the neural network is the result of processing an input audio signal from outside the chip using the STFT circuit.

27. The ear-worn device according to any one of claims 1 to 26, wherein, The neural network is configured to perform denoising on the input audio signal.

28. The ear-worn device according to any one of claims 1 to 27, wherein, The neural network chip is configured to disable a subset of multiple computing units in the computing unit array.

29. The ear-worn device according to claim 28, wherein, The neural network chip is configured to deactivate a subset of multiple computing units in the computing unit array based on the size of the matrix A and / or the size of the activation vector X.

30. The ear-worn device according to claim 28, wherein, The neural network chip is also configured to disable a subset of multiple computing units in the computing unit array when estimating the signal-to-noise ratio (SNR) of the input audio signal.

31. The ear-worn device according to claim 30, wherein, The neural network chip is also configured to use a computing unit to estimate the SNR of the input audio signal.

32. The ear-worn device according to any one of claims 28 to 30, wherein, The neural network chip is also configured to select a subset of multiple computing units in the computing unit array to be deactivated based on the target denoising amount provided by the neural network.

33. The ear-worn device according to any one of claims 1 to 32, wherein, The neural network is configured to perform spatial focusing on the input audio signal.

34. The ear-worn device according to any one of claims 1 to 33, further comprising one or more output signal generators configured to output sound based on the output from the neural network.

35. The ear-worn device according to any one of claims 1 to 34, further comprising one or more microphones configured to receive sound for conversion into the input audio signal.

36. The ear-worn device according to any one of claims 1 to 35, wherein, The ear-worn device includes a hearing aid.

37. A neural network chip, comprising: Multiple computing units are arranged in a computing unit array, each computing unit including a memory, multiple multiply-accumulate circuits (MAC) and routing circuits; in: The memory of each computing unit is configured to store a subset of elements of matrix A, which includes the weights of the neural network; Each computing unit is configured to receive and store elements of an activation vector X, wherein the activation vector X is derived from an input signal; All or a subset of the plurality of computing units are configured to perform matrix-vector multiplication A by performing multiplication and accumulation suboperations in parallel among all or a subset of the plurality of computing units. X; and The routing circuits of the computing units in the computing unit array are configured to combine the results of multiplication and accumulation sub-operations.

38. The neural network chip according to claim 37, wherein, The computational units among the plurality of computational units are configured to share elements of the activation vector X by performing computations performed by the plurality of MACs in the computational units.

39. The neural network chip according to claim 38, wherein, The computing units in the plurality of computing units are configured to share the elements of the activation vector X by means of computing performed by all MACs in the computing units when the elements of the activation vector X are shared by means of computing performed ...

40. The neural network chip according to claim 38, wherein, The computing units in the plurality of computing units are configured to share the elements of the activation vector X by means of computing performed by the plurality of MACs in the computing units within a single clock cycle when the elements of the activation vector X are shared by means of computing performed by the plurality of MACs in the computing units.

41. The neural network chip according to claim 38, wherein, The computing units in the plurality of computing units are configured to share the elements of the activation vector X by means of computing performed by all MACs in the computing units within a single clock cycle when the elements of the activation vector X are shared across computing performed by multiple MACs in the computing unit.

42. The neural network chip according to any one of claims 37 to 41, wherein, The computing units of the plurality of computing units are configured to fan out a single element of the activation vector X from the memory to each of the plurality of MACs in the computing unit.

43. The neural network chip according to claim 42, wherein, The computing unit among the plurality of computing units is configured to: when fan-out a single element of the activation vector X from the memory to each of the plurality of MACs in the computing unit, fan out a single element of the activation vector X from the memory to all MACs in the computing unit.

44. The neural network chip according to claim 42, wherein, The computing unit among the plurality of computing units is configured to: simultaneously fan out a single element of the activation vector X from the memory to each of the plurality of MACs in the computing unit while fanning out a single element of the activation vector X from the memory to each of the plurality of MACs in the computing unit.

45. The neural network chip according to claim 42, wherein, The computing unit among the plurality of computing units is configured to: while fan out a single element of the activation vector X from the memory to each of the plurality of MACs in the computing unit, simultaneously fan out a single element of the activation vector X from the memory to all MACs in the computing unit.

46. ​​The neural network chip according to any one of claims 38 to 45, wherein, The multiple MACs in the computing unit include all MACs in the computing unit.

47. The neural network chip according to any one of claims 37 to 46, wherein, The computing unit groups in the computing unit array are coupled to the vector memory only through a single shared bus.

48. The neural network chip according to claim 47, wherein, Each of the computing unit groups comprises computing units in a single column of the computing unit array.

49. The neural network chip according to any one of claims 37 to 48, wherein, At least one computing unit in the computing unit array is not configured to transmit data to at least one adjacent computing unit in the computing unit array.

50. The neural network chip according to claim 49, wherein, At least one computing unit in the computing unit array is not configured to transmit data to another computing unit in the same column of the computing unit array.

51. The neural network chip according to any one of claims 49 to 50, wherein, The computing unit array lacks independent connections between adjacent computing units in the columns of the computing unit array.

52. The neural network chip according to any one of claims 37 to 51, wherein, The total memory in all the plurality of computing units includes no more than about 40 Mbit of memory for the weights of the neural network.

53. The neural network chip according to any one of claims 37 to 52, wherein, The neural network chip is configured to achieve a power efficiency of at least 4 billion operations per milliwatt when operating at 100 billion operations per second. The neural network includes up to 10M 8-bit weights, performs 100 billion operations per second on time-series data, and the neural network chip operates at 40 degrees Celsius using a power supply voltage between 0.5 and 1.8V without idle time.

54. The neural network chip according to any one of claims 37 to 53, wherein, The neural network chip has a diameter of less than 20 mm. 2 The area.

55. The neural network chip according to any one of claims 37 to 54, wherein, The routing circuits of the computing units in the computing unit array are configured to combine the results of the multiplication and accumulation sub-operations from each corresponding row of computing units in the computing unit array when combining the results of the multiplication and accumulation sub-operations.

56. The neural network chip according to any one of claims 37 to 55, wherein, Each computing unit includes 16-128 MACs or equal to 16-128 MACs.

57. The neural network chip according to any one of claims 37 to 56, wherein, The neural network includes a recurrent neural network.

58. The neural network chip according to any one of claims 37 to 57, wherein, All computing units in the columns of the array are configured to receive the same element of X.

59. The neural network chip according to any one of claims 37 to 58, wherein, The memory and MAC of any given computing unit are arranged in a space no larger than 0.25 mm. 2 Within the area.

60. The neural network chip according to any one of claims 37 to 59, wherein: The neural network chip also includes multiple bias circuits, each bias circuit being electrically coupled to a row of computing units in the array of computing units, and including a bias memory and a routing circuit. Each of the plurality of bias circuits is configured to receive one or more biases and store them in the bias memory; as well as The routing circuits of the computing units in the corresponding computing unit row of the computing unit array and the routing circuits of the bias circuits electrically coupled to the corresponding computing unit row are configured to combine the results of the multiplication and accumulation sub-operations with one or more biases.

61. The neural network chip according to any one of claims 37 to 60, wherein, The neural network chip also includes a short-time Fourier transform (STFT) circuit and an inverse short-time Fourier transform (iSTFT) circuit, which are configured to perform STFT on signals from outside the chip and iSTFT on signals destined for outside the chip, respectively.

62. The neural network chip as described in claim 61, wherein, The activation vector X used for the first layer of the neural network is the result of processing input signals from outside the chip using the STFT circuit.

63. The neural network chip according to any one of claims 37 to 62, wherein, The neural network is configured to perform noise reduction on audio signals.

64. The neural network chip according to any one of claims 37 to 63, wherein, The neural network chip is configured to disable a subset of multiple computing units in the computing unit array.

65. The neural network chip according to claim 64, wherein, The neural network chip is configured to deactivate a subset of multiple computing units in the computing unit array based on the size of the matrix A and / or the size of the activation vector X.

66. The neural network chip according to claim 64, wherein, The neural network chip is also configured to disable a subset of multiple computing units in the computing unit array when estimating the signal-to-noise ratio (SNR) of the input signal.

67. The neural network chip according to claim 66, wherein, The neural network chip is also configured to estimate the SNR of the input signal using a computing unit.

68. The neural network chip according to any one of claims 64 to 66, wherein, The neural network chip is also configured to select a subset of multiple computing units in the computing unit array to be deactivated based on the target denoising amount provided by the neural network.

69. The neural network chip according to any one of claims 37 to 68, wherein, The input signal includes the input audio signal.

70. The neural network chip according to claim 69, wherein, The neural network is configured to perform spatial focusing on the input audio signal.

71. An ear-worn device comprising a neural network chip according to any one of claims 37 to 70.

72. The ear-worn device of claim 71, further comprising one or more output signal generators configured to output sound based on the output from the neural network.

73. The ear-worn device according to any one of claims 71 to 72 further includes one or more microphones configured to receive sound for conversion into an input audio signal.

74. The ear-worn device according to any one of claims 71 to 73, wherein, The ear-worn device includes a hearing aid.

Images