Neural network system and accelerator for implementing the neural network
By designing the accelerator on the chip to share resources with the host processor, using weight compression and parallel computing, the problems of slow neural network processing speed and high power consumption are solved, and efficient neural network processing at low power is realized, suitable for real-time applications such as Discourse recognition.
Patent Information
- Application Number
- CN201910363399.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2019-04-16
- Filing Date
- 2019-04-30
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2039-04-30
AI Technical Summary
The prior art is slow in performing neural network processing and consumes too much power, making it difficult to meet the needs of real-time applications such as speech recognition.
By designing the accelerator on the chip to share resources with the host processor, using weight compression and parallel computing, reduce memory and power consumption, and process as input data is stable to save power.
It realizes efficient execution of neural network processing at low power, reduces the power consumption of computing neural networks, and is suitable for miniaturized and long-lived devices such as hearing aids, improving real-time recognition capabilities.
Smart Images

Figure CN110428047B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to systems for neural networks and more particularly to neural network implementations on a system-on-a-chip. Background Art
[0002] Neural networks are tools of artificial intelligence that can be used in digital signal processing applications that use a processor or digital signal processor (DSP) to perform complex recognition functions. Processors can be configured by software instructions to compute (i.e., process) neural networks, but may be too slow for many real-time applications such as speech recognition. DSPs may be specialized microprocessors with an architecture optimized for repeatedly running algorithms in real time over received data. DSPs may use fewer resources (e.g., power and memory) than processors for computing devices (e.g., ARM processors), but may consume too much power for low-power applications. Therefore, there is a need for new hardware and software systems that perform neural network processing at improved speeds and lower power consumption. Summary of the invention
[0003] The present invention generally describes a method for computing a neural network. The disclosed method uses an accelerator and a host processor (i.e., a processor) that share resources, such as a bus and memory, so that various operations of the computation, such as applying an activation function, can be performed in hardware (i.e., on the accelerator) or in software (i.e., running on the processor). The disclosed method also saves memory and power by reducing the amount of data used to represent the weights of the neural network (i.e., caused by training). The reduction is produced by pruning weights that are effectively zero (e.g., have values below a threshold) and storing the remaining weights in a compressed form. Accordingly, the disclosed accelerator is configured to decompress the compressed weights using a (e.g., 4-bit to 8-bit) lookup table. The disclosed method also saves power by computing portions of the neural network in parallel and keeping the input data stable during the computation. Keeping the input data stable also reduces fetch operations to save power.
[0004] In one general aspect, the present disclosure describes a method for implementing a neural network. The method includes receiving input data and extracting weights of the neural network from a memory (e.g., compressed weights, weights including compressed weights, or uncompressed weights). The method also includes performing a first portion of processing for the neural network. The first portion of processing is implemented in hardware by an accelerator. The method also includes performing a second portion of processing for the neural network. The second portion of processing is implemented in software by a processor. In the method, the accelerator and the processor use a bus to communicate and share access to the memory.
[0005] In another general aspect, the disclosure describes a neural network system. The system includes a memory configured to store compressed weights of a neural network. The system also includes a processor data bus coupled between the memory and a processor. The system also includes an accelerator coupled to the processor and sharing the processor data bus with the processor. The accelerator is configured to extract and decompress the compressed weights of the neural network from the memory and perform at least a portion of the processing for the neural network while the processor performs other tasks. The portion of the processing includes a plurality of multiply and accumulate (MAC) operations performed in parallel.
[0006] In another general aspect, the present invention describes an accelerator for implementing a neural network. The accelerator includes a plurality of multiply and accumulate (MAC) units operating in parallel. Each MAC unit is configured to repeatedly multiply input values and weights to accumulate all sums of products representing values corresponding to neurons in the neural network. The accelerator also includes a lookup table for decompressing compressed weights stored in a memory to generate weights for each MAC unit at each repetition. The accelerator also includes a circular buffer that feeds input values to each unit at each repetition.
[0007] The foregoing illustrative summary as well as other exemplary objects and / or advantages of the present invention and the manner in which they are achieved are further explained in the following detailed description and its accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0008] Figure 1 is a graphical description of an example neural network that can be used in this article.
[0009] Figure 2 is a block diagram of a neural network system according to an embodiment of the present invention.
[0010] Figure 3 is a hardware block diagram of an accelerator for a neural network according to a possible implementation of the present invention.
[0011] Figure 4 is a functional block diagram of a possible stand-alone implementation of an accelerator for a neural network.
[0012] Figure 5 is a functional block diagram of a possible implementation of an accelerator for a neural network in which activation functions are implemented in software.
[0013] Figure 6 is a functional block diagram of a possible implementation for an accelerator based on a neural network in which activation functions may be implemented in hardware or software.
[0014] Figure 7 is a flow chart of a method for implementing a neural network according to a possible embodiment of the present invention.
[0015] The components shown in the drawings are not necessarily drawn to scale relative to each other. The same reference numerals indicate corresponding parts throughout the several views. DETAILED DESCRIPTION
[0016] The present disclosure generally describes a device configured to implement a neural network in a mobile application (e.g., a hearing aid) to improve speech recognition (e.g., voice commands) or sound recognition (e.g., background noise type) in a power efficient manner (e.g., to save battery life). The device includes an accelerator as a semiconductor (i.e., hardware) platform (i.e., block) that assists a processor in implementing an artificial intelligence (AI) algorithm (e.g., a neural network). The accelerator includes hard-coded logic and mathematical functions that can be controlled (e.g., by a state machine configured by a processor) to process the neural network. The accelerator can process the neural network faster and more (power) efficiently than conventional software running on, for example, a DSP. The DSP approach may require additional processing / power resources to fetch software instructions, perform calculations serially, and perform calculations using a much higher bit depth than may be desirable for a particular application (e.g., a hearing aid). Instead, the disclosed accelerator avoids the fetching of software instructions, performs processing (e.g., calculations) in parallel, and processes using a bit depth appropriate for the neural network for a particular application (e.g., a hearing aid).
[0017] In a possible application, a neural network may require more than 200,000 cycles to obtain an output, and the output must be obtained every 10 milliseconds (ms). These requirements may result in clock speeds of approximately 20 megahertz, which is too high for a typical low-power DSP used in a hearing aid. The disclosed method provides an apparatus and method for implementing neural networks in applications with the same (or similar) requirements, making these applications smaller in size and low-power.
[0018] One feature of the disclosed method is that the use of accelerators allows neural network processing to be performed in software (i.e., running on a processor) or in hardware (i.e., in an accelerator) to facilitate versatility. An accelerator can be generally described as on-chip dedicated logic (i.e., a block of hardware) that performs a specific set of functions that is more efficient (e.g., faster) than executing on an instruction set processor (i.e., because the software is running on the processor).
[0019] Another feature of the disclosed method is that the accelerator and the processor can share resources (e.g., buses and memory) to save time, complexity, and cost. Another feature of the disclosed method is that the accelerator promotes techniques (e.g., synaptic weight compression and synaptic pruning) to save memory and processing. Another feature of the disclosed method is that the accelerator promotes parallelism (e.g., multiple multiplication and accumulation units and circular buffers) to save power.
[0020] A neural network accelerator may be implemented as part of a hearing aid. The hearing aid may utilize a neural network to improve recognition of commands spoken by a user. Based on the recognized command (e.g., turn up the volume), the hearing aid may perform a function (e.g., increase the volume). Additionally or alternatively, the hearing aid may utilize a neural network to improve recognition of the background environment. Based on the recognized environment, the hearing aid may (automatically) perform a function (e.g., change noise cancellation settings). The use of an accelerator may reduce the power consumption required to compute the neural network, which may be frequently required for the described hearing aid scenarios. The reduced power is beneficial to hearing aids, which require a smaller size and a longer operating life.
[0021] exist Figure 1 A graphical description of an example neural network that can be implemented herein is shown in . The neural network includes a set of computational processes for receiving a set of inputs 100 (i.e., input values) and returning a set of outputs 120 (i.e., output values). The neural network can be applied to artificial intelligence recognition problems, where each output 120 represents a possible recognition (e.g., machine recognition of a speech command or image identification). In this case, the output 120 with the highest value represents the recognition that most likely corresponds to the input. For example, each output 120 may represent a speech command and each input may represent a speech. For example, audio samples transformed to the frequency domain and stored in a circular input buffer may be fed into the neural network for each frame (e.g., 10 milliseconds) of audio data. In other words, the circular buffer can keep the audio length (e.g., 0.4 to 2 seconds) that the neural network runs. When the input speech causes the output to exceed a threshold, the speech can be recognized as a speech command, which in turn can trigger additional functions in a larger system.
[0022] The neural network includes layers 110A, 110B, 110C, 110D composed of artificial neurons (e.g., represented as circles). For simplicity, artificial neurons will be simply referred to as neurons in the present invention. As a simulation of biological neurons, each neuron has a value (i.e., activation value) corresponding to the activity of the neuron. For example, the activation value can be a value between 0 and 1 or a value between -1 and +1. The value for each neuron (i.e., node) is determined by a set (i.e., arrow) of synapses 130 that couple each neuron to other neurons in the previous layer. The value of a given neuron involves the accumulated weighted sum of all neurons in the previous layer. In other words, the value of each neuron in the first layer is multiplied by the corresponding synaptic weight and these values are added together to help calculate the activation value of the neuron in the second layer. In addition, a bias can be added to the sum to help regulate the overall activity of the neuron. In addition, the sum containing the bias can be applied to the activation function, which maps the sum to a range (e.g., zero to 1). Possible activation functions may include (but are not limited to) rectified linear units (ReLu), sigmoids, or hyperbolic tangents (TanH).
[0023] The neural network is trained for recognition before implementation in an application. In addition, the training of the neural network can be updated based on tentative data collected during the implementation. The training results in a set of weights for synapses and biases for final neuron accumulation. A relatively large amount of memory may be required to store the weights of each synapse and the biases for each node. For example, assuming 8-bit weights, a neural network with more than 100,000 synapses may require 100 kilobytes (kB) of memory, which is large for devices with smaller memory capacity, such as hearing aids. Some of the weights can be eliminated to save memory. For example, when a first neuron value multiplied by a very low weight will have little effect on the cumulative sum, even if the first neuron value is very high (i.e., even if the first neuron is highly active). In some cases, these low-weighted synapses can be pruned from the neural network without significantly reducing the accuracy of the neural network. This pruning can save processing (e.g., multiplication, addition) and memory requirements. In some embodiments, 90% of the weights can be pruned without significant (e.g., any) loss of accuracy. The techniques and hardware disclosed herein support the use of reduced neural networks.
[0024] The weights can be compressed by a compression scheme to allow efficient storage. The accelerator supports nonlinear compressed weights through a built-in programmable lookup table that can expand the compressed weights (e.g., 4-bit to 8-bit expansion). Weights with zero values can be encoded as a single bit (e.g., 0), and non-zero weights can be encoded with additional header bits (e.g., 1). An example of a compression scheme can be represented as 0=simplified weight (expanded to 0x00), and xxxxxxxx1=non-simplified weight (in 8-bit mode, 4-bit mode can be xxxx1). The entire bit stream can be packaged together and start at a 32-bit boundary in the memory. The bit stream can be read from a 32-bit word in the memory, decoded, and in some cases 4-bit weights are expanded to 8 bits. Although an example of a compression scheme is described in detail above, other embodiments may use a modified version of an example compression scheme or a completely different compression scheme.
[0025] Even modest neural networks may require a high number of cycles to compute (e.g., in software running on a processor). Additionally, an application may require a neural network to implement multiple (e.g., 100) times of seconds. An increase in cycle speed to accommodate the large number of required computations corresponds to an increase in power. However, clock speeds for some mobile applications may be limited to a low rate (e.g., 50 MHz) to save power. Although digital signal processors (DSPs) may be used for computation, DSPs are generally configured for high precision (e.g., 24-bit, 32-bit). However, processing a neural network may require much less precision. In this case, the added precision of the DSP is a waste of power. The disclosed circuits and techniques may provide advantages over DSP implementations because it is possible to obtain more processing per cycle and use less precision to save power without losing accuracy.
[0026] Neural networks (e.g., deep neural networks) may require a very large number of computations (e.g., between 10 and 100,000) to reach a conclusion. Moreover, the neural network may need to reach many conclusions per second in response to a stream of input data. For example, the methods disclosed herein may be more advantageous than a DSP running a neural network algorithm, which may require a very fast clock (i.e., a high number of clock cycles) to perform neural network algorithm processing in real time. The disclosed methods may also be more advantageous than a DSP that may consume too much power for some applications. The disclosed methods may be more advantageous than other methods that limit the power consumption by limiting the size of the neural network, thereby limiting the performance of the neural network (e.g., power of recognition).
[0027] exist Figure 2 A block diagram of a SOC implementation of a neural network system is shown in FIG. Figure 1 The aspects shown in can be used Figure 2The neural network system is configured to receive input values from an input / output (I / O) component 240 (e.g., a microphone) and recognize the input values by processing a neural network that has been trained to recognize specific input values as having specific meanings. For example, the input values may be Mel-frequency cepstral coefficients (MFCCs) generated from an audio stream. In one possible implementation, frame audio samples are acquired regularly (e.g., every 10 milliseconds) and transformed to the frequency domain for input to the neural network.
[0028] The neural network system 200 includes a processor 210 (i.e., a host processor) coupled to a processor data bus (i.e., processor bus, data bus, bus) 250. The processor 210 can perform a portion (e.g., none, a portion) of the processing for the neural network via software running on the processor 210. The system 200 also includes a processor memory (i.e., memory) 230 coupled to the processor data bus 250 and storing trained weights for the synapses (i.e., weights) of the neural network. The system 200 also includes an accelerator 220 coupled to the processor data bus 250.
[0029] The accelerator may perform a portion (e.g., all, part) of the processing for the neural network. The accelerator may (in some embodiments) use the same processor data bus 250 and the same processor memory 230 as the host processor (e.g., ARM processor) 210. The accelerator may use the processor data but when it is not used by the host processor. For embodiments where the task (e.g., calculation) of the neural network is split between the accelerator and the processor, the accelerator may trigger the processor to perform the task by generating an interrupt. After receiving the interrupt, the processor 210 may read the input value from the (shared) processor memory 230, perform the task, write the result to the processor memory 230, and return control to (i.e., restart) the accelerator 220.
[0030] exist Figure 3A hardware block diagram of a general embodiment of the accelerator is shown in FIG. The memory can be accessed by an address generation unit 310 to obtain input data 312 (e.g., MFCC values generated from an audio stream) and the weights of the neural network resulting from training (i.e., trained weights 314). The weights can be 8 bits or 4 bits (to allow for an instance logarithmic distribution of weights available for use, such as in a-law and μ-law audio codecs) using a programmable or fixed 4 to 8-bit lookup table; however, other embodiments may use weights of different set sizes. The input data and the trained weights, which can be 8 bits in length, are fed (e.g., four at a time) into an array of four (8-bit) multiplier-accumulator (MAC) units 320. Logic and control 325 is used to manage the formation and calculation of the sum of the products of neuron and synaptic weights using MAC units. Next, an activation function 330 (e.g., ReLu, S-shaped, TanH, etc.) is applied to map the sum to a range. Although not shown, bias can be extracted and added before the activation function. The activation values for the nodes in the output layer 340 of the neural network are (8-bit) values and are buffered (e.g., four at a time) into memory.
[0031] The accelerator can process large neural networks very quickly because of its use of parallel MAC units 320, which can accommodate a very large number of MAC operations. For example, a neural network that may be calculated using the disclosed method may have 680 nodes in the input layer and 64 nodes in the output layer. In addition, the neural network may have three hidden layers, each with 256 nodes. A single processing of this neural network to obtain the output may require a total of 321,536 MAC operations.
[0032] exist Figure 3 In the embodiment shown in , four neurons are processed in each cycle by multiplying the input with four different weights to produce four synapses. In the next cycle, the input and weights change. The new input and weights are multiplied by the results of the previous cycle and accumulated with the results of the previous cycle. This is not a low power method because the input and weights can change between cycles, which consumes power. The disclosed accelerator consumes less power by keeping the input stable when the weights change between cycles.
[0033] In the following (i.e., Figures 4 to 6 ), presents some possible implementations for accelerators. The possible implementations all interact with the processor but differ in the amount of neural network processing performed by the accelerator versus the amount performed by software running on the processor. Figure 4 In general, a possible implementation is described in which the majority of the processing is performed by the accelerator, with only the initial configuration being provided by the processor. Figure 5A possible implementation is described in which much of the processing is performed in software running on a processor. Figure 6 A possible implementation is described in which the portion of the processing performed by software running on a processor is scalable.
[0034] Figure 4 A possible implementation of the accelerator is described. The possible accelerator implementation is a standalone implementation because it has an internal memory 401 and a finite state machine (FSM). The processor can configure the FSM with parameters and / or control logic for processing a neural network, such as the number of inputs (nbr_input), the number of neurons (nbr_neuron), the weight memory address (weigth_addr), the input data memory address (input_data_addr), the output data memory address (output_data_addr), and the bias memory address (bias_addr). After completing the configuration of the FSM, Figure 4 A standalone implementation of may not require further interaction with the processor (or processor bus or processor memory). However, in a possible implementation, the accelerator may transmit the results of the neural network to the processor.
[0035] Figure 4 The standalone accelerator embodiment of includes shift registers 410 to store temporary outputs of the MAC, one for each neuron. The shift register is shown as having four register positions, which is the same number as the number of synaptic multiplications. However, the number of register positions is not generally limited to the number of synaptic multiplications.
[0036] Figure 4 The operation of a possible accelerator implementation generally includes the processing of multiple neurons (e.g., four as shown) on multiple synapses (i.e., weights). In the first cycle, the four synapses associated with the first neuron are multiplied by four inputs (e.g., layer inputs) and the sum is stored in one of the registers of the shift register 410. In the next cycle, a different set of synapses associated with the second neuron are multiplied by the (same) four inputs and the cumulative sum is stored in the next register of the shift register 410. This process repeats until all registers are written. Once all registers are written, a new set of four inputs for the first neuron is obtained, multiplied by the weights, and accumulated with the previously stored register values. The process continues until each node in the computation layer. At this point, bias and activation functions are applied to the node values before being stored in the internal storage device / memory 401.
[0037] exist Figure 4In a possible implementation, four synaptic multiplications per cycle are performed, allowing data to be packed into 32-bit words when using 8-bit operations. Shift register bank 410 holds the input static for multiple (e.g., 4) cycles, as each traversal processes multiple (e.g., 4) neurons. Keeping the input data stable reduces bus traffic (i.e., power consumption). An implementation includes a finite state machine (FSM) and counters controlled by configuration registers provided (e.g., by a processor) for each layer of the neural network.
[0038] Figure 5 A possible implementation of an accelerator is described in which activation functions and biases are applied to each node in software running on a processor. Accordingly, this implementation of the accelerator includes accessing a shared memory over a shared (processor) bus.
[0039] Figure 5 The embodiment of includes a circular register set (i.e., circular buffer 510) to feed the neural network samples from the streaming data in a time windowed manner. Each time the four inputs of the neuron are processed as Figure 4 The possible implementation is carried out as described, with the difference that instead of the shift register 410 a rotation register 520 is used.
[0040] After calculating each neuron, an interrupt is generated and sent to the host processor. The processor reads the accumulated neuron value, adds a bias, and executes the activation function. For example, a rectified linear unit (ReLu) can be implemented in two cycles by minimum (i.e., min) and maximum (i.e., max) instructions. The resulting neuron value is stored in memory and used as input for the next layer of the neural network. For each layer, the processor starts the accelerator until the entire neural network is processed.
[0041] Figure 6 A possible implementation of an accelerator is described in which the activation function can be implemented in hardware or software. Figure 4 Compared to the implementation shown in Figure 6 The embodiment shown in does not have internal memory to store the output of the layer. Instead, the output is stored in the processor's memory. Figure 5Compared to the embodiment shown in , there is a lookup table (LUT) 601 implementing the activation function and an adder 602 for calculating the bias. This embodiment of the accelerator allows software to control the neural network calculation and hardware or software to apply the activation function. The application of the activation function can be configured by selecting the input to the multiplexer 610. When using hardware, the upper input 611 of the multiplexer 610 is selected and when using software, the bottom input 612 of the multiplexer 610 is selected. When the activation function is applied in hardware, writeback of the activation value is possible and the entire layer can be processed without interaction with the host processor. In operation, the bias can be extracted from the memory and added to the cumulative sum. Subsequently, the activation function can be executed in hardware and the resulting neuron value is stored in the memory. This process can be repeated for other neurons in the layer. After the configured amount of neurons has been processed and stored, an interrupt can be generated for the host processor. Upon receiving the interrupt and after updating the configuration register 613, the processor can restart the accelerator again for the next layer and the process repeats until the entire neural network has been processed.
[0042] The accelerator can use the processor data bus to access memory when it is not available to the host processor (e.g., to access input data, weights, and output data). In order to increase the speed of processing, the number of parallel MAC units can be increased. However, this increase means that more weights need to be extracted from memory per cycle, and extracting too many weights in the process can become a bottleneck. Weight compression reduces the amount of data extracted per cycle and can facilitate increasing the number of parallel MAC units that can be implemented, which increases the overall processing speed. The input data must also be extracted and the input data extraction can limit the weight data extraction. In order to reduce the input data extraction, the amount of neurons processed (i.e., calculated) at the same time (i.e., parallelism) in the accelerator can be increased. Fewer input data extractions allow more extraction of weights and can increase processing throughput.
[0043] The weights of the neural network are learned during training. The weights are compressed and stored offline in memory. The accelerator is configured to extract the weights from the memory and decompress the weights for processing. Weight decompression can be achieved using a hardware mechanism on the accelerator to extract the weight data and perform decompression. After each weight extraction, the address pointer corresponding to the weight storage location in the memory can be incremented. The extracted compressed weights can be kept in a buffer (e.g., 64 bits) having a buffer full level status register corresponding to the amount of bits available in the buffer. The compressed weights (e.g., 4-bit data) can be expanded (e.g., 8-bit data) by a programmable lookup table implemented in a flip-flop on the accelerator. Zero values are encoded as a single bit (e.g., "0"), while non-zero weights are encoded by additional header bits (e.g., "1"). Compression reduces memory and extraction requirements, especially when the neural network is heavily streamlined (e.g., 90% of the weights are set to zero).
[0044] The accelerator reduces power consumption by processing multiple neurons at the same time while keeping the input data at the multiplier stable over multiple cycles. Keeping the input data stable reduces the amount of switching at the input of the multiplier. Therefore, less power is consumed (i.e., less than if they were not kept stable). The accelerator also reduces power consumption by performing multiple multiplications in parallel. This parallelism reduces the amount of timing required for the accumulator. Therefore, less power is consumed (i.e., less than without the added parallelism).
[0045] Figure 77 is a flow chart of a method for implementing a neural network according to a possible embodiment of the present invention. Input data is received in the method 700. The input data may be data derived from a frame of audio data. For example, the frames of audio data may be acquired sequentially (e.g., every 10 milliseconds) and mathematically converted into coefficients (e.g., MFCCs) representing the content of the audio data. The input data is received 710 for processing by the neural network. For example, a circular buffer of the accelerator may buffer the length of audio (e.g., 0.4 seconds to 2 seconds) on which the neural network operates. Additionally, compressed weights of the neural network obtained from previous training may be extracted from memory 720 and used for processing. For example, the buffer may hold compressed weights (e.g., 64 bits) as part of the extraction operation, and the compressed weights may be decompressed (e.g., from 4 bits to 8 bits) prior to processing of the neural network. The neural network is processed using an accelerator and a processor. Accordingly, a first portion of the processing for computing the neural network may be performed 730 by the accelerator, and a second portion of the processing for computing the neural network may be performed 740 by the (host) processor. The first and second parts of the processing form output data (e.g., output layer values) for the set of input data. This process can be repeated (e.g., continuously) when new input data is received (e.g., for streaming input data). In addition, the output data can be analyzed (e.g., the output layer value is compared to a threshold) for identification.
[0046] The disclosed circuits and techniques can be used to perform a method for implementing a neural network. The method can start by receiving input data and extracting weights of the neural network from a memory. Subsequently, a first portion of the processing for the neural network can be performed in hardware by an accelerator, while a second portion of the processing for the neural network can be performed by software running on a processor. The first portion of the processing implemented in hardware includes multiple parallel multiply and accumulate (MAC) operations, and the accelerator and processor use a bus to communicate and share access to the memory.
[0047] In one possible implementation, the weights include compressed weights. In these implementations, the first portion of the processing may include decompressing the compressed weights using a lookup table. In some implementations, the weights include condensed weights and uncondensed weights. The condensed weights include a single bit representation of the synapse while the uncondensed weights include additional bits.
[0048] In another possible implementation, multiple parallel multiply and accumulate operations are performed by multiple MAC units operating in parallel and repeatedly to produce a sum of products corresponding to each neuron in the neural network.
[0049] In some embodiments, the first part of the processing includes adding a bias to each of the sums of products for each neuron, while in other embodiments the second part includes adding a bias to each sum of products for each neuron in the neural network.
[0050] In some embodiments, the first part of the processing comprises applying an activation function to each sum of products (with an added bias) for each neuron in the neural network, while in other embodiments the second part of the processing comprises applying an activation function to each sum of products (with an added bias) for each neuron in the neural network.
[0051] In the specification and / or drawings, typical embodiments have been disclosed. The present invention is not limited to such exemplary embodiments. The use of the term "and / or" includes any and all combinations of one or more of the associated listed items. The drawings are schematic representations and are therefore not necessarily drawn to scale. Unless otherwise indicated, specific terms have been used in a general and descriptive sense, not for limiting purposes.
[0052] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those generally understood by those skilled in the art. Methods and materials similar to or equivalent to those described herein can be used to practice or test the present invention. As used in the specification and the appended claims, unless the context clearly dictates otherwise, the singular forms "a / an", "the" include multiple indicators. The term "comprising" and its variants as used herein are used synonymously with the term "including" and its variants and are open, non-restrictive terms. The term "optional" or "optionally" used herein means that the subsequently described features, events or situations may or may not occur, and the description includes situations where the features, events or situations occur and situations where the features, events or situations do not occur. The range can be expressed herein as from "about" a specific value and / or to "about" another specific value. When such a range is expressed, the aspect includes from a specific value and / or to another specific value. Similarly, when a value is expressed as an approximation by using the preposition "about", it should be understood that the specific value forms another aspect. It will be further understood that the endpoints of each of the ranges are significant both in relation to the other endpoint, and independently of the other endpoint.
[0053] Although certain features of the described embodiments have been illustrated as described herein, those skilled in the art will now appreciate many modifications, substitutions, changes and equivalents. Therefore, it should be understood that the appended claims are intended to cover all such modifications and changes as fall within the scope of the embodiments. It should be understood that they have been presented only by way of example and not limitation, and various changes in form and detail may be made. Any part of the apparatus and / or method described herein may be combined in any combination, except for mutually exclusive combinations. The embodiments described herein may include various combinations and / or sub-combinations of the functions, components and / or features of the different embodiments described.
Claims
1. A neural network system, include: a memory configured to store compressed weights of a neural network; processor; a processor data bus coupled between the processor and the memory; as well as an accelerator coupled to the processor and sharing the processor data bus with the processor, wherein the accelerator is configured to: extracting and decompressing the compressed weights of the neural network from the memory; as well as performing a first portion of processing for the neural network while the processor performs a second portion of processing for the neural network, the first portion of processing comprising: performing a plurality of MAC operations using a plurality of multiply and accumulate MAC units operating in parallel to generate a neuron value; Using a multiplexer coupling the plurality of MAC units to select the neuron value before applying bias and activation functions; as well as The neuron value is written to the memory via the processor data bus.
2. The neural network system according to claim 1, in: The compressed weights stored in the memory are each 4 bits; as well as The decompressed weights used in the first portion of the processing of the neural network performed by the accelerator are each 8 bits.
3. The neural network system of claim 1 , wherein the accelerator comprises a lookup table to decompress the compressed weights.
4. The neural network system of claim 3, wherein the lookup table is implemented in software running on the processor and loaded into the accelerator.
5. The neural network system of claim 3, wherein the lookup table is implemented in hardware on the accelerator.
6. The neural network system of claim 1 , wherein the compressed weights stored in the memory are compact so that zero-valued weights are encoded as a single zero bit and non-zero-valued weights are encoded with a header bit of one.
7. The neural network system of claim 1 , wherein the plurality of MAC units are implemented in hardware on the accelerator and operate in parallel to execute the plurality of multiply and accumulate MAC operations at the same time.
8. The neural network system of claim 7, wherein the accelerator comprises a circular buffer coupled to the plurality of MAC units and receiving input data for the neural network.
9. The neural network system of claim 8, wherein the circular buffer comprises banks that each store a frame of input data until the frame is coupled to one of the plurality of MAC units, at which time the bank receives a new frame of input data.
10. An accelerator for implementing the neural network system according to claim 1, wherein the accelerator include: a plurality of multiply-and-accumulate MAC units operating in parallel, each MAC unit configured to repeatedly multiply input values and weights to accumulate an overall sum of products representing values corresponding to neurons in the neural network system; a lookup table for decompressing the compressed weights stored in the memory to generate the weights for each MAC unit at each repetition; as well as A circular buffer feeds the input value to each MAC unit at each repetition, wherein the inputs to the plurality of MAC units remain stable while extracting and decompressing the compressed weights between each repetition.
11. The accelerator for implementing the neural network system according to claim 10, wherein the accelerator for implementing the neural network system is included in a hearing aid.
