Methods, devices, and circuits for decoding weights of neural networks

Abstract

The present application relates to methods, devices and circuits for decoding weights of a neural network. According to one aspect, a neural network circuit for decoding weights of a neural network includes a weight memory configured to store encoded weights of the neural network, wherein the encoded weights comprise index weight words; and a decompression logic circuit configured to retrieve the encoded weights from the weight memory, decode the encoded weights using the index weight words to obtain one or more non-clipped weight words and a sequence of one or more clipped weight words, and provide the non-clipped weight words and the sequence of clipped weight words to a plurality of input-weight multipliers.

Description

Technical Field

[0001] This disclosure relates to neural network weight encoding. Background Technology

[0002] Neural networks can be used for signal processing. A neural network consists of multiple layers, each with multiple neurons. Between each layer, neurons are associated with weight factors. Since the number of weights is the product of the number of neurons in the two layers, the amount of weights can be relatively large (thus increasing the amount of memory needed to store these weights), and the amount of multiplication can be relatively large (thus increasing the amount of processing power needed to perform multiplication). Summary of the Invention

[0003] According to one aspect, a neural network circuit for decoding the weights of a neural network includes: a weight memory configured to store encoded weights of the neural network, wherein the encoded weights include index weight words; and decompression logic configured to retrieve the encoded weights from the weight memory, decode the encoded weights using the index weight words to obtain a sequence of one or more unpruned weight words and one or more pruned weight words, and provide the sequence of unpruned weight words and pruned weight words to a plurality of input-weight multipliers. In some examples, the pruned weight words include zero-value weight words, and the unpruned weight words include non-zero-value weight words.

[0004] According to one aspect, the neural network circuit may include one or more (or any combination thereof) of the following features: An index weight word comprising a plurality of bits, wherein each bit has a first bit value or a second bit value, the first bit value indicating a pruned weight word in the sequence, and the second bit value indicating an unpruned weight word in the sequence. Decompression logic circuitry is configured to decode the encoded weights within a single clock cycle. The decompression logic circuitry includes a weight word decoder, wherein the weight word decoder includes a plurality of shifters, a plurality of digital logic gates coupled to the plurality of shifters, and an index word decoder coupled to the plurality of shifters and the plurality of digital logic gates. The index word decoder is configured to control the plurality of shifters and the plurality of digital logic gates based on the index weight word. The plurality of shifters are connected in parallel to the index word decoder. Each of the plurality of shifters is configured to receive an unpruned weight word and a control signal from the index word decoder, the index word decoder controlling a shift operation applied to the unpruned weight word. Each of the digital logic gates is configured to receive a control signal from the index word decoder, the index word decoder controlling a Boolean operation performed by the corresponding digital logic gate. The plurality of digital logic gates includes a first digital logic gate and a second digital logic gate, the first digital logic gate being coupled to the output of a first shifter among a plurality of shifters, and the second digital logic gate being coupled to the output of a second shifter among a plurality of shifters. The decompression logic circuitry includes: a first weight decoder configured to decode a first portion of the encoded weights; a second weight decoder configured to decode a second portion of the encoded weights; and control logic circuitry configured to control the first and second weight decoders. The first weight decoder is configured to receive from the control logic circuitry a signal indicating the number of available weight words stored in a weight word buffer, and is configured to determine the number of bits having a first value in an indexed weight word, and decode the first portion of the encoded weights in response to the number of available weight words stored in the weight word buffer being equal to or greater than the number of bits having a first value.

[0005] According to one aspect, an apparatus includes: a neural network configured to receive a set of inputs and generate a set of outputs, wherein the set of inputs includes speech data and the set of outputs includes one or more potential speech commands corresponding to the speech data; a weight memory configured to store encoded weights of the neural network; and an accelerator configured to execute the neural network. The accelerator includes decompression logic configured to retrieve encoded weights from the weight memory and decode the encoded weights to obtain a sequence of one or more pruned weight words and one or more unpruned weight words, wherein the pruned weight words include zero-value weight words and the unpruned weight words include non-zero-value weight words. The accelerator includes a plurality of input-weight multipliers configured to receive the sequences of pruned weight words and unpruned weight words.

[0006] According to one aspect, the neural network circuit may include one or more of the above / below features (or any combination thereof). The decompression logic circuit includes: a weight word buffer configured to temporarily store encoded weights, wherein the encoded weights include a first index weight word and a second index weight word; a first weight decoder configured to receive encoded weights from the weight word buffer and use the first index weight word to generate a first set of decoded weight words; and a second weight decoder configured to receive a portion of the encoded weights from the first weight decoder and use the second index weight word to generate a second set of decoded weight words. The device includes: a control logic component configured to control the first and second weight decoders; a shifter configured to receive the first set of decoded weight words from the first weight decoder and the second set of decoded weight words from the second weight decoder; and a plurality of weight registers coupled to the shifter, wherein each of the plurality of weight registers is configured to receive a separate decoded weight word from the shifter. The encoded weights include index weight words and one or more unpruned weight words, and the decompression logic circuit includes a weight word decoder. The weight word decoder includes multiple shifters, multiple AND gates coupled to the multiple shifters, and an index word decoder coupled to the multiple shifters and the multiple AND gates, wherein the index word decoder is configured to control the multiple shifters and the multiple AND gates based on the bit values ​​of individual bits of the index weight word. Each of the multiple shifters is configured to receive a truncated weight word and a first control signal from the index word decoder, wherein each shifter is configured to perform a shift operation on the truncated weight word according to the first control signal. Each of the AND gates is configured as the output of the corresponding shifter and a second control signal from the index word decoder, wherein each AND gate is configured to perform an AND operation on the output of the corresponding shifter and the second control signal. The encoded weights include an index weight word followed by a truncated weight word, wherein the encoded weights do not include the weight value of the truncated weight word. The index weight word includes multiple bits, wherein each bit has a first bit value or a second bit value. The first bit value indicates a truncated weight word in the sequence. The second bit value indicates a truncated weight word in the sequence. The clipping weight word is either the most significant byte or the least significant byte of the corresponding encoding weight.

[0007] According to one aspect, a method for decoding weights of a neural network includes: retrieving encoded weights from a weight memory via a processor data bus, wherein the encoded weights include index weight words; decoding the encoded weights using the index weight words to obtain a sequence of one or more pruned weight words and one or more uncropped weight words, wherein the pruned weight words include zero-value weight words and the uncropped weight words include non-zero-value weight words; and providing the sequence of uncropped weight words and pruned weight words to a plurality of input-weight multipliers.

[0008] In some aspects, the index weight word comprises multiple bits, each having a first value or a second value. The first value indicates a pruned weight word in the sequence, and the second value indicates a non-pruned weight word in the sequence. The pruned weight word is decoded to zero in response to the bit with the first value within the index weight word. This decoding may include controlling multiple shifters and multiple digital logic gates based on the index weight word.

[0009] The foregoing illustrative invention, as well as other exemplary objectives and / or advantages and implementation methods of this disclosure, are further explained in the following detailed description and accompanying drawings. Attached Figure Description

[0010] Figure 1A A neural network system for encoding and decoding weights of a neural network based on one aspect is shown.

[0011] Figure 1B A neural network with all connections based on one aspect is shown.

[0012] Figure 1C A neural network is shown that is partially connected according to one aspect.

[0013] Figure 1D A neural network trainer is shown that is configured to prune synapses of a neural network according to one aspect and store the weights of the pruned synapses as zero-value weights.

[0014] Figure 2 An encoding scheme for encoding the weights of a neural network based on one aspect is shown.

[0015] Figure 3 An encoding scheme for encoding the weights of a neural network, based on another aspect, is shown.

[0016] Figure 4 An encoding scheme for encoding the weights of a neural network, based on another aspect, is shown.

[0017] Figure 5 The decompression logic circuit for decoding the encoded weights is shown according to one aspect.

[0018] Figure 6 A neural network system based on one aspect is shown.

[0019] Figure 7 An accelerator for a neural network system based on one aspect is shown.

[0020] Figure 8 The decompression logic circuit of an accelerator based on one aspect is shown.

[0021] Figure 9 A weighted decoder based on a decompression logic circuit of one aspect is shown.

[0022] Figure 10 It is a flowchart of a method for decoding the weights of a neural network system based on one aspect.

[0023] The components in the accompanying drawings are not necessarily drawn to scale relative to each other. Similar reference numerals may be used in several drawings to indicate corresponding parts. Detailed Implementation

[0024] Figures 1A to 1D A neural network system 100 for encoding and / or decoding the weights of a neural network 106 is shown, which increases execution speed while reducing the memory requirements of the neural network system 100. For example, refer to Figure 1B and Figure 1C As shown, the neural network 106 includes multiple layers 129 of neurons 131 connected to synapses 138 (also called weights or weights). Because the number of weights is the product of the number of neurons 131 in adjacent layers 129, in some examples the amount of weights may be relatively large, and therefore may require a relatively large memory device to store the weights. For example, a neural network 106 with more than 100,000 weights may require 100 kilobytes (kB) of memory (assuming 8-bit weights), which is relatively large for devices with relatively small memory capacities.

[0025] Furthermore, the amount of multiplications to be performed may be relatively large, which could result in relatively slow execution speed and / or require increased processing power. For example, a neural network 106 may require a large number of loops to obtain an output (e.g., more than 50,000 loops, more than 100,000 loops), and the neural network 106 may need to obtain multiple outputs within a given time range (e.g., more than 50 times / second, more than 100 times / second, etc.). Increasing the loop speed to accommodate the large number of required operations corresponds to increasing power. However, the clock speed of some applications may be limited to a low rate (e.g., less than 100MHz, less than 50MHz, less than 20MHz) to save power.

[0026] Some weights can be pruned (or removed) to save memory. For example, when the value of the first neuron is multiplied by a low weight value (e.g., a very low weight value), even if the value of the first neuron is very high (e.g., even if the first neuron is highly active), it may have little effect on the summation. In some examples, these low-value weights can be pruned (e.g., removed) from neural network 106 without significantly reducing the accuracy of neural network 106 (if any). This pruning saves processing (e.g., multiplication, addition) and memory requirements. In some examples, a specific percentage (e.g., more than 50%, more than 70%, more than 80%, or more than 90%, etc.) of weights can be pruned without significant (e.g., any) loss of accuracy. However, pruning (or removing) low-value weights may result in irregularities in neural network 106 (e.g., incomplete connections).

[0027] However, according to the techniques discussed herein, the neural network system 100 provides an efficiency mechanism for storing and decoding the encoded weights 114, which reduces the size of the memory required to store the encoded weights 114, reduces the complexity of the decoding logic, and / or increases the speed at which the encoded weights 114 are decoded, thereby enabling the neural network 106 to execute quickly so that the device consumes relatively low power.

[0028] For example, weights 114a (e.g., weight word sequence a) are encoded according to an encoding scheme such that the encoded weights 114 do not include the actual value of the pruned weight word 126 (e.g., the pruned weight word 126 is removed from the encoded weights 114), thereby reducing the amount of memory required to store the encoded weights 114. In other words, the pruned weight word 126 is pruned (or removed) from the encoded weights 114, but the encoded weights 114 include encoded data 116 representing the position of the pruned weight word 126 within the weight sequence. The neural network system 100 includes decoding logic that decodes the encoded weights 114 to obtain a sequence of pruned weight words 126 and unpruned weight words 120. For example, the decoding logic is configured to decode the pruned weight words 126 to zero and include them in the correct sequence of unpruned weight words 120. The sequence of pruned weight words 126 and unpruned weight words 120 is provided to an input-weight multiplier 108 for multiplication with the input value.

[0029] Compared to some conventional methods that pass weights and their neuron offsets to the tiles on which they are computed, the decoding mechanism discussed in this paper computes a cropped weight word 126 that is decoded to zero (and therefore does not require the additional switching power in the input-weight multiplier 108), thus eliminating the need for neuron offsets. In some examples, a relatively fast decoding mechanism is required, where the encoded weights 114 are decoded in a single clock cycle between 15 MHz and 80 MHz, and conventional methods of passing weights and neuron offsets would not be fast enough to decode the encoded weights 114. For example, the computation of weights and neuron offsets may be too slow for some applications. Additionally, in some conventional methods, run-length encoding (RLE) is used in convolutional neural networks for stored or transmitted weights, and the weights are in floating-point format in terms of RLE. For example, when one or more weights are zero, these weights are encoded as floating-point numbers (e.g., negative zero, subnormal, non-numerical, infinity) that are not typically used in neural networks, where unused bits are used to encode the amount of zero weight. However, compared to some conventional methods, in some examples the encoding / decoding mechanism discussed in this paper does not use floating-point numbers, but instead uses integer operations, thus resulting in smaller logic and lower power consumption. Accordingly, floating-point numbers (e.g., small floating-point numbers) can be used in neural networks because they offer a larger dynamic range compared to the same number of bits in integer programming. In some examples, the weight pruning and decompression mechanism discussed in this paper can be used in conjunction with floating-point weights.

[0030] In some examples, the neural network system 100 uses index weight words 118 for each group of weight words to implement hardware decoding of the pruned weight word 126. In some examples, the number of weight words (e.g., a combination of pruned word 126 and unpruned weight word 120) is equal to the number of bits in the index weight word 118. In some examples, the neural network system 100 uses a hardware implementation that employs a shifter 140 to produce faster logic timing, thereby allowing multiple index weight words 118 to be decoded in a single clock cycle.

[0031] refer to Figure 1AThe neural network system 100 includes a computing device 103 and a device 102. The computing device 103 may be one or more computers that train the neural network 106 to generate weights 114a and encode the weights 114a to generate encoded weights 114. The encoded weights 114 can then be stored in a weight memory 112 of the device 102. During the execution of the neural network 106, the device 102 may retrieve the encoded weights 114 from the weight memory 112, decode the encoded weights 114, and use the decoded weights and input data captured by the device 102 to execute the neural network 106 to compute its output. The output of the neural network 106 may depend on the type of application utilized by the device 102, and the output of the neural network 106 may trigger other functions of the device 102.

[0032] In some examples, computing device 103 includes one or more computers separate from device 102. In some examples, computing device 103 includes one or more server computers. In some examples, device 102 is communicatively coupled to computing device 103 (e.g., via a wired or wireless connection) such that device 102 can receive and then store encoded weights 114. In some examples, device 102 is communicatively coupled to computing device 103 during the production of device 102 (in order to store encoded weights 114 on device 102), but is not communicatively coupled to computing device 103 when device 102 is placed in its usage area. In some examples, device 102 is communicatively coupled to computing device 103 (while device 102 is in its usage area) to receive retrained (re-encoded) weights 114 from computing device 103 via a network. In some examples, device 102 is communicatively coupled to computing device 103 via a network-based connection such as an Internet connection (e.g., a Wi-Fi connection, a mobile network interface connection) or a private wide area network (WAN) connection. In some examples, computing device 103 generates encoded weights 114 during manufacturing (e.g., production of device 102), then stores the encoded weights 114 in weight memory 112 of device 102, and then places device 102 in its field of use. In some examples, device 102 is connected to computing device 103 via a wireless connection, and device 102 may receive (e.g., periodically receive) updated weights 114 (e.g., retraining / recoding weights) for a portion of its device lifespan.

[0033] In some examples, device 102 includes digital circuitry that is smaller in processing power and memory size than computing device 103. In some examples, computing device 103 may be one or more high-power computers configured to train neural network 106 to generate weights 114a, then encode the weights 114a to generate encoded weights 114, and device 102 may be considered a low-power (relatively small) device that performs neural network 106 with encoded weights 114 determined by computing device 103.

[0034] Device 102 can be any type of device that can use encoded weights 114 to execute neural network 106. Device 102 includes neural network circuitry 104, wherein neural network circuitry 104 includes weight memory 112 storing encoded weights 114, neural network 106, and accelerator 156 configured to assist in executing neural network 106. In some examples, weight memory 112 includes random access memory (RAM). In some examples, neural network circuitry 104 is a system-on-chip (SOC) device (e.g., an integrated circuit coupled to a semiconductor substrate). In some examples, by using the encoding scheme discussed herein, weight memory 112 can be relatively small because the pruned weight word 126 is not included in encoded weights 114. In some examples, neural network 106 is considered part of accelerator 156.

[0035] Accelerator 156 may include decompression logic circuitry 110 and input-weight multiplier 108. In some examples, decompression logic circuitry 110 includes one or more hardware decoding components capable of relatively fast decoding of encoded weights 114. In some examples, decompression logic circuitry 110 includes a hardware implementation using shifter 140, which produces faster logic timing, thereby allowing simultaneous decoding of multiple weight words. Decompression logic circuitry 110 may retrieve encoded weights 114 (via a data bus) from weight memory 112, decode encoded weights 114, and provide the decoded weights to input-weight multiplier 108 to perform appropriate multiplication. Input-weight multiplier 108 can be considered as part of accelerator 156 that multiplies the inputs of synapses 138 with corresponding weights, which connect two layers 129 of neural network 106. In some examples, input-weight multiplier 108 is considered as part of one or more multiplication and accumulation units (e.g., MAC units).

[0036] In some examples, device 102 is a speech recognition device. In some examples, device 102 is a hearing aid device. Neural network circuitry 104 is configured to receive audio input and determine an audio-speech command based on that audio input. In some examples, device 102 utilizes neural network 106 to improve the recognition of commands spoken by a user. Based on the recognized command (e.g., increase volume), device 102 can perform a function (e.g., increase volume). Additionally or alternatively, device 102 can utilize neural network 106 to improve the recognition of a background environment. Based on the recognized environment, device 102 can (automatically) perform a function (e.g., change noise cancellation settings). Using accelerator 156 can reduce the power consumption required to compute neural network 106, which may be frequently needed in the speech recognition scenario. This power reduction may be advantageous for relatively small devices with relatively low power consumption (e.g., hearing aids).

[0037] In some examples, device 102 using neural network 106 and accelerator 156 can improve speech recognition (e.g., voice commands) or sound recognition (e.g., background noise types) in a highly efficient manner (e.g., to conserve battery life). In some examples, accelerator 156 is a semiconductor (i.e., hardware) platform (i.e., block) that helps the processor implement neural network 106. Accelerator 156 includes hard-coded logic and mathematical functions that can be controlled (e.g., via a state machine configured by the processor) to process neural network 106. In some examples, accelerator 156 can process neural network 106 faster and more efficiently (highly efficient) than conventional software running on, for example, a digital signal processor (DSP). DSP methods may require additional processing / power resources to fetch software instructions, perform computations in series, and use a much higher bit depth than a particular application might expect. Conversely, in some examples, accelerator 156 avoids fetching software instructions, performs processing (e.g., computations) in parallel, and uses a bit depth of neural network 106 suitable for a particular application.

[0038] Neural network 106 (e.g., a deep neural network) may require a very large number of operations (e.g., between 10,000 and 100,000 or greater) to obtain a result. Additionally, neural network 106 may need to perform many computations per second to respond to the input data stream. In some examples, neural network 106 may need to perform more than 200,000 operations to obtain an output, and must obtain an output every 10 milliseconds, which may require a relatively large amount of power. Furthermore, these requirements may result in clock speeds of approximately 20 MHz, which may be too high for typical low-power DSPs for some applications. Some of the techniques discussed in this paper can implement neural network 106 in applications with the same (or similar) requirements, making these applications smaller and less power-intensive. For example, accelerator 156 can reduce power consumption by processing multiple neurons 131 simultaneously while keeping the input data at the input-weight multiplier 108 stable for multiple loops. Keeping the input data stable reduces the amount of switching at the input of the multiplier. Therefore, less power can be consumed (i.e., less than if these input data were not kept stable). Accelerator 156 can also reduce power consumption by performing multiple multiplications in parallel (e.g., the execution of input-weight multiplier 108 can be performed in at least partially parallel). This parallelism reduces the amount of clock cycles required by the accumulator. Therefore, less power is consumed (i.e., less than without the added parallelism).

[0039] Computing device 103 includes a neural network trainer 122 configured to train neural network 106 using training data 105, wherein the training results determine weights 114a. Weights 114a can represent the strength of connections between units. If the weights 114a from neuron A to neuron B have a large value, it means that neuron A has a large influence on neuron B. Weights 114a comprise a sequence of weight words, where each weight word has a specific size (e.g., four-bit, eight-bit, sixteen-bit, etc.). Training of neural network 106 can be updated based on heuristic data collected during implementation. This training produces a set of weights 114a for synapses and a bias for the final neuron accumulation. During this training, weights 114a with relatively low values ​​are set to zero. Therefore, neural network trainer 122 can generate weights 114a that include pruned weight words 126 and unpruned weight words 120. A specific pruning weight word 126 or a specific uncropped weight word 120 is a weight value associated with a single synapse 138. Pruning weight word 126 may be referred to as a weight word that is set to zero and removed from the encoded weights 114. In some examples, pruning weight word 126 is a zero-value weight word (e.g., "0000" if the weight word is four bits). Unpruned weight word 120 may be referred to as an unpruned (e.g., not removed from the encoded weights) weight word. In some examples, unpruned weight word 120 includes non-zero-value weight words (e.g., "AAAA" if the weight word is four bits). In some examples, depending on the encoding scheme used, unpruned weight word 120 may include one or more zero-value weight words.

[0040] In some examples, most of the weights 114a include a pruned weight word 126. The computing device 103 includes a weight encoder 128 that encodes the weights 114a to generate encoded weights 114. The encoded weights 114 include encoded data 116 and unpruned weight words 120. The encoded weights 114 may not include the value of the pruned weight word 126 from the encoded weights 114 (and thus be pruned or removed).

[0041] Encoded data 116 may include numeric data indicating the presence of one or more pruned weight words 126 within the sequence of unpruned weight words 120. In some examples, encoded data 116 includes an index weight word 118. In some examples, the index weight word 118 may be considered a header followed by one or more unpruned weight words 120 (or empty). In some examples, the index weight word 118 is a single bit with a first bit value (e.g., "0") or a second bit value (e.g., "1"). In some examples, the index weight word 118 includes multiple bits. In some examples, the index weight word 118 is a four-bit index word. In some examples, the index weight word 118 is an eight-bit index word. In some examples, the index weight word 118 is a sixteen-bit index word. In some examples, the index weight word 118 is a thirty-two-bit index word.

[0042] In some examples, the 16-bit weight is encoded by the weight encoder 128 using an 8-bit weight word. However, the number of bits in the weight word can cover any value, and the number of bits in the weight can cover any value. In some examples, the most significant byte (MSB) (or most significant portion) is truncated and the least significant byte (LSB) (or least significant portion) is not truncated, and then the least significant byte (LSB) (or least significant portion) is sign-extended to create a signed 16-bit value (or any bit value used for encoding). For example, when the truncated weight word is the most significant portion, the sign of the least significant portion is used to fill the most significant portion. In some examples, the index weight word 118 can be an 8-bit index word. For each 16-bit weight, the weight encoder 128 is configured to recognize the most significant byte (MSB) of the 16-bit weight as the truncated weight word 126 and the least significant byte (LSB) as the untruncated weight word 120, and to sign-extend the least significant byte (LSB). In some examples, weight encoding 128 is configured to recognize the least significant byte (LSB) of the sixteen-bit weight as the pruned weight word 126 and the most significant byte (MSB) as the unpruned weight word 120, and in this case, the least significant byte (LSB) is zero.

[0043] Figure 1BA fully connected neural network 106 according to one aspect is illustrated. The neural network 106 includes a set of computational processes for receiving a set of inputs 135 (e.g., input values) and generating a set of outputs 136 (e.g., output values). In some examples, each output value in the set of outputs 136 may represent a speech command, and the input values ​​may represent speech (e.g., audio data in the frequency domain). The neural network 106 includes multiple layers 129, each layer 129 including multiple neurons 131. The multiple layers 129 may include an input layer 130, one or more hidden layers 132, and an output layer 134. In some examples, each output of the output layer 134 represents a possible recognition (e.g., machine recognition of a speech command or image identifier). In some examples, the output of the output layer 134 with the highest value represents the recognition most likely corresponding to the input.

[0044] In some examples, neural network 106 is a deep neural network (DNN). For example, a deep neural network (DNN) may have one or more hidden layers 132 positioned between an input layer 130 and an output layer 134. However, neural network 106 can be any type of artificial neural network (ANN), including convolutional neural networks (CNNs). Neurons 131 in one layer 129 are connected to neurons 131 in another layer via synapses 138. For example, Figure 1B Each arrow in the diagram can represent a single synapse 138. Fully connected layers 129 (such as...) Figure 1B (As shown) Each neuron 131 in one layer 129 is connected to each neuron in the adjacent layer 129 via synapses 138.

[0045] Each synapse 138 is associated with a weight 114a. Weights 114a are parameters that transform input data within the hidden layer 132 of the neural network 106. When input enters a neuron 131, the input is multiplied by a weight value (e.g., a weight word), and the resulting output is observed or passed to the next layer in the neural network 106. For example, each neuron 131 has a value corresponding to neuronal activity (e.g., an activation value). This activation value can be, for example, a value between 0 and 1 or between -1 and +1. The value of each neuron 131 is determined by the set of synapses 138 that couple each neuron 131 to other neurons 131 in the previous layer 129. The value of a given neuron 131 is associated with the cumulative weighted sum of all neurons 131 in the previous layer 129. In other words, the value of each neuron 131 in the first layer 129 is multiplied by its corresponding weight, and these values ​​are summed to calculate the activation value of the neuron 131 in the second layer 129. Additionally, biases can be added to the sum to adjust the overall activity of the neurons 131. Furthermore, a sum including the bias can be applied to an activation function that maps the sum to a range (e.g., 0 to 1). Possible activation functions may include (but are not limited to) the Modified Linear Unit (ReLU), sigmoid, or hyperbolic tangent (TanH).

[0046] Figure 1C A partially connected neural network 106 is shown. For example, similar to... Figure 1B The neural network 106 includes a set of computational processes for receiving a set of inputs 135 (e.g., input values) and generating a set of outputs 136 (e.g., output values). Additionally, the neural network 106 includes multiple layers 129, each layer 129 including multiple neurons 131, and each layer 129 including an input layer 130, one or more hidden layers 132, and an output layer 134. Neurons 131 in one layer 129 are connected via synapses 138 to neurons 131 in adjacent layers 129. However, with... Figure 1B Unlike other neural networks, neural network 106 is not fully connected, where each neuron 131 in one layer 129 is not connected to each neuron in an adjacent layer 129 via synapses 138. For example, as described above, weights 114a include pruned weight words 126 and unpruned weight words 120. If a synapse 138 is associated with a pruned weight word 126, then that synapse 138 (and therefore the corresponding weight) can be considered to be pruned or removed from neural network 106, resulting in a partially connected (or irregular) neural network 106.

[0047] Figure 1D An example of a neural network trainer 122 is shown, which can set very small (and therefore almost insignificant) weights 114a to zero during the learning process (e.g., during the determination of weights 114a). For example, as Figure 1D As shown, a portion of the neural network 106 includes a first layer 129-1 and a second layer 129-2. The first layer 129-1 has four neurons 131 (e.g., first neuron 131-1, second neuron 131-2, third neuron 131-3, and fourth neuron 131-4), and the second layer 129-2 also has four neurons 131 (e.g., first neuron 131-1, second neuron 131-2, third neuron 131-3, and fourth neuron 131-4). Before the pruning process, each neuron 131 in the second layer 129-2 is connected to each neuron 131 in the first layer 129-1 via a synapse 138 (e.g., the second layer 129-2 is a fully connected layer). However, if the weight 114a associated with the corresponding synapse 138 is below a threshold level, the weight 114a is set to zero, resulting in a less complex neural network 106. Figure 1D As shown, after the pruning process, there are five synapses 138 representing the unpruned weight word 120: a (with weight word value "aaaa"), b (with weight word value "bbbb"), c (with weight word value "cccc"), d (with weight word value "dddd"), and e (with weight word value "eeee"). The remainder of weight 114a is set to zero ("0000"), thus representing pruned weight word 126. For example, neural network trainer 122 is configured to generate weight 114a that includes pruned weight word 126 and unpruned weight word 120. Although Figure 1D The weight word is shown as four bits, but each weight word may include any number of bits, including an eight-bit weight word or a sixteen-bit weight word, etc. If the weight 114a is stored in the memory on device 102, the size of that memory (e.g., weight memory 112) may be quite large because the memory must store many clipped weight words 126.

[0048] Figures 2 to 4 Different encoding mechanisms for bit encoding of the 126-bit weighted word are shown. For example, Figures 2 to 4 Different encoding schemes are shown, which the weight encoder 128 can use to generate encoded weights 114 such that the value of the clipped weight word 126 is not included in the encoded weights 114, thereby reducing the amount of memory (e.g., weight memory 112) required to store the encoded weights 114.

[0049] Figure 2 It shows the use of one aspect for... Figure 1D The weight 114a is encoded to generate an encoding scheme 213 with encoded weight 214. Encoded weight 214 can be... Figures 1A to 1D Examples of encoding weight 114 may be provided, and may include any details discussed herein with respect to these figures. As stated above, Figure 1D Weight 114a can represent sixteen weight words, including pruned weight word 126 and undpruned weight word 120. In this example, each weight word is four bits. Encoding scheme 213 uses single-bit encoding for pruned weight word 126.

[0050] The sequence of encoded weights 214 does not include the value of the pruned weight word 126 (“0000”). Instead, encoded weights 214 includes encoded data 216 representing the pruned weight word 126 within the sequence of weight words. Encoded data 216 can be a single bit representing the pruned weight word 126. In some examples, such as... Figure 2 As shown, a low logic value ("0") indicates the presence of clipping weight word 126. In some examples, a high logic value ("1") indicates the presence of clipping weight word 126.

[0051] use Figure 1D Example of weight 114a, according to Figure 2 In encoding scheme 213, the first weight word (“0000”) is a pruned weight word 126, so the first bit in the encoded weight 214 is a single bit with a value of zero (e.g., encoded data 216). The next weight word (“aaaa”) is an unpruned weight word 120, so the next bit in the encoded weight 214 is the value of that weight word (“aaaa”). In some examples, encoded data 215 (e.g., a single bit value “1”) is placed before the unpruned weight word 120 to indicate that the next bit is the unpruned weight word 120. The third and fourth weight words are pruned weight words 126, so the next bit in the encoded weight 214 includes a representation of two single bits with a value of zero (e.g., encoded data 216). The fifth weight word is an unpruned weight word 120 (“bbbb”), so the next bit in the encoded weight 214 is the value of that weight word (“bbbb”), which is preceded by encoded data 215 (e.g., a single bit with a value of “1”).

[0052] The sixth, seventh, and eighth weight words are pruned weight words 126, so the next bit in the encoded weight 214 is three single bits with a value of zero (e.g., encoded data 216). The ninth weight word is an unpruned weight word 120 (“cccc”), so the next bit in the encoded weight 214 is the value of that weight word (“cccc”), which is preceded by encoded data 215 (e.g., a single bit with a value of “1”). The tenth weight word is pruned weight word 126, so the next bit in the encoded weight 214 includes a representation of a single bit with a value of zero (e.g., encoded data 216). The eleventh weight word is an unpruned weight word 120 (“dddd”), so the next bit in the encoded weight 214 is the value of that weight word (“dddd”), which is preceded by encoded data 215 (e.g., a single bit with a value of “1”). The twelfth, thirteenth, fourteenth, and fifteenth weight words are clipped weight words 126, so the next bit in the encoded weight 214 includes a representation of four single bits with a value of zero (e.g., encoded data 216). The sixteenth weight word is an un-clipped weight word 120 (“eeee”), so the next bit in the encoded weight 214 is the value of this weight word (“eeee”), which is preceded by encoded data 215 (e.g., a single bit with a value of “1”).

[0053] for Figure 2 Encoding scheme 213 allows decoding logic to be performed at the bit level. Figure 2 In the diagram, arrow 219 indicates the number of index shifts required to decode the encoded weights 214. In this example, decompression logic 110 may be needed to perform a sixteenth index shift to decode the encoded weights 214. The details of the decoding performed by decompression logic 110 will be explained further later in this disclosure.

[0054] Figure 3 It shows the use of one aspect for... Figure 1D The weight 114a is encoded to generate an encoding scheme 313 with encoding weight 314. Encoding weight 314 can be... Figures 1A to 1D Examples of encoding weights 114, and may include any details discussed therein with respect to these figures. As stated above, Figure 1D The weight 114a can represent sixteen weight words, including a pruned weight word 126 and a uncropped weight word 120. The encoded weight 314 is generated using an index weight word 318 followed by one or more uncropped weight words 120 (or empty). For example, encoding scheme 313 uses an index weight word 318 followed by weight words with four-bit word boundaries (e.g., using an octet compression scheme to compress the four-bit weight words).

[0055] The number of bits in the index weight word 318 can be equal to the number of weight words in the group. For example, a specific four-bit index weight word 318 can represent four weight words. If two bits in the index weight word 318 have a logic value of "0" and two bits in the other bits of the index weight word 318 have a logic value of "1", then the sequence will include two clipped weight words 126 and two unclipped weight words 120. Because Figure 1D The weight 114a includes sixteen word weights (and the index weight word 318 is four bits), so there will be four index weight words 318 to represent the sixteen word weights.

[0056] refer to Figure 1D The weight 114a, the first group of four weight words can be represented by the first index weight word 318 in the encoded weight 314, wherein the first group includes “0000”, “aaaa”, “0000” and “0000”. Figure 3 The first index weight word 318 includes "0100". Since the number of "1"s in index weight word 318 is one, there should be a non-pruned weight word 120 after the first index weight word 318. The first bit in index weight word 318 indicates that the first weight word is a pruned weight word 126 (e.g., with a value "0"). The second bit in index weight word 318 indicates that the second weight word is a non-pruned weight word 120 (e.g., with a value "1"), where the value of the non-pruned weight word 120 ("aaaa") is located after index weight word 318. The third bit in index weight word 318 indicates that the third weight is a pruned weight word 126 (e.g., with a value "0"). The fourth bit in index weight word 318 indicates that the fourth weight is a pruned weight word 126 (e.g., with a value "0").

[0057] Then, the second group of four weight words can be represented by the second index weight word 318 in the sequence of encoded weights 314, where the next four weight words are "bbbb", "0000", "0000", and "0000". Since the number of "1s" in the second index weight word 318 is one, there should be a non-pruned weight word 120 ("bbbb") after the second index weight word 318. The first bit in the index weight word 318 indicates that the first weight word in the group is the non-pruned weight word 120, where the value of the non-pruned weight word 120 ("bbbb") is located after the second index weight word 318. The second bit in the second index weight word 318 indicates that the second weight word in the group is the pruned weight word 126. The third bit in the second index weight word 318 indicates that the third weight word in the group is the pruned weight word 126. The fourth bit in the index weight word 318 indicates that the fourth weight word in the group is the pruned weight word 126.

[0058] Then, the third group of four weight words can be represented by the third index weight word 318 in the sequence of encoded weights 314, where the next four weight words are "cccc", "0000", "dddd", and "0000". Since the number of "1s" in the third index weight word 318 is two, there should be two untrimmed weight words 120 ("cccc", "dddd") after the third index weight word 318. The first bit in the third index weight word 318 indicates that the first weight word in the group is the untrimmed weight word 120, where the value of the untrimmed weight word 120 ("cccc") is located after the second index weight word 318. The second bit in the third index weight word 318 indicates that the second weight word in the group is the trimmed weight word 126. The third bit in the third index weight word 318 indicates that the third weight word in the group is the untrimmed weight word 120, where the value of the untrimmed weight word 120 ("dddd") is located after the weight word "cccc". The fourth bit in the third index weight word 318 indicates that the fourth weight word in this group is the clipping weight word 126.

[0059] Then, the fourth group of four weight words can be represented by the fourth index weight word 318 in the sequence of encoded weights 314, where the next four weight words are "0000", "0000", "0000", and "eeee". Since the number of "1s" in the fourth index weight word 318 is one, there should be a non-pruned weight word 120 ("eeee") after the fourth index weight word 318. The first bit in the fourth index weight word 318 indicates that the first weight word in the group is the pruned weight word 126. The second bit in the fourth index weight word 318 indicates that the second weight word in the group is the pruned weight word 126. The third bit in the fourth index weight word 318 indicates that the third weight word in the group is the pruned weight word 126. The fourth bit in the fourth index weight word 318 indicates that the fourth weight word in the group is the non-pruned weight word 120, where the value of the non-pruned weight word 120 ("eeee") is located after the fourth index weight word 318.

[0060] for Figure 3 Encoding scheme 313 allows decoding logic to be performed at the four-word level. Figure 3 In the diagram, arrow 319 indicates the number of index shifts required to decode the encoded weights 314. In this example, decompression logic 110 may be needed to implement four index shifts to decode the encoded weights 314. The details of the decoding performed by decompression logic 110 will be explained further later in this disclosure. In some examples, compared with... Figure 2 Compared to decoding the encoded weight 214, it is used for decoding... Figure 3 The decoding logic for encoding weight 314 is not too complex and is faster.

[0061] Figure 4 It shows the use of one aspect for... Figure 1D The weight 114a is encoded to generate an encoding scheme 413 with encoding weight 414. Encoding weight 414 can be... Figures 1A to 1D Examples of encoding weights 114, and may include any details discussed therein with respect to these figures. As stated above, Figure 1D The weight 114a can represent sixteen weight words, including a pruned weight word 126 and a uncropped weight word 120. The encoded weight 414 is generated using an index weight word 418 followed by one or more uncropped weight words 120 (or empty). For example, encoding scheme 413 uses an index weight word 418 followed by weight words with octet boundaries.

[0062] The number of "1s" in index weight word 418 indicates the number of non-clipping weight words 120. (Reference) Figure 1D and Figure 4 Since the number of unpruned weight words 120 is five, there are five "1s" in the index weight word 418. Because the index weight word 418 is eight bits, and the four-bit word weights are stored in eight-bit boundaries, a single index weight word 418 can represent... Figure 1D The sixteen weighted words.

[0063] Specifically, since the word weight is four bits, but encoding scheme 413 stores the word weight in eight bits, the first bit in the index weight word 418 represents the first two weight words (e.g., 0000, aaaa). Because this pair of weight words includes the untrimmed weight word 120, the first bit in the index weight word 418 has a logical value "1". The "1" indicates that the weight word follows the index weight word 418. Since this encoding scheme uses an eight-bit boundary for the four-bit weight, the first four bits are "0000", followed by the untrimmed weight word 120 "aaaa".

[0064] refer to Figure 1DThe next two words have weights of clipped weight word 126 ("0000", "0000"), so the second bit in index weight word 418 has a logical value of "0". The value "0" indicates that there are no weight words following (e.g., because these weight words are clipped weight word 126). Then, the next two words in the list have weights of "bbbb" and "0000", and because this pair includes the non-clipped weight word 120, the third bit in index weight word 418 has a logical value of "1", where "bbbb" follows the last weight word "aaaa". Then, because the next two words in the list have weights of "0000" and "0000" (e.g., clipped weight word 126), the fourth bit in index weight word 418 has a logical value of "0". The next two word weights in the list are "0000" and "cccc", and because this pair includes the untrimmed weight word 120, the fourth bit in the index weight word 418 has a logical value of "1", where "0000" and "cccc" follow the last word weight "bbbb". The next two word weights in the list are "0000" and "dddd", and because this pair includes the untrimmed weight word 120, the fifth bit in the index weight word 418 has a logical value of "1", where "0000" and "dddd" follow the last word weight "cccc". The next four word weights in the list are "0000", "0000", "0000", and "0000", so the sixth and seventh bits in the index weight word 418 could have a logical value of "0", but in this example, because the sixth bit in the index weight word 418 has a logical value of "1", the word weights "0000" and "0000" follow the last word weight "dddd". Since the next weight word is "eeee", the eighth bit in the index weight word 418 is used as the logical value "1", where "eeee" follows the last word weight "0000".

[0065] for Figure 4 Encoding scheme 413 allows decoding logic to be performed at the octet level. Figure 4 In the diagram, arrow 419 indicates the number of index shifts required to decode the encoded weight 314. In this example, the decoding logic counts the number of "1"s in the index weight word 418 to determine the number of unpruned weight words 120, and implements a single index shift to... Figure 1D Decode the sixteen characters. In some examples, with... Figure 3 Compared to decoding with an encoding weight of 314, this is used for decoding... Figure 4 The decoding logic for encoding weight 414 can be less complex and faster (e.g., much faster).

[0066] Figure 5An example of a decompression logic circuit 510 according to an example is shown. The decompression logic circuit 510 can be... Figures 1A to 1D Examples of decompression logic circuitry 510 are provided, and any details discussed with reference to these figures may be included. Decompression logic circuitry 510 may include weight decoder 511 configured to decode encoded weights 514. In some examples, decompression logic circuitry 510 includes multiple weight decoders 511 (e.g., two or more), each weight decoder 511 (e.g., in parallel or sequentially) decoding a separate portion of encoded weights 514. Encoded weights 514 may be generated according to any of the encoding schemes discussed herein. In some examples, encoded weights 514 include an index weight word 518 followed by a sequence of one or more unpruned weight words 520 (if any). The index weight word 518 includes a sequence of bits (e.g., four bits, eight bits, sixteen bits, etc.), where each bit has a first bit value (e.g., "0") or a second bit value ("1"). The first bit value may indicate a pruned weight word (e.g., a weight word that has been removed from encoded weights 514). The second value can indicate the uncropped weight word 520 (e.g., the uncropped weight word 520 following the index weight word 518).

[0067] The weight decoder 511 includes an index word decoder 544, a plurality of shifters 540, and a plurality of digital logic gates 542. The shifters 540 include a first shifter 540-1, a second shifter 540-2, and so on, up to an Nth shifter 540-N. The number of shifters 540 may depend on the amount of decoded words provided to the input-weight multiplier 508. In some examples, the shifters 540 are connected in parallel to the index word decoder 544, which allows for relatively fast decoding. In some examples, each shifter 540 is configured to receive from the index word decoder 544 a sequence of untrimmed weight words 520 following an index weight word 518, along with control signals that determine the shift operation to be applied to the sequence of untrimmed weight words 520. In some examples, the shifters 540 include one or more digital circuits that can shift a data word by a specified number of bits. In some examples, the shifters 540 are a series of multiplexers. In some examples, the shifters 540 include cascaded flip-flops. In some examples, shifter 540 includes one or more cylindrical shifters.

[0068] Digital logic gates 542 may include first digital logic gate 542-1, second digital logic gate 542-2, and third digital logic gates 542-3 through Nth digital logic gates 542-N. Second digital logic gates 542-2 and third digital logic gates 542-3 through Nth digital logic gates 542-N may be connected to the outputs of corresponding shifters 540. In some examples, the first decoded word does not require shifting operations and therefore does not require shifter 540 (e.g., first digital logic gate 542-1 receives the untrimmed weighted word 520 and control signals from the index word decoder 544). In some examples, each digital logic gate 542 includes an AND gate configured to perform an AND operation. A first digital logic gate 542-1 is configured to receive a non-pruned weight word 520 (e.g., a shift operation will not be applied to the non-pruned weight word 520 to recover the first decoded word), a second digital logic gate 542-2 is connected to the output of the first shifter 540-1, and third digital logic gates 542-3 through Nth digital logic gates 542-N are respectively connected to second shifters 540-2 through Nth shifters 540-N. Each digital logic gate 542 receives a control signal from the index word decoder 544 to control a Boolean operation (e.g., an AND operation) implemented by the corresponding digital logic gate 542. For example, the first digital logic gate 542-1 may receive a control signal "1" or "0" and receive the non-pruned weight word 520, and perform a Boolean operation using "1" or "0" and the non-pruned weight word 520. The output of the first digital logic gate 542-1 is decode word 1, the output of the second digital logic gate 542-2 is decode word 2, the output of the third digital logic gate 542-3 is decode word 3, and the output of the Nth digital logic gate 542-N is decode word N.

[0069] Specifically, the index word decoder 544 receives the index weight word 518 and controls the shifter 540 and digital logic gates 542 based on the index weight word 518 to obtain decoded words 1 to N. Control signals (or at least a portion thereof) generated by the index word decoder 544 to control shifts performed by the shifter 540 may be based on the bit values ​​of the index weight word 518. Additionally, control signals (or at least a portion thereof) generated by the index word decoder 544 to control operations performed by the digital logic gates 542 may be based on the bit values ​​of the index weight word 518.

[0070] use Figure 4 and Figure 1DFor example, the index weight word 518 can be "10101101" (e.g., an 8-bit index word). The first digital logic gate 542-1 can receive a sequence of untrimmed weight words 520 (e.g., weight words 1 to N). Control signals generated by the index word decoder 544 for controlling the first digital logic gate 542-1 can be based on the bit value of the first bit. The first digital logic gate 542-1 can receive untrimmed weight words 520, and in some examples, because the first bit in the index weight word 518 includes the logic value "1," indicating the presence of an untrimmed weight word 520, the index word decoder 544 can generate a control signal "1" and send this control signal to the first digital logic gate 542-1. The first digital logic gate 542-1 can use "1" and the untrimmed weight word 520 to perform an AND operation, which produces a first decoded word 1 ("0000aaaa").

[0071] The first shifter 540-1 can receive the sequence of untrimmed weight words 520, and because the second bit in the index weight word 518 has a value of "0", the index word decoder 544 can control the first shifter 540-1 so that the untrimmed weight word 520 is not shifted (or any shift value is transmitted because the output of the shifter will not be used). The second digital logic gate 542-2 can receive the output of the first shifter 540-1, and in some examples, because the second bit in the index weight word 518 includes the logic value "0", the index word decoder 544 can generate a control signal "0" and send this control signal to the second digital logic gate 542-2. The second digital logic gate 542-2 can use "0" and the output of the first shifter 540-1 to perform an AND operation, which produces decoded word 2 ("00000000").

[0072] The second shifter 540-2 can receive the sequence of unpruned weight words 520, and because the third bit in the index weight word 518 has a value of "1", indicating the presence of unpruned weight word 520, the index word decoder 544 can control the second shifter 540-2 to shift the unpruned weight word 520 (by one byte or eight bits) so that the next eight word boundaries ("bbbb0000") are aligned at the output of the second shifter 540-2. The third digital logic gate 542-3 can receive the output of the second shifter 540-2, and in some examples, because the third bit in the index weight word 518 includes the logic value "1", the index word decoder 544 can generate a control signal "1" and send this control signal to the third digital logic gate 542-3. The third digital logic gate 542-3 can use the "1" and the output of the second shifter 540-2 to perform an AND operation, which produces decoded word 3 ("bbbb0000"). This process will continue until all weight words are recovered from encoded weight 514. (Relative to...) Figure 1D and Figure 4 For example, due to Figure 5 Each decoded word contains two weight words, for a total of sixteen weight words (e.g., Figure 1D As shown), the number of shifters 540, N will be seven, and the number of digital logic gates 542, N will be eight, thus obtaining decoded words 1 to 8.

[0073] Figure 6 A neural network circuit 604 according to one aspect is shown. The neural network circuit 604 can be... Figures 1A to 1D Examples of neural network circuits 104, and may include any details and / or information regarding these figures. Figures 2 to 5 Details are provided below. The neural network circuit 604 includes a processor memory 650, input / output (I / O) components 652, a processor data bus 654, an accelerometer 656, and a processor 658. In some examples, the processor 658 is a host processor. In some examples, the neural network circuit 604 is a system-on-a-chip (SOC) (e.g., an integrated circuit coupled to a semiconductor substrate). In some examples, the neural network circuit 604 is part of a speech or voice recognition device. In some examples, the neural network circuit 604 is part of a hearing aid device. While the following description relates to a speech or voice recognition device, the concepts discussed herein can be applied to other applications.

[0074] The neural network circuit 604 can receive input values ​​from I / O component 652 (e.g., a microphone) and recognize those input values ​​by processing a neural network trained to identify specific input values ​​as having specific meanings. For example, the input value could be Mel-frequency cepstral coefficients (MFCCs) generated from an audio stream. In some examples, frame audio samples are captured periodically (e.g., every 10 milliseconds) and converted to the frequency domain for input into the neural network (e.g., ...). Figure 1A The neural network 106).

[0075] Processor 658 is coupled to processor data bus 654. In some examples, processor 658 may perform a portion of the neural network processing (e.g., no processing, partial processing) via software running on processor 658. Processor memory 650 is coupled to processor data bus 654. In some examples, processor memory 650 includes... Figure 1A The weight memory 112 stores the encoded weights 114 of Figure 1. Accelerator 656 is coupled to the processor data bus 654. Accelerator 656 can be an example of accelerator 156 of Figure 1, which includes... Figure 1A The decompression logic circuit 110 and the input-weighted multiplier 108.

[0076] Accelerator 656 can perform a portion of the neural network processing (e.g., full processing, partial processing). In some examples, accelerator 656 may use the same processor data bus 654 and the same processor memory 650 as processor 658. Accelerator 656 may use processor data, but only when that processor data is not being used by processor 658. In implementations that split the neural network task (e.g., computation) between accelerator 656 and processor 658, accelerator 656 may trigger processor 658 to execute the task by generating an interrupt. Upon receiving the interrupt, processor 658 may read input values ​​from (shared) processor memory 650, execute the task, write the result to processor memory 650, and return control to (i.e., restart) accelerator 656.

[0077] Figure 7 An example of an accelerator 756 according to one aspect is shown. Accelerator 756 can be... Figure 1A An example of accelerator 156. Accelerator 756 includes an input data extractor 760, decompression logic circuitry 710, bias extractor 762, and output writer 764. Additionally, accelerator 756 includes a counter logic unit 766, which is configured to generate interrupt commands and... Figure 6 The processor memory 650 interacts with the processor. Each of the input data extractor 760, decompression logic circuitry 710, bias extractor 762, and output writer 764 can interact with the processor data bus (e.g., ...). Figure 6 The processor data bus 654 interacts with the input data extractor. In some examples, the input data extractor 760 is a ring buffer. The input data extractor 760 can receive and store audio samples converted to the frequency domain. The input data extractor 760 can hold the audio length for the neural network to run (e.g., 0.4 seconds to 2 seconds). The decompression logic circuit 710 can be... Figure 1A An example of decompression logic circuit 710. Decompression logic circuit 710 can be found from... Figure 6 The processor memory 650 retrieves the encoded weights and decodes them.

[0078] Accelerator 756 also includes an input register 770 configured to receive input data from input data extractor 760, and a weight register 768 configured to receive decoded weights from decompression logic circuitry 710. Accelerator 756 includes multiple input-weight multipliers 708 that multiply the decoded weights from weight register 768 with the input data from input register 770. Input-weight multipliers 708 include four multipliers, for example, a first input-weight multiplier 708-1, a second input-weight multiplier 708-2, a third input-weight multiplier 708-3, and a fourth input-weight multiplier 708-4. Although... Figure 7 Four input-weight multipliers 708 are shown, but the number of input-weight multipliers 708 can be any number greater than four, such as twenty, forty, sixty, etc. Accelerator 756 includes: a summing unit 772 configured to sum the results of the input-weight multipliers 708; an accumulator register 774 for receiving the results of the summing unit 772; and an accumulator 776 for accumulating the contents of the accumulator register 774. Accelerator 756 includes a bias adder 778 that receives a bias from a bias extractor 762 and adds the bias to the output of the accumulator 776. Accelerator 756 includes a lookup table (LUT) 601 implementing activation functions and a multiplexer 782 configured to generate the output of neural network layers.

[0079] The operation of accelerator 756 typically involves processing multiple neurons (e.g., four as shown in the figure) at multiple synapses (i.e., weights). In the first loop, the four synapses associated with the first neuron are multiplied by four inputs (e.g., layer inputs), and the sum is stored in one of the accumulator registers 774. In the next loop, a different set of synapses associated with the second neuron are multiplied by the (same) four inputs, and the accumulated sum is stored in the next register of accumulator register 774. This process is repeated until all accumulator registers 774 are written. Once all accumulator registers 774 are written, a new set of four inputs for the first neuron is obtained, multiplied by the weights, and accumulated with the previously stored register values. This process continues until each node in the layer has been computed. At this point, a bias is applied to the neuron values ​​by bias adder 778, and an activation function is applied to the neuron values ​​by lookup table 780, and then applied to multiplexer 782.

[0080] In some examples, accelerator 756 allows software control of neural network processing and allows activation functions to be applied, either in hardware or software. The application of the activation function can be configured by selecting one of the inputs to multiplexer 782. The upper input of multiplexer 782 is selected when using hardware, and the lower input of multiplexer 782 is selected when using software. When the activation function is applied in hardware, the activation values ​​can be written back, and this can be done without interaction with the host processor (e.g., ...). Figure 6 The entire layer is processed in interaction with the processor 658. During operation, a bias can be retrieved from memory and added to an accumulated sum. An activation function can then be executed in hardware, and the resulting neuron values ​​are stored in memory. This process can be repeated for other neurons in the layer. After processing and storing multiple neurons, the process can be repeated (by counter logic unit 766) for the host processor (e.g., ...). Figure 6 The processor 658 generates an interrupt. Upon receiving the interrupt and after updating the registers, the host processor (e.g., ...) Figure 6 The processor 658 can restart the accelerator 756 again for the next layer and repeat the process until the entire neural network has been processed.

[0081] Figure 8 An example of a decompression logic circuit 810 is shown. The decompression logic circuit 810 can be... Figure 1A The decompression logic circuit 110 and / or Figure 7 Examples of decompression logic circuitry 710 are provided, and may include any details discussed herein. Decompression logic circuitry 810 includes shifter 880, weight word buffer 882, first weight decoder 811-1, second weight decoder 811-2, control logic unit 888, shifter 890, and weight register 868.

[0082] Shifter 880 can move encoded weights 814 from weight memory (e.g., according to weight buffer fill level 884 calculated by control logic unit 888) based on weight buffer fill level 884. Figure 6 Processor memory 650 or Figure 1A The weight memory 112 is aligned to the weight word buffer 882. In some examples, the control logic unit 888 is configured to transmit a weight memory read request to the memory (via arrow 813) and receive an acknowledgment message (via arrow 807). In some examples, the control logic unit 888 is configured to determine the memory read state 892 to determine the level of the encoded weight 814 read from the memory. The encoded weight 814 may be encoded according to any encoding technique discussed herein.

[0083] Shifter 880 can be retrieved from memory (e.g., Figure 6 Processor memory 650, Figure 1AThe weight memory 112 receives the encoded weight 814 and aligns it into the weight word buffer 882. The weight word buffer 882 may temporarily store the encoded weight 814 until it is provided to the first weight decoder 811-1 (via arrow 819). In some examples, the first weight decoder 811-1 is configured to decode any number of weight words up to a maximum number within a single clock cycle. The maximum number of weight words could be eight weight words, sixteen weight words, thirty-two weight words, etc.

[0084] In one example, if the maximum number of weight words that can be decoded by the first weight decoder 811-1 is sixteen, but the total number of weight words is thirty-two, then the first weight decoder 811-1 can provide the remaining weight words to the second weight decoder 811-2 (via arrow 821). Then, if any remaining words exist, the second weight decoder 811-2 provides those remaining words back to the weight word buffer 882 (via arrow 837). Thus, in some examples, the first weight decoder 811-1 can decode a portion of the encoded weights 814, and the second weight decoder 811-2 can decode another portion of the encoded weights 814. In some examples, the first weight decoder 811-1 receives a first index weight word and a sequence of unpruned weight words, then decodes the first index weight word to obtain a first set of weight words, and the second weight decoder 811-2 receives a second index weight word (e.g., a subsequent index weight word) and the remaining unpruned weight words, then decodes the second index weight word to obtain a second set of weight words. Although two weight decoders 811 are shown, the decompression logic circuit 810 may include any number of weight decoders 811 (depending on the specific implementation).

[0085] The first weight decoder 811-1 may receive a signal (via arrow 815) from control logic unit 888, which indicates the number of available buffer words. For example, if the compression scheme uses an octet compression size, each weight word is eight bits (or one byte). If weight word buffer 882 contains four weight words (e.g., one index weight word plus three untrimmed weight words), the signal from control logic unit 888 will indicate the number four for the number of available buffer words.

[0086] In other words, control logic unit 888 can monitor and track weight buffer fill level 884 (e.g., the amount of weight words in weight word buffer 882). The number of available buffer words can be the amount of weight words stored in weight word buffer 882 that can be used for decoding. In some examples, first weight decoder 811-1 can receive the amount of available buffer words in weight word buffer 882 (via arrow 815) to determine if there are a sufficient number of weight words to decode. In some examples, first weight decoder 811-1 can also receive a signal from control logic unit 888 (via arrow 817) that includes a decoding request. In some examples, the decoding request includes the number of weights requested. For example, control logic unit 888 can determine the amount of weights to be decoded by first weight decoder 811-1 and transmit that number in a signal to first weight decoder 811-1 (via arrow 817). In some examples, the first weight decoder 811-1 may compare the number of requested weights (received via arrow 817) with the number of available buffer words (received via arrow 815) to determine whether to decode the encoded weights 814 in the weight word buffer 882. For example, if the number of available buffer words is equal to or greater than the number of requested weights, the first weight decoder 811-1 may decode the weight word of the encoded weights 814 and provide the decoded word to the shifter 890 (via arrow 835).

[0087] In some examples, the first weight decoder 811-1 generates a signal and sends it to the second weight decoder 811-2 (via arrow 823), where the signal indicates the amount of words used (e.g., the amount of weight words decoded by the first weight decoder 811-1). Additionally, the first weight decoder 811-1 generates a signal and sends it to the second weight decoder 811-2 (via arrow 825), where the signal indicates the number of available decoded words. In some examples, the first weight decoder 811-1 generates a signal and sends it to the second weight decoder 811-2 (via arrow 821), where the signal includes remaining words. Remaining words can be weight words that are still available (e.g., weight words exceeding the maximum number of weight words that can be decoded by the first weight decoder 811-1 and are available in the weight word buffer 882).

[0088] The second weight decoder 811-2 can receive the remaining words (via arrow 821), the number of words used (via arrow 823), and the number of decoded words available from the first weight decoder 811-1 (via arrow 825). Additionally, the second weight decoder 811-2 can receive decoding requests from the control logic unit 888 (via arrow 827). The second weight decoder 811-2 can decode the remaining words from the encoded weight 814 and provide the decoded words to the shifter 890 (via arrow 829). If any remaining words remain after the second weight decoder 811-2 performs the decoding operation, it returns those remaining words to the weight word buffer 882 (via arrow 837). Furthermore, the second weight decoder 811-2 can generate a signal and send it to the control logic unit 888 (via arrow 831), where the signal indicates the number of available decoded words. Additionally, the second weight decoder 811-2 can generate a signal and send the signal to the control logic unit 888 (via arrow 833), wherein the signal indicates the number of words used (e.g., the number of weight words decoded by the second weight decoder 811-2).

[0089] The decompression logic circuit 810 includes a shifter 890 and a weight register 868. The weight register 868 can be... Figure 7 An example of weight register 768. Shifter 890 may receive decoded words from a first weight decoder 811-1 (via arrow 835) and / or a second weight decoder 811-2 (via arrow 829) and align the decoded words into the appropriate weight register 868. The decoded words from weight register 868 are then provided to the corresponding input-weight multiplier 808. Input-weight multiplier 808 may be... Figure 7 An example of an input-weighted multiplier 708. In some examples, control logic 888 may communicate with shifter 890 to monitor the amount of weight in weight register 868. For example, control logic 888 may determine weight register fill level 886 based on information received from shifter 890, and control the amount of decoded word placed in weight register 868 based on weight register fill level 886.

[0090] In some examples, a new decoded weight word is provided to the weight register 868 in each clock cycle. In some examples, the number of input-weight multipliers 808 is sixteen, and the decompression logic circuitry 810 is configured to decode sixteen weights in one clock cycle. In some examples, the clock cycle is greater than twenty MHz. In some examples, the clock cycle is greater than forty MHz. In some examples, the clock cycle is greater than sixty MHz.

[0091] The following description illustrates exemplary operation of a decompression logic circuit 810 for decompressing eighty weights, which have been based on... Figure 4 Encoding scheme 413 uses weight pruning for compression. Table 1 below shows the contents of a 32-bit memory that stores the compressed weights in little-endian format (e.g., the first weight word is in the top right box and read from right to left). For example, these compressed weights could be stored in... Figure 1A Weight memory 112 or Figure 6 The processor memory is 650.

[0092] Table 1

[0093] cccc0000 bbbb0000 0000aaaa 10101101 0000ffff 01101010 0000eeee dddd0000 00000100 kkkkjjjj iiii0000 hhhhgggg 0000mmmm 11010001 00000000 llll0000 XXXXXXXX qqqq0000 0000pppp oooonnnn

[0094] In this example, the weight word size is four bits, and the compressed word size is eight bits, with bold numbers indicating the index weight word (e.g., Figure 4 The index weight word 418). In some examples, the decompression logic circuit 810 decodes the weights stored in Table 1 to obtain eighty weights (e.g., forty weight words, each weight word being four bits), which are then provided to the sixteen-input weight multiplier 808. With this configuration, a maximum of eight weight words can be requested, thus only the first weight decoder 811-1 is used.

[0095] In the initial state, the length of the weight word buffer 882 is zero, and the content of the weight word buffer 882 is empty. The first weight decoder 811-1 receives the number of requested weights from the control logic unit 888 (via arrow 817). In this example, since the first weight decoder 811-1 can decode eight weights, the number of requested weights is eight. Additionally, the first weight decoder 811-1 receives the number of available words (via arrow 815). In this example, since the weight word buffer 882 is empty (e.g., no words are available yet), the number of available words is zero. Because the number of available words is less than the number of requested weight words, the first weight decoder 811-1 is configured not to perform a decoding operation. Furthermore, the first weight decoder 811-1 can determine that the number of words used is zero (e.g., because no weight words are decoded) and the number of remaining words is zero.

[0096] The decompression logic circuit 810 can read the encoded weights 814 from memory. The following description uses a 32-bit read (e.g., four bytes of data, since the compressed word size is eight bits (or one byte), so four weight words will be read in each read). However, the decompression logic circuit 810 can read any number of bits from memory, including memory reads greater than 32 bits.

[0097] For example, decompression logic circuit 810 is configured to perform a first read of memory contents, wherein shifter 880 aligns the following weights in weight word buffer 882: cccc0000; bbbb0000; 0000aaaa; 10101101. After the first read, the buffer length of weight word buffer 882 is four (e.g., four bytes of data), and the buffer contents of weight word buffer 882 include: cccc0000; bbbb0000; 0000aaaa; 10101101. First weight decoder 811-1 receives the number of requested weights from control logic unit 888 (via arrow 817). In this example, after the first read, the number of requested weights is eight because first weight decoder 811-1 is capable of decoding eight. Additionally, first weight decoder 811-1 receives the number of available decodeable weight words (via arrow 815). In this example, the number of available decodeable weight words is zero. The first weight decoder 811-1 can count the number of 1s in the index weight word (the count is five). Since the weight word buffer 882 only contains three bytes of data (but five bytes of data are needed to decode the first index weight word), the first weight decoder 811-1 is configured not to perform a decoding operation because there is not a sufficient number of weight words in the weight word buffer 882. In addition, the first weight decoder 811-1 can determine that the number of words used is zero (e.g., because no weight words are decoded) and the number of remaining words is four (e.g., because they are four octet weight words).

[0098] Then, the decompression logic circuit 810 is configured to perform a second read of the memory contents (e.g., another 32 bits read from the memory). After the second read, the buffer length of the weight word buffer 882 is eight (e.g., eight bytes of data), and the buffer contents of the weight word buffer 882 include: 0000ffff; 01101010; 0000eeee; dddd0000; cccc0000; bbbb0000; 0000aaaa; 10101101. The first weight decoder 811-1 receives the number of requested weights from the control logic unit 888 (via arrow 817). In this example, after the second read, the number of requested weights is eight because the first weight decoder 811-1 is capable of decoding eight. Additionally, the first weight decoder 811-1 receives the number of available decodeable weight words (via arrow 815). In this example, the number of available decodeable weight words is eight. The first weight decoder 811-1 counts the number of 1s in the index weight word, and since the number of weight words in the weight word buffer 882 is equal to or greater than the number of 1s in the index weight word, the first weight decoder 811-1 is configured to decode the weight words to obtain a set of decoded weight words. The first weight decoder 811-1 is configured to decode six weight words (e.g., the index weight word "10101101" plus five weight words following the index word in little-endian format). In other words, to determine the number of words used, the first weight decoder 811-1 may count the number of 1s in the index weight word (which is five), and then add one word to the index weight word itself, for a total of six words used. The first weight decoder 811-1 may then determine the number of remaining words, which in this example is two (e.g., the amount of weight words (8) minus the amount of words used (6)).

[0099] Then, the decompression logic circuit 810 is configured to perform a third read of the memory contents. After the third read, the buffer length of the weight word buffer 882 is six (e.g., six bytes of data), and the buffer contents of the weight word buffer 882 include: 00000100; kkkkjjjj; iiii0000; hhhhgggg; 0000ffff; 01101010. The first weight decoder 811-1 receives the number of requested weights from the control logic unit 888 (via arrow 817). In this example, after the third read, the number of requested weights is eight because the first weight decoder 811-1 is capable of decoding eight. Additionally, the first weight decoder 811-1 receives the number of available decoded weight words (via arrow 815). In this example, the number of available words is eight. Similarly, the first weight decoder 811-1 determines the number of 1s in the index weight word (which is four), and since the weight word buffer 882 contains at least four bytes of data, the first weight decoder 811-1 is configured to decode the weight. After the third read, the first weight decoder 811-1 is configured to decode five weight words (e.g., the index weight word "01101010" plus the four weight words following the index word in little-endian format). In other words, to determine the number of words used, the first weight decoder 811-1 may count the number of 1s in the index weight word (which is four), and then add one word to the index weight word itself for a total of five words used. The first weight decoder 811-1 may then determine the number of remaining words, which in this example is one (e.g., the amount of weight words in the buffer (6) minus the amount of words used (5)).

[0100] Then, the decompression logic circuit 810 is configured to perform a fourth read of the memory contents. After the fourth read, the buffer length of the weight word buffer 882 is five (e.g., five bytes of data), and the buffer contents of the weight word buffer 882 include: 0000mmmm; 11010001; 00000000; llll0000; 00000100. The first weight decoder 811-1 receives the number of requested weights from the control logic unit 888 (via arrow 817). In this example, after the fourth read, the number of requested weights is eight because the first weight decoder 811-1 is capable of decoding eight. Additionally, the first weight decoder 811-1 receives the number of available decoded weight words (via arrow 815). In this example, the number of available decoded weight words is eight. Because the number of 1s in the index weight word is equal to or greater than the number of weight words in the weight word buffer 882, the first weight decoder 811-1 is configured to decode the weights. After the fourth read, the first weight decoder 811-1 is configured to decode the weights using two weight words (e.g., the index weight word "00000100" plus a weight word following the index word in little-endian format). In other words, to determine the number of words used, the first weight decoder 811-1 may count the number of 1s in the index weight word (which is one), and then add a word to the index weight word itself, for a total of two used words. The first weight decoder 811-1 may then determine the number of remaining words, which in this example is three (e.g., the amount of weight words in the buffer (5) minus the amount of used words (3)).

[0101] Then, the decompression logic circuit 810 is configured to perform a fifth read of the memory contents. After the fifth read, the buffer length of the weight word buffer 882 is seven (e.g., seven bytes of data), and the buffer contents of the weight word buffer 882 include: XXXXXXXX; qqqq0000; 0000pppp; oooonnnn; 0000mmmm; 11010001; 00000000. The first weight decoder 811-1 receives the number of requested weights from the control logic unit 888 (via arrow 817). In this example, after the fifth read, the number of requested weights is eight because the first weight decoder 811-1 is capable of decoding eight. Additionally, the first weight decoder 811-1 receives the number of available decoded weight words (via arrow 815). In this example, the number of available decoded weight words is eight. Since the number of 1s (which is zero) is equal to or greater than the number of weight words in the weight word buffer 882, the first weight decoder 811-1 is configured to decode that weight. After the fifth read, the first weight decoder 811-1 is configured to decode the weight using a weight word (e.g., the index weight word "00000000"). The first weight decoder 811-1 can then determine the number of remaining words, which in this example is six (e.g., the amount of weight words in the buffer (7) minus the amount of words used (1)).

[0102] After the fifth read, since the contents of the memory have already been read, there is no need to read from the memory again. After the fifth read, the buffer length of the weight word buffer 882 is now six, and the buffer contents of the weight word buffer 882 include: XXXXXXXX; qqqq0000; 0000pppp; oooonnnn; 0000mmmm; 11010001. The number of requested weights is eight, and the number of decoded weight words is eight. The first weight decoder 811-1 decodes weight 814 using five weight words, and the remaining word is one (e.g., XXXXXXXX). In this way, all decoded weights are provided to the input-weight multiplier.

[0103] Figure 9 An example of a weight decoder 911 based on one aspect is shown. The weight decoder 911 can be... Figure 8 The first weight decoder 811-1 or the second weight decoder 811-2. Alternatively, the weight decoder 911 can be... Figure 5 An example of a weighted decoder 511.

[0104] The weight decoder 911 is configured to decode the encoded weights. The encoded weights can be generated according to any encoding scheme discussed herein. In some examples, the encoded weights consist of an index weight word followed by a sequence of one or more uncropped weight words (if any). The index weight word consists of a sequence of bits (e.g., four bits, eight bits, sixteen bits, etc.), where each bit has a first bit value (e.g., "0") or a second bit value ("1"). The first bit value may indicate a uncropped weight word (e.g., a weight word that has been removed from the encoded weights). The second bit value may indicate an uncropped weight word (e.g., an uncropped weight word following the index weight word).

[0105] The weight decoder 911 includes an index word decoder 944, a plurality of shifters 940, and a plurality of AND gates 942. The shifters 940 include first shifters 940-1, second shifters 940-2, third shifters 940-3, and fourth shifters 940-4 through Nth shifters 940-N. In some examples, the shifters 940 are connected in parallel to the index word decoder 944, which allows for relatively fast decoding. In some examples, each shifter 940 is configured to receive a sequence of unpruned weight words following the index weight words, and a control signal from the index word decoder 944, which determines the shift operation to be applied to the sequence of unpruned weight words. For example, in Figure 9 In this context, the first character (character 0) can be considered an index-weighted character, and the subsequent characters (character 1…n) can be considered non-clipping-weighted characters.

[0106] The weighted decoder 911 can receive the number of words used from the previous decoder (e.g., if the weighted decoder 911 is...). Figure 8 The second weighted decoder 811-2), the number of available words (e.g., Figure 8 The weight decoder 911 can calculate the number of words used, such as the number of weight words (including uncropped weight words and indexed weight words) used for decoding weights, in addition to the number of available weight words in the weight word buffer 882. The weight decoder 911 can also generate and send the number of available weight words that have been decoded.

[0107] AND gate 942 may include a first AND gate 942-1, a second AND gate 942-2, and third AND gates 942-3 through Nth AND gates 942-3. Each AND gate 942 may be connected to the output of a separate shifter 940, except for the first shifter 940-1 (which outputs a residual word). Figure 9In the example, the first AND gate 942-1 is connected to the output of the second shifter 940-2, the second AND gate 942-2 is connected to the output of the third shifter 940-3, and the third AND gate 942-3 to the Nth AND gate 942-N are respectively connected to the fourth shifter 940-4 to the Nth shifter 940-N.

[0108] Each AND gate 942 receives a control signal from the index word decoder 944 to control the AND operation implemented by the corresponding AND gate 942. For example, the first AND gate 942-1 may receive a control signal "1" or "0" and receive the output of the second shifter 940-2, and use "1" or "0" and the output of the second shifter 940-2 to perform a Boolean operation. The output of the first AND gate 942-1 is decoded word 0, the output of the second AND gate 942-2 is decoded word 1, the output of the third AND gate 942-3 is decoded word 2, and the output of the Nth AND gate 942-N is decoded word N.

[0109] Specifically, the index word decoder 944 receives the index weight word (word 0) and controls the shifter 940 and AND gate 942 based on the index weight word to obtain decoded words 0 to N, and provides the remaining words to another weight decoder or provides a return buffer. Control signals (or at least a portion of these control signals) generated by the index word decoder 944 to control the shifts performed by the shifter 940 may be based on the bit values ​​of the index weight word. Additionally, control signals (or at least a portion of these control signals) generated by the index word decoder 944 to control the operations performed by the AND gate 942 may be based on the bit values ​​of the index weight word (e.g., index weight words 118, 318, 418, or 518).

[0110] use Figure 4 Encoding scheme 413 (and Figure 1D The index weight word can be "10101101" (e.g., an 8-bit index word). The first shifter 940-1 can receive a sequence of untrimmed weight words (words 1…n), and the index word decoder 944 can control the first shifter 940-1 to shift by five bytes, so that the remaining word is output from the first shifter 940-1. For example, the index word decoder 944 can count the number of 1s in the index weight word, and since the number of 1s is five (meaning five weight words follow the index weight word), the index word decoder 944 can control the first shifter 940-1 to shift by five bytes (e.g., each compressed weight word in…n). Figure 4 The encoding scheme 413 uses eight bits, which allows the remaining words to be provided to the next decoder or to the buffer.

[0111] The second shifter 940-2 can receive a sequence of unpruned weighted words, and the index word decoder 944 can control the second shifter 940-1 to not shift the unpruned weighted words, such that the first compressed weighted word (e.g., word 1) is aligned with the second shifter 940-2 of "0000aaaa". In some examples, the second shifter 940-2 is omitted because it will not need to be shifted (e.g., similar to...). Figure 5 (Implementation scheme). In those examples, the arrow from the undecoded word is extended to the first AND gate 942-1. The control signal generated by the index word decoder 944 to control the first AND gate 942-1 (connected to the second shifter 940-2) can be based on the bit value of the first bit of the index weight word. The first AND gate 942-1 can receive the output of the second shifter 940-2, and in some examples, because the first bit in the index weight word includes the logic value "1", which indicates the presence of a non-clipped weight word, the index word decoder 944 can generate a control signal "1" and send that control signal to the first AND gate 942-1. The first AND gate 942-1 can use "1" and the output of the second shifter 940-2 to perform an AND operation, which produces the decoded word 0 ("0000aaaa").

[0112] The third shifter 940-3 can receive a sequence of untrimmed weight words (words 1…2), and because the second bit in the index weight word has a value of "0", the index word decoder 944 can control the third shifter 940-3 to not shift the untrimmed weight words (or transmit any shift value because the shifter's output will not be used). In some examples, the third shifter 940-3 can shift with only words 1 and 2 as inputs; therefore, the third shifter 940-3 can be a two-input multiplexer. Additionally, the control signal generated by the index word decoder 944 to control the second AND gate 942-2 can be based on the bit value of the second bit in the index weight word. The second AND gate 942-2 can receive the output of the third shifter 940-9, and in some examples, because the second bit in the index weight word includes the logic value "0", the index word decoder 944 can generate a control signal "0" and send this control signal to the second AND gate 942-2. The second AND gate 942-2 can use "0" and the output of the third shifter 940-3 to perform an AND operation, which produces the decoded word 1 ("00000000").

[0113] The fourth shifter 940-4 can receive a sequence of unpruned weight words (words 1…3), and because the third bit in the index weight word has a value of "1", indicating the presence of an unpruned weight word, the index word decoder 944 can control the fourth shifter 940-4 to shift the unpruned weight word (by one byte or 8 bits) so that the subsequent weight word ("bbbb0000") is aligned with the fourth shifter 940-4. Additionally, the control signal generated by the index word decoder 944 to control the third AND gate 942-3 can be based on the bit value of the third bit in the index weight word. The third AND gate 942-3 can receive the output of the fourth shifter 940-4, and in some examples, because the third bit in the index weight word includes the logic value "1", the index word decoder 944 can generate a control signal "1" and send this control signal to the third AND gate 942-3. The third AND gate 942-2 can perform an AND operation using "1" and the output of the fourth shifter 940-4, which produces the decoded word 2 ("bbbb0000"). This process will continue until all weight words are recovered from the encoded weights. (Relative to...) Figure 1D and Figure 4 For example, due to Figure 4 Each decoded word contains two weight words, for a total of sixteen uncompressed weight words. Therefore, the number of shifters 940 will be eight, and the number of AND gates 942 will be eight, thus obtaining decoded words 0 to 7.

[0114] As described above, the truncated weight word is decoded into a zero-value weight word, and multiplication using the zero-value weight word is performed by an input-weight multiplier (e.g., input-weight multiplier 108, 508, 708, or 808). In other words, compared to some conventional methods that skip multiplication, multiplication is still performed in hardware (e.g., zeros are fed into the input-weight multiplier) even though the weight word is truncated.

[0115] Figure 10 A flowchart 1000 is shown depicting an exemplary operation of decoding the encoded weights of a neural network. Although relative to... Figures 1A to 1D The neural network system 100 is used to describe Figure 10 The flowchart 1000 is provided, but flowchart 1000 may be applied to any implementation described herein. In some examples, neural network system 100 is part of device 102 (e.g., a speech recognition device). In some examples, device 102 is a hearing aid device that receives speech (e.g., a voice command) and then uses neural network 106 to determine which command corresponds to that speech.

[0116] Operation 1002 includes retrieving encoded weight 114 from weight memory 112, wherein the encoded weight includes index weight word 118. Encoded weight 114 may be encoded according to any encoding scheme discussed herein. In some examples, index weight word 118 includes multiple bits, where each bit has a first bit value (e.g., zero) or a second bit value (e.g., one). The first bit value indicates a pruned weight word 126 in the sequence, and the second bit value indicates a uncropped weight word 120 in the sequence. Additionally, one or more uncropped weight words 120 may follow index weight word 118.

[0117] Operation 1004 includes decoding the encoded weights 114 using index weight words 118 to obtain a sequence of one or more pruned weight words 126 and one or more unpruned weight words 120. In some examples, pruned weight words 126 include zero-value weight words, and unpruned weight words 120 include non-zero-value weight words. In some examples, pruned weight words 126 are decoded to zero in response to a bit having a first value (e.g., zero) within index weight words 118. In some examples, the decoding includes controlling multiple shifters (e.g., based on index weight words 126) Figure 5 The system includes a shifter 540 and multiple digital logic gates 542. In some examples, the encoded weights 114 are decoded within a single clock cycle. In some examples, the clock cycle is between 15 MHz and 80 MHz.

[0118] Operation 1006 includes providing a sequence of unpruned weight words 120 and pruned weight words 126 to multiple input-weight multipliers 108. For example, multiplication is still performed in hardware even if the weight words are pruned, where zeros are fed to the input-weight multipliers 108.

[0119] Typical embodiments have been disclosed in the specification and / or drawings. This disclosure is not limited to such exemplary embodiments. The use of the term "and / or" includes any and all combinations of one or more of the associated listed items. The drawings are schematic representations and therefore not necessarily drawn to scale. Unless otherwise stated, particular terms are used in a general and descriptive sense and not for limiting purposes.

[0120] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art. Methods and materials similar to or equivalent to those described herein may be used in practice or testing of this disclosure. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” include multiple referents unless the context clearly specifies otherwise. The term “comprising” and its variations are used synonymously with the term “including” and its variations as used herein and are open-ended, non-limiting terms. The terms “optional” or “optionally” as used herein mean that a feature, event, or condition subsequently described may or may not occur, and the description includes instances where the feature, event, or condition occurs and instances where it does not occur. Scope may be expressed herein as from “about” a particular value and / or to “about” another particular value. When such a scope is expressed, an aspect includes from one particular value and / or to another particular value. Similarly, when a value is expressed as an approximate value using the antecedent “about,” it should be understood that the particular value forms another aspect. It should also be understood that each endpoint of a range is significant relative to another endpoint and independent of the other endpoint.

[0121] While certain features of the described embodiments have been illustrated herein, many modifications, alternatives, variations, and equivalents will now occur to those skilled in the art. Therefore, it should be understood that the appended claims are intended to cover all such modifications and variations falling within the scope of the embodiments. It should be understood that these modifications and variations are presented by way of example only and not limitation, and various changes in form and detail are possible. Any parts of the apparatus and / or methods described herein can be combined in any way, except for mutually exclusive combinations. The embodiments described herein may include various combinations and / or sub-combinations of the functions, components, and / or features of the different embodiments described.

Claims

1. A neural network circuit for decoding the weights of a neural network, the neural network circuit comprising: A weight memory configured to store encoded weights for the neural network, the encoded weights including index weight words; and Decompression logic circuit, wherein the decompression logic circuit is configured as follows: Retrieve the encoded weights from the weight memory; The encoded weights are decoded using the index weight words using multiple shifters to obtain decoded weights. The decoded weights include a sequence of one or more unpruned weight words and one or more pruned weight words. The pruned weight words include zero-value weight words, and the unpruned weight words include non-zero-value weight words. as well as The decoded weights are provided to multiple input-weight multipliers.

2. The neural network circuit according to claim 1, wherein the index weight word comprises a plurality of bits, each bit having a first bit value or a second bit value, the first bit value indicating a pruned weight word in the sequence, and the second bit value indicating a non-pruned weight word in the sequence.

3. The neural network circuit of claim 1, wherein the decompression logic circuit is configured to decode the encoded weights within a single clock cycle.

4. The neural network circuit of claim 1, wherein the decompression logic circuit includes a weight word decoder to decode the encoded weights, the weight word decoder comprising: The plurality of shifters; Multiple digital logic gates, wherein the multiple digital logic gates are coupled to the multiple shifters; and An index word decoder, coupled to the plurality of shifters and the plurality of digital logic gates, is configured to control the shift operations of the plurality of shifters and the logic operations of the plurality of digital logic gates based on the index weight words. The plurality of shifters are connected in parallel to the index word decoder. Each of the plurality of shifters is configured to receive the unpruned weight word of the encoded weight and a control signal, the control signal controlling a shift operation applied to the unpruned weight word. Each of the digital logic gates is configured to receive a control signal from the index word decoder, the control signal controlling the Boolean operation performed by the corresponding digital logic gate. The plurality of digital logic gates include a first digital logic gate and a second digital logic gate, wherein the first digital logic gate is coupled to the output of a first shifter among the plurality of shifters, and the second digital logic gate is coupled to the output of a second shifter among the plurality of shifters.

5. The neural network circuit according to claim 1, wherein the decompression logic circuit comprises: A first weight decoder is configured to decode a first portion of the encoded weights; A second weight decoder is configured to decode a second portion of the encoded weights; and Control logic circuitry, configured to control the first weight decoder and the second weight decoder. The first weight decoder is configured to receive from the control logic circuit a signal indicating the number of available weight words stored in the weight word buffer. The first weight decoder is configured to determine the number of bits having the first bit value in the index weight word, and to decode the first portion of the encoded weight in response to the number of available weight words stored in the weight word buffer being equal to or greater than the number of bits having the first bit value.

6. A computing device, the device comprising: A neural network module configured to receive a set of inputs and generate a set of outputs, the set of inputs including speech data, and the set of outputs including one or more potential speech commands corresponding to the speech data; A weight memory, configured to store encoded weights for the neural network module; and An accelerator configured to perform computations on the neural network module, the accelerator comprising: A decompression logic circuit is configured to retrieve the encoded weights from the weight memory and decode the encoded weights using multiple shifters to obtain decoded weights, the decoded weights comprising a sequence of one or more pruned weight words and one or more unpruned weight words, the pruned weight words comprising zero-value weight words and the unpruned weight words comprising non-zero-value weight words. and Multiple input-weight multipliers are configured to receive the decoded weights.

7. The computing device according to claim 6, wherein the decompression logic circuit comprises: A weight word buffer is configured to temporarily store the encoded weights, the encoded weights including a first index weight word and a second index weight word; A first weight decoder is configured to receive the encoded weights from the weight word buffer and use the first index weight word to generate a first set of decoded weight words. A second weight decoder is configured to receive a portion of the encoded weights from the first weight decoder and use the second index weight words to generate a second set of decoded weight words; A control logic component, configured to control the first weight decoder and the second weight decoder; A shifter configured to receive a first set of decoded weight words from a first weight decoder and a second set of decoded weight words from a second weight decoder; and Multiple weight registers coupled to the shifter, each of the multiple weight registers being configured to receive a separate decoded weight word from the shifter.

8. The computing device of claim 6, wherein the encoded weights include index weight words and one or more non-pruned weight words, and the decompression logic circuit includes a weight word decoder, the weight word decoder comprising: The plurality of shifters; A plurality of AND gates, wherein the plurality of AND gates are coupled to the plurality of shifters; and An index word decoder, coupled to the plurality of shifters and the plurality of AND gates, is configured to decode the encoded weights based on the bit values ​​of each bit of the index weight word by controlling the operations of the plurality of shifters and the plurality of AND gates. Each of the plurality of shifters is configured to receive the unpruned weight word of the encoded weight and a first control signal from the index word decoder, and each shifter is configured to perform a shift operation on the unpruned weight word of the encoded weight according to the first control signal. Each of the AND gates is configured as the output of a corresponding shifter and a second control signal from the index word decoder, and each AND gate is configured to perform an AND operation on the output of the corresponding shifter and the second control signal.

9. A method for decoding the weights of a neural network, the method comprising: Encoded weights, including index weight words, are retrieved from a weight memory via a processor data bus. Using a first plurality of shifters, the encoded weights are decoded using the index weight words to obtain decoded weights, the decoded weights comprising a sequence of pruned weight words and one or more unpruned weight words, the pruned weight words comprising zero-value weight words, and the unpruned weight words comprising non-zero-value weight words; as well as The decoding weights are provided to multiple input-weight multipliers using a second plurality of shifters.

10. The method of claim 9, wherein the index weight word comprises a plurality of bits, each bit having a first bit value or a second bit value, the first bit value indicating a clipped weight word in the sequence, the second bit value indicating a non-clipped weight word in the sequence, the clipped weight word being decoded to zero in response to a bit in the index weight word having the first bit value, wherein the decoding includes controlling a plurality of shifters and a plurality of digital logic gates based on the index weight word.

11. A neural network circuit for decoding the weights of a neural network, the neural network circuit comprising: A weight memory configured to store encoded weights for the neural network, the encoded weights including index weight words; and Decompression logic circuit, wherein the decompression logic circuit is configured as follows: Retrieve the encoded weights from the weight memory; The index weight words are used to decode the encoded weights to obtain a sequence of one or more unpruned weight words and one or more pruned weight words, wherein the pruned weight words include zero-value weight words and the unpruned weight words include non-zero-value weight words. as well as The sequence of the unpruned weight words and the pruned weight words is provided to a plurality of input-weight multipliers.

Images