Computing devices and systems including a skip circuit block and methods

Abstract

A computing device and system including a skip circuit block and method for performing partial and accumulated summation of the product of an input vector and a weight vector in a word-by-word input and bit-by-bit weighted manner to obtain a partially accumulated product sum. The partially accumulated product sum is compared to a threshold condition after each weight bit, and when the partially accumulated product sum satisfies the threshold condition, a skip indicator is asserted to indicate a skip and remaining computation of the accumulated summation.

Description

Technical Field

[0001] Embodiments of the present invention relate to a computing device, system, and method that includes skipping circuit blocks. Background Technology

[0002] A multiply-accumulator can be used to multiply input data with corresponding weights bit-by-bit. The output of this operation can be used in artificial intelligence networks to form connections between nodes. In this case, the output of the multiply-accumulator can be provided to an activation function. One such activation function is the rectified linear unit (RLU) activation function. The function returns 0 if the input is less than 0, and otherwise returns a value. Summary of the Invention

[0003] According to one aspect of an embodiment of the present invention, a method for product and accumulation is provided, comprising: performing partial summation on the product of an input vector and a weight vector in a word-by-word input and bit-by-bit weighted manner to obtain a partially accumulated product sum; comparing the partially accumulated product sum with a threshold condition; and asserting a skip indicator to indicate skipping the remaining computation of the accumulation when the partially accumulated product sum satisfies the threshold condition.

[0004] According to another aspect of an embodiment of the present invention, a computing device including a skip circuit block is provided, comprising: a multiplication-accumulation circuit block configured to multiply a group of M input vectors by a first weight bit of a group of M weight vectors to calculate a first group of M products, sum the first group of M products together to calculate a first product sum, and provide the first product sum on a product sum output pin group; and an accumulator circuit block including an accumulator input pin group coupled to the product sum output pin group, the accumulator circuit block being configured to replace a replacement accumulator value by receiving a first value on the accumulator input pin group and adding the first value to an accumulated value in the accumulator circuit block. The system accumulates a first value and provides the accumulated value on the accumulated value output pin group; a skip circuit block, including an accumulated value input pin group coupled to the accumulated value output pin group, is configured to compare the accumulated value with a threshold and assert a skip signal on the skip pin when the threshold is met; and a control circuit block is configured to check if there is a skip signal on the skip pin, and when the skip signal is not asserted: cause the multiplication-accumulation circuit block to process the next weight bit of the group of M weight vectors, and when the skip signal is asserted: terminate the processing of the remaining part of the group of M weight vectors, assert a ready output signal, and provide the accumulated value to the activation circuit.

[0005] According to another aspect of the present invention, a computational system including a skip module is provided, comprising: a multiply-accumulate device configured to calculate the product of an input node and a weight vector in the input in a bit-weighted manner, summing each of the products to calculate a product sum for each bit, weighting the product sum according to the bit value position of each bit, and accumulating the product sums to an accumulated product sum; a skip module configured to compare the product sum with a threshold, and assert a skip signal when the threshold is met; a control logic device configured to cause the multiply-accumulate device to process the next bit of the weight vector when the skip signal is not asserted, and to cause the multiply-accumulate device to stop processing the remaining part of the weight vector when the skip signal is asserted; and an activation device that determines the convolution value of the input node based on the value of the accumulated product sum. Attached Figure Description

[0006] The various aspects of the invention can be best understood from the following detailed description when read in conjunction with the accompanying drawings. It should be emphasized that, in accordance with standard industrial practice, the various components are not drawn to scale and are for illustrative purposes only. In fact, for clarity of discussion, the dimensions of the various components may be arbitrarily increased or decreased.

[0007] Figure 1 The illustration shows an example of a 3x3 convolution commonly used for processing image data in machine learning, according to some embodiments.

[0008] Figure 2 The illustration shows a more general representation according to some embodiments. Figure 1 The concept shown.

[0009] Figure 3 The diagram illustrates a ReLU activation function according to some embodiments.

[0010] Figure 4 The illustration shows the input data, weight vector, and binary representation of the MAC in a hardware algorithm implementation according to some embodiments.

[0011] Figure 5 The illustration shows mathematical formulas for processing input values ​​and weight vectors bitwise, according to some embodiments.

[0012] Figure 6 The illustration shows an example operation of input I and weight vector W according to some embodiments.

[0013] Figures 7 to 12 The illustration shows the skip evaluation and activation components according to some embodiments.

[0014] Figure 13 The illustration shows a process flowchart 100 for skipping the evaluation and activation components according to some embodiments.

[0015] Figures 14 to 17 Various views of a circuit block diagram of a MAC skip circuit according to some embodiments are illustrated.

[0016] Figures 18 to 22 Various views illustrating the operation of a MAC skip circuit according to some embodiments are shown.

[0017] Figure 23A and Figure 23B The illustration shows a flowchart of providing a threshold to the skip module according to some embodiments.

[0018] Figure 24A , Figure 24B and Figure 25 The diagram illustrates state machine models and signal values ​​according to some embodiments.

[0019] Figure 26 The figure illustrates a system diagram including a MAC skip circuit according to some embodiments.

[0020] Figure 27 The illustration shows the results of test data for reducing energy consumption using a MAC skip circuit according to some embodiments. Detailed Implementation

[0021] The following disclosure provides numerous different embodiments or examples for implementing various features of the invention. Specific embodiments or examples of components and arrangements are described below to simplify the invention. Of course, these are merely examples and not intended to be limiting. For example, in the following description, forming a first component above or on a second component can include embodiments where the first and second components are in direct contact, and can also include embodiments where an additional component can be formed between the first and second components, such that the first and second components are not in direct contact. Furthermore, reference numerals and / or letters may be repeated in various examples. This repetition is for simplicity and clarity and does not in itself indicate a relationship between the various embodiments and / or configurations discussed. It should be understood that signals can be asserted as high 1 or low 0, and unless the context or convention otherwise specifies, “1” as used herein is understood to mean “assertion”, and “0” as used herein is understood to mean “no assertion”. Depending on the device and design, those skilled in the art can readily reverse these signals as needed.

[0022] In the field of artificial neural networks, machine learning acquires input data, performs some operations on the input data, and then applies activation functions to process the data. The output of the activation function is essentially a simplified representation of the input data. The input data can be nodes in a layer of nodes. Figure 1The illustration shows an example of a 3x3 convolution commonly used in processing image data in machine learning. Image 10 consists of individual pixels 11. The image can be represented in a color space such as RGB (Red-Green-Blue) or HSL (Hue-Saturation-Emissivity), where each pixel is assigned a value for each color space variable. Node 12 of the image is a 3x3 block of pixels, and each pixel 11 in node 12 has an input value I for each color space variable of pixel 11 in node 12. 1-9 One possible computation in a 3x3 convolution uses a product sum operation, where each input value I... 1-9 Multiply by the weight values ​​W of weighting matrix 14 respectively 1-9 As each multiplication is performed, the sum of each product can be maintained. This product summation operation can be called multiply-accumulate computation / operation (MAC)16. The output of MAC 16 is fed to the activation function 18. In 3x3 convolutions, the activation function used is typically a rectified linear activation function (rectified linear unit or ReLU). ReLU is a piecewise function of the output y = max(0, x), where x is the result of MAC 16. Therefore, all negative values ​​are set to 0, and non-zero values ​​are linearly consistent with the input.

[0023] Figure 2 The illustration shows a more general approach (i.e., for input nodes of any length). Figure 1 The concepts shown. Input I0–I N-1 Each of them is multiplied by the weight vector W0–W N-1 These values ​​are then summed in the product summation (MAC) and subjected to the ReLU activation function. The output O is the output of the ReLU activation function.

[0024] Figure 3 The diagram illustrates a graph of the ReLU activation function and its piecewise representation. For example... Figure 3 As shown in the diagram, for all values ​​of x ≤ 0, the value of y = 0, and for all values ​​of x > 0, the value of y = x. It's worth noting that in this case, if the value of x is equal to 0, the output is "0" regardless of which segment of the function is used (therefore, if the function is defined as y = {x, x ≥ 0 | 0, x < 0}, the output will be equal). Some modifications can be made to the ReLU activation function.

[0025] So far, these calculations have been discussed in a general sense. For example, a computer program that executes on a general-purpose processor can be written, consisting of a simple for loop performing a MAC on the INPUT and WEIGHT arrays, and then passing the output of the MAC to ReLU, such as the following logic:

[0026] Initialize a counter integer to 0.

[0027] Initialize a storing variable (e.g., MAC) to 0.

[0028] Provide an INPUT array having the length n with input values.

[0029] Provide a WEIGHT array having the length n with signed weight values.

[0030] (Provide a weight array of length n with signed weight values).

[0031] For counter = 0, counter <n,counter++{

[0032] MAC=MAC+(INPUT[counter]*WEIGHT[counter])

[0033] }

[0034] Output = the maximum value of (0, MAC).

[0035] Provide output value.

[0036] For large datasets, execution on a general-purpose processor is inefficient. To improve efficiency, the algorithm can be implemented in dedicated hardware, such as in an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA). However, implementing this logic in dedicated hardware (e.g., an ASIC) involves using binary arithmetic processes within a block of digital logic. This will be described in the context of implementing MAC (and ReLU) in hardware. Hardware implementation involves calculating MAC and implementing ReLU in binary format.

[0037] Figure 4The diagram illustrates the input data, weight vector, and binary representation of the MAC algorithm implemented in hardware. The hardware implementation will be discussed in more detail below, along with a skip module. For data points in a node, the input data is displayed as unsigned values ​​(e.g., orders of magnitude) of the node. The length of the input data is N bits. For example, N can be 4 bits, 8 bits, 16 bits, etc. For example, if N is 8, each input value is between 0 and 255. The weight vector is a signed weight value in two's complement format. Therefore, negative numbers will start with 1 in the most significant bit (MSB). Each weight in the weight vector is K bits long. N can be equal to K or can be different values. For example, if K is 8 bits, each weight value can be between -128 and 127. In the representation, for input values, the i-th input corresponds to the input index of the input data point in the node. Each weight will have the i-th weight index of the corresponding weight vector. In other words, there is a one-to-one correlation between the i-th input and the i-th weight vector. Conversely, the j-th bit of each input or each weight vector is arranged from left to right, such that the MSB is bit 0, and the least significant bit (LSB) is the (N-1)-th bit of the input and the (K-1)-th bit of the weight vector. Since N and K can be different values, the j-th position in the total number of input data may differ from the j-th position in the weight vector. As an example representation, where i = 2 and j = 5, I... i,j This bit corresponds to the sixth bit of the third input data. Similarly, W for i=3 and j=4... i,j This position corresponds to the fifth position of the fourth weight vector. For example... Figure 4 As shown, the total number of bits produced by the MAC is equal to the logarithm of N plus K plus M (base 2), rounded to the nearest integer (Roundup). For example, if the number of inputs in a node is 9 (e.g., corresponding to a 9-point convolution) and N and K are both 8, then the number of bits in the MAC output is 8 + 8 + Roundup(log29) = 20. This value can also be expressed as Roundup(N + K + log2M).

[0038] Figure 5 The diagram illustrates the mathematical formula for processing input values ​​and weight vectors bitwise. Specifically, each input value is multiplied by each bit of the weight vector, and the results are summed after each iteration. The left side of the equation is the general formula for summing the products of the i inputs and the corresponding i weight vectors. Since the mathematical operation to be performed is binary, it can be broken down into the right side of the equation, which includes a first term for processing the sign bit of the weight vector and a second term for processing the remaining bits.

[0039] The first term represents the summation of the N-bit unsigned input and the sign bit of each of the K-bit signed weight vectors. For example... Figure 4As shown, the MSB of the weight vector stores the sign bit and is represented as the 0th bit of the weight vector when bit j = 0. The first term multiplies the input by the 0th bit of the weight vector (representing the sign bit) and multiplies the result by 2. K-1 The value of the 0th bit is then recorded as negative. Essentially, the product of the input and the sign bit determines the maximum negativeness of the weight vector. For example, if the weight vector is 8 bits and negative, the sign bit represents 2. 7 The position value is "1". This is equivalent to taking the 2s complement of the input and shifting it left by 7 times. This is for each input I. i The iteration is complete, and the first term represents the sum of all these products. Zeros are added if the corresponding weight vector is not negative.

[0040] The second term involves two nested summation operations. The inner summation represents the weight vector W. i Each of the remaining j bits is multiplied by the input I. i Multiply by the weight vector W i The sum of the values ​​at the corresponding j-th position in the input I. External summation for each input I... i and weight vector W i Repeat the internal summation and add all these sums together.

[0041] Figure 6 The diagram illustrates an example operation with input I and weight vector W, where M = 1, N = 8, and K = 8. I0 = 77 (01001101) and W0 = -116 (1000 1100). In the summation... In the middle, the first item can be adjusted to -1·(01001101)·(1·2 7 )=1011 0011·2 7 =1101 1001 1000 0000 (Note that the sign bit has been padded with an additional leading '1'). The second term can be adjusted to 77·(0·2) 6 )+77·(0·2 5 )+77·(0·2 4 )+77·(1·2 3 )+77·(1·2 2 )+77·(0·2 1 )+77·(0·2 0 ) = 77·2 3 +77·2 2 =616(0010 0110 1000)+308(0001 0011 0100)=924(0011 1001 1100). The sum of the first term and the second term is -8932(1101 11010001 1100).

[0042] This example shows that when the weight vector is negative, bitwise math sets the weight vector to -128 times the input, and then subsequent bits add the positive part back to the negative number (making it less negative) until the final result is reached.

[0043] When the weight vector is positive, the first term will be "0", and the second term will be the sum of the remaining bits of the weight vector, similar to the result for a negative weight vector.

[0044] about Figures 7 to 12 This paper discusses skipping the evaluation and activation components (also known as skipping modules), which, when used with compatible activation functions such as ReLU, can simplify the computational complexity of the MAC summation multiplier accumulator for the input node and the corresponding weight vector. As mentioned above, for example, the ReLU activation function provides an output of MAX(0,input), where the input equals the MAC summation. The MAC summation is determined by a series of partial accumulations as the bits of the weight vector are iteratively processed. Therefore, if the partial accumulation of the product after a particular iteration does not become positive at any time (or never satisfies the activation function's condition), it can be determined that the remaining computation can be skipped.

[0045] In the case of a weight vector W processed bit-by-bit iteratively, after each iteration, the output of the partial product and accumulation can be compared with the "worst-case" assumption for each remaining bit of the weight vector W. The "worst-case" is the case that would produce the most computation cycles, i.e., no skip conditions. Here, the worst-case scenario is where all remaining bits of the weight vector W are assumed to be equal to 1. This means, for example, an 8-bit weight vector W... i If W i Negative (i.e., starting at W) i,0 =1), then the smallest negative number less than 0 is -1 (all binary 1s). If W i Positive (i.e., starting at W) i,0 If =0), then the largest number is 127 (where the remaining bits are binary 1s).

[0046] Figure 7 The summation formula 30 is illustrated, which explains the “unknown” part of the weight vector for any given iteration n. Figure 7 The formula diagram illustrates the relationship with Figure 5The formula is similar to the one in the example. It includes a general product and summation formula 31, which is decomposed into a sign bit summation term 32 and a nested summation of the remaining bits of the weight vector. The nested summation of the remaining bits is decomposed into a first nested summation term 34 for the first n bits (n>0) of the weight vector and a second nested summation term 36 for the remaining K-1 bits. For iteration n, the first nested summation term 34 is the product summation of the known n bits of the weight vector after processing the nth bit, and the second nested summation term 36 contains the remaining bits from the weight vector W from the n+1th bit to the remaining K-1 bits. As described above, in some embodiments, the unknown weight vector bits can be assumed to be the "worst-case" where all bits are equal to 1. As the weight vector is processed bit by bit from MSB to LSB, more actual weight vector bits are known, and the effective value of the unknown hypothetical weight vector bits decreases with each iteration.

[0047] Figures 8 to 12 An application of the skip module according to some embodiments is illustrated. Specifically, Figures 8 to 12 It shows how to... Figure 13 Implementation of the flowchart (discussed below) Figure 7 The summation formula. Figures 8 to 12 Each diagram in the figure represents a set of M=9 input vectors I i Each has N=8 bits and M=9 weight vectors W. i Each has K = 8 bits. For demonstration purposes, the values ​​of each element in the input vector are randomly generated and are 77, 138, 179, 76, 175, 159, 153, 212, 128. For demonstration purposes, the values ​​of each element in the weight vector are randomly generated and are -116, 76, 90, -83, 33, 8, -60, -98, -22. Figures 8 to 12 The diagram also shows the binary representation of each of these numbers.

[0048] Figure 8 and Figure 9 The diagram illustrates the calculation of summation formula 30, where iteration n = 0. During iteration n = 0, the summation formula is adjusted at the first weight bit j = 0. Figure 8 The diagram illustrates the sign bit of each input vector multiplied by the corresponding weight vector, and also multiplied by the position value of the sign bit (2). 7 If the weight vector is positive, the sign bit is set to "0" and incremented by "0". If the weight vector is negative, the sign bit is set to "1" and incremented by a value equal to the negative input vector multiplied by 2. 7 Summing these products. For example... Figure 8As shown, the sum of these products is -103,040. This value is the most negative possible accumulated value. If all other bits in the weight vector were "0", the output value would be -103,040. If any remaining bits in the weight vector are "1", the value will cause the accumulated value to become less negative. Therefore, each other operation either has no effect on the accumulated value or has only a positive effect.

[0049] Figure 9 The diagram illustrates the processing of the remaining two terms. The first nested summation term is 34 (see...). Figure 7 There will be no operation to accumulate to it because the summation starts from j=1. In this case, n=1, j=0, so the first term will not produce any value. Suppose the second nested summation term 36 is the "worst case," as mentioned above, this is the case that will result in the most computation cycles. Here, the most computation cycles will occur if the final summation is greater than 0. Therefore, the worst case can be taken, where it is assumed that each bit of the remaining weight vector is "1". This value is 127 in decimal, and therefore each of the input values ​​is multiplied by the value of 127 of the weight vector of each corresponding assumption, and then summed together to get the number 164,719. If we compare this worst case sum with the accumulated value -103,040, we can see that the final value could be as high as -103,040 + 164,719 = 61,679. Since this value is non-negative, that is, the result after the activation function is a non-zero value, therefore more bits need to be processed.

[0050] Figure 10 The diagram illustrates the summation where n = 1, i.e., where j = 0, 1. Where j = 0 has been calculated as -103,040. Where j = 1, in the first nested summation term 34 (see...). Figure 7 In ), the input vector I i Each of the values ​​in the matrix is ​​multiplied by the corresponding weight vector (i.e., W). i,1 In ), j = 1 bit, and this value is multiplied by the position value of 1 bit (2). K-j-1 ->2 8-1-1 ->2 6 Therefore, when W i,1 When W = 0, the increment will be 0, and when W i,1 When = 1, the increment will be the corresponding input multiplied by 2. 6 These are calculated based on the outer summation of the first nested summation term 34, and then summed. In this example, the sum equals 48,448. When this value is added to the value j=0, the total is -54,592.

[0051] Assumption Figure 10The second nested summation term 36 is the "worst case," which, as mentioned above, leads to the most computation cycles, where each remaining unknown bit of the weight vector is assumed to be "1." When n = 1, this value is 63 in decimal, so each input value is multiplied by the corresponding assumed value of 63 in the weight vector, and then summed to get the number 81,711. Comparing this worst-case sum to the accumulated value -54,592, we can see that the final value could be as high as -54,592 + 81,711 = 27,119. Since this value is non-negative—meaning the result after the activation function is non-zero—more bits need to be processed. It should be noted that because the second bit of the weight vector (j = 1) is not actually the worst case, the difference between the accumulated value and the worst-case value—27,119—after processing the second bit is less positive than the difference—61,679—after processing only the first bit.

[0052] Figure 11 The diagram illustrates the summation when n=2, i.e., when j=0,1,2. When j=0,1, it has already been calculated as -54,592. When j=2, in the first nested summation term 34 (see... Figure 7 In ), the input vector I i Each of the values ​​in the matrix is ​​multiplied by the corresponding weight vector (i.e., W). i,2 In the original text, j = 2 bits, and this value is multiplied by the 2-bit bit value (2). K-j-1 ->2 8-2-1 ->2 5 Therefore, when W i,2 When W = 0, the increment will be 0, and when W i,2 When = 1, the increment will be the corresponding input multiplied by 2. 5 These are calculated and then summed based on the outer sum of the first nested summation term 34. In this example, the sum equals 17,216. When this value is added to the j=0,1 value, the total is -37,376.

[0053] Assumption Figure 11 The second nested summation term, 36, is the "worst-case" scenario, as mentioned above, which results in the most computation cycles, where each remaining unknown bit of the weight vector is assumed to be "1". When n = 2, this value is 31 in decimal, so each input value is multiplied by the corresponding assumed value of 31 in the weight vector, and then summed to obtain the number 40,207. Comparing this worst-case sum to the accumulated value of -37,376, we can see that the final value could be as high as -37,376 + 40,207 = 2,831. Since this value is non-negative, meaning the result after the activation function is non-zero, even more bits need to be processed.

[0054] Figure 12The diagram illustrates the summation for n=3, i.e., when j=0,1,2,3. When j=0,1,2, it has already been calculated as -37,376. When j=3, the summation term in the first nested term is 34 (see...). Figure 7 In ), the input vector I i Each of the values ​​is multiplied by its corresponding weight vector (i.e., W). i,3 In ), j = 3 bits, and this value is multiplied by the 3-bit bit value (2). K-j-1 ->2 8-3-1 ->2 4 Therefore, when W i,3 When W = 0, the increment will be 0, and when W i,3 When = 1, the increment will be the corresponding input multiplied by 2. 4 These are calculated based on the outer summation of the first nested summation term 34, and then summed. In this example, the sum equals 8,800. When this value is added to the values ​​j = 0, 1, 2, the total is -28,576.

[0055] Assumption Figure 12 The second nested summation term, 36, is the "worst-case" scenario, as mentioned above, which results in the most computation cycles, where each remaining unknown bit of the weight vector is assumed to be "1". When n = 3, this value is 15 in decimal, so each input value is multiplied by the corresponding assumed value of 15 in the weight vector, and then summed to get the number 19,455. Comparing this worst-case sum to the accumulated value -28,576, we can see that the final value could be as high as -28,576 + 19,455 = -9121. Since this value is negative for the "worst-case" situation, it can be determined that no remaining weight vector value would result in a non-negative sum. In other words, having no remaining bits in the weight vector when n = 4, 5, 6, 7 will result in a non-negative product and accumulation. For any remaining unprocessed bits in the weight vector, since the accumulated value is always negative, the result will always be 0 when the accumulated value is passed to the ReLU activation function. Therefore, processing any further bits would be a waste of resources. In this situation, the skip module can activate a skip signal and the next input block will be processed.

[0056] Figure 13The flowchart 100 illustrates the process of skipping the evaluation and activation components. At 110, partial sum accumulation, as part of the MAC and multiplicative sum accumulation, is performed in a word-by-word input and bit-by-bit weighted manner. This manner is as described above, where the entire unsigned input value (any bit length) is multiplied bit-by-bit by the signed weight vector. The partial sum accumulation aspect reflects the iterative process of processing the weight vector bit-by-bit. Therefore, at 110, one bit of the weight vector is processed. At 120, the product sum of the partial accumulation is evaluated against a skip condition. The skip condition can be based on a corresponding activation function. For example, in some embodiments, the activation function can be a ReLU activation function, such that if the output of the MAC and multiplicative sum is negative, the output of the activation function is zero. Therefore, the skip condition can evaluate the product sum of the partial accumulation to predict whether the MAC and multiplicative sum might be positive or negative. In some embodiments, the skip condition can be based on a predefined threshold (e.g., see...). Figure 23A and Figure 23B (and their accompanying descriptions). In other embodiments, the skip conditions can be dynamically calculated based on predictions of the remaining unprocessed weighted bits.

[0057] At 130, if the partially accumulated product sum is determined to satisfy the skip condition, then at 140, an assertion signal is given to indicate that subsequent operations can be skipped. Subsequent operations may include, for example, memory access read operations (e.g., loading input or weight values) or computation operations (e.g., subsequent iterations). If at 130 the partially accumulated product sum is determined not to satisfy the skip condition, then at 150 it is determined whether all weight bits have been processed. If all weight bits have been processed, the process ends, and the partially accumulated product sum is accumulated as the MAC and product output at 160. After the output is determined at 160, an activation function is applied to the output at 170. If at 150 all weight bits have not been processed, then the next weight bit is advanced at 180, and processing is repeated at 110. It should be noted that after 140, if the skip condition is satisfied and an assertion signal is given to skip subsequent operations, the output may optionally be the accumulated product sum, and the activation function at 170 may be applied to the output.

[0058] Figure 14A high-level circuit block diagram of the hardware implementation of the MAC skip circuit 200 is illustrated. In some embodiments, the MAC skip circuit 200 can be implemented on a single semiconductor substrate. In other embodiments, the MAC skip circuit 200 can be implemented on multiple semiconductor substrates and interconnected as needed. Input block 210 obtains the input value from unsigned input vector 205. Weight block 220 obtains its input from signed weight vector 215. Input vector 205 is multiplied by the next bit of weight vector 215. If the next bit is the first bit, the result is converted to 2s complement format and added to the left-shifted partial sum 255 (which is initialized to "0") by addition block 230, and then stored as the new partial sum 235; otherwise, the result is added to the left-shifted partial sum 255 by addition block 230. If all bits of weight vector 215 have been processed, the partial sum 235 is output 240. Otherwise, after each iteration, the partial sum 235 is evaluated by skip module 245 to determine if a skip condition exists. If so, a skip signal is asserted. If not, the process is repeated for each of the K bits in the weight vector until the skip signal is asserted or until the remainder of the K bits has been processed. If the skip signal is asserted, the partial sum 235 can be used as output 240, can be modified and then used as output 240, or can be zero as output 240.

[0059] Figure 15 A more detailed block diagram of a MAC skip circuit 200 according to some embodiments is shown. The same reference numerals are used to refer to circuits related to the above. Figure 14 The MAC skip circuit 200 discussed uses the same components. It should be understood that pins with the same markings are coupled together (e.g., an output pin marked "x" will be coupled to an input pin marked "x"). Input vector 205 comprises a set of M N-bit vectors. The slashes in the arrow lines from input vector 205 to input flip-flops (FF) 210 indicate one line used to represent multiple lines. In some embodiments, there may be M lines leading to input FF 210, with one line used for each of the input vectors 205. In some embodiments, there may be N lines used for each of the M vectors, or N×M lines leading to input FF 210. In such embodiments, each bit of the M vectors can be processed in parallel. Input vector 205 can latch one bit at a time, or it can latch 8 bits at a time, for example, word-by-word.

[0060] Input FF 210 is a trigger circuit block used to latch input vector 205 into MAC skip circuit 200. The IN_LAT pin provides a latch signal input to input FF 210, which, when activated, causes input FF 210 to latch input vector 205 into the input FF 210 trigger. The RST pin is a reset signal input to the input FF 210 trigger to accommodate a general reset signal, which can be provided to various blocks, causing the state of MAC skip circuit 200 (including input FF 210 triggers) to return to its initial / reset state. In some embodiments, input FF 210 includes enough trigger states to accommodate each bit of each input vector 205, i.e., M×N trigger states. The triggers can be arranged as a series of registers, for example, an N-bit register for each of the M input vectors 205.

[0061] Weight FF 220 is a trigger circuit block used to latch weight vector 215 into MAC skip circuit 200. The W_LAT pin provides a latch signal input to weight FF 220, which, when activated, causes weight FF 220 to latch weight vector 215 into its trigger. The RST pin is a reset signal input to the weight FF 220 trigger to accommodate a general reset signal, which can be provided to various blocks, causing the state of MAC skip circuit 200 (including weight FF 220 triggers) to return to its initial / reset state. In some embodiments, weight FF 220 latches all weight vectors 215 and has enough trigger states to accommodate M K-bit weight vectors 215, i.e., M×K trigger states. In other embodiments, weight FF 220 latches only one bit from each weight vector 215 at a time, starting from the MSB, i.e., K trigger states. The output of weight FF 220 may include parallel outputs for each weight bit with the same positional value for each weight vector 215.

[0062] Multiplier 225 is a multiplier circuit block used to multiply each of the input vectors 205 latched in input FF 210 with each corresponding weight vector 215 latched in weight FF 220 in a bit-by-bit manner. In other words, only one bit from each weight vector 215 is multiplied with the corresponding input vector 205 at a time. Multiplier 225 also includes a Flow_Thru pin, which, when activated, causes multiplier 225 to pass the input vectors 205 through regardless of the bit values ​​from weight FF 220.

[0063] Figure 14The adder block 230 is decomposed into adder 230a and accumulator 230b. Adder 230a is an adder circuit block used to add together each of the bit-weighted input vectors 205. An adder tree circuit block is used as shown; however, other types of adders can be used. The adder strategy preserves the carry. The number of output bits of adder 230a is related to the number of bits (N) of each input vector and the number of input vectors (M). The adder will output N + Roundup (log₂M) bits. Therefore, for the example convolution of nine 8-bit input vectors, the adder will output 8 + 4 = 12 bits. The output pin of adder 230a is coupled to the input pin of accumulator 230b.

[0064] Accumulator 230b is essentially a 2x1 adder circuit block that adds the incoming sum product to the previously shifted sum product, which is fed back to another input pin of accumulator 230b. Accumulator 230b includes an ADD pin, which, when activated, instructs accumulator 230b to add the two inputs instead of subtracting them. The output of accumulator 230b is provided to the input pins of shift register 235 and skip module 245.

[0065] Shift register 235 includes multiple flip-flops arranged as a register with shift capability. Shift register 235 includes a SHIFT (shift) 255 input pin, which, when activated, causes shift register 235 to shift its contents left. Shift register 235 also includes an ACC_LAT pin that provides a latch signal input to latch the output of accumulator 230b into shift register 235. The RST pin is a reset signal input for shift register 235 to accommodate a general-purpose reset signal that can be provided to various blocks, causing the MAC skip circuit 200 (including shift register 235) to return to its initial / reset state.

[0066] Skip module 245 is a circuit block that determines whether a skip condition has occurred. Details of the skip module will be provided below. Figure 17 For a more detailed discussion, skip module 245 includes an input pin that receives the output of accumulator 230b and an output pin (SKIP 250) that can provide a skip (SKIP) signal to controller 260. The RST pin is the reset signal input of skip module 245 to accommodate a general reset signal that can be provided to various blocks, which can cause the state of MAC skip circuit 200 (including skip module 245) to return to the initial / reset state.

[0067] Controller 260 is a circuit block that includes a state machine and drives the necessary signals to control the above. Figure 15The interaction between the various circuit blocks. The controller 260 will be discussed in more detail below.

[0068] Figure 16 A block diagram of the controller 260 circuit block is shown. The controller 260 includes several sub-circuit blocks, including a finite state machine (FSM) 305 circuit block, a state logic 310 circuit block, a counter (CNT) 315 circuit block, a counter logic (CNT logic) 320 circuit block, a decoded state flip-flop (SFF) 325 circuit block, a logic 330 circuit block for control signals, and a jump (JMP) logic 335 circuit block. The controller 260 has pins for receiving SKIP 250 signal inputs, START signal inputs, NEXT signal inputs, and RST signal inputs, each received on a pin with the same name. The controller 260 has pins for providing control signals, including IN_LAT, W_LAT, ACC_LAT, ADD, SHIFT 255, SKIPFF_LAT, SKIPSR_LAT, SkipSHIFT, Flow_Thru, and OUT_RDY signals, each provided on a pin with the same name. These control signals are provided by logic 330 for control signals.

[0069] The SKIP 250 pin receives a signal from the skip module 245, which will be discussed in further detail below. The NEXT pin receives a signal to indicate whether the controller 260 should enter the next state in the state machine. The signal received by the NEXT pin can trigger the state machine to continue to the next step. The signal received by the NEXT pin can come from outside the system and assists in the control system. The START pin receives a signal to indicate whether the state machine should transition from the first state to the second state. Logic can combine the START pin signal with the NEXT switch so that when START = 1 and NEXT is triggered, the state machine enters the next state. The signal received by the START pin can come from outside the system and assists in the control system. The RST pin receives a signal to indicate whether the controller 260 should reset all latches and return to the initial state. The signal received by the RST pin can come from outside the system and assists in the control system.

[0070] The IN_LAT, W_LAT, ACC_LAT, ADD, SHIFT 255, SKIPFF_LAT, SKIPSR_LAT, SkipSHIFT, and Flow_Thru pins, along with their corresponding circuit blocks discussed above, have been discussed. The OUT_RDY pin provides a signal that, when activated, indicates that the output of the MAC skip circuit 200 is ready to be fetched or provided, for example, by circuitry implementing activation functions (such as ReLU activation functions).

[0071] The FSM 305 is a circuit block that determines the current state and the next state. The current state is output on the ST pin. In this embodiment, the ST pin may include three pins <0:2>, representing the states according to the following... Figure 24A and Figure 24B One of the eight possible states in the described state diagram. Based on the current state, the value at the START pin, the value at the RST pin, the value at the JMP0 pin, the value at the JMP1 pin, and the triggering of the value at the NEXT pin, the next state is generated and placed on the ST pin.

[0072] The ST pin is coupled to the corresponding pin in the State Logic 310 block. State Logic 310 uses the ST and NEXT pins to determine the decoded output placed on the hot-mode pin ST_d, which comprises eight pins. Hot-mode decoding converts each of the eight possible states into an output condition, where only one output pin is high at a time while the others remain low, effectively assigning one pin to each possible state. The NEXT pin, when activated, signals State Logic 310 to look for new inputs.

[0073] The Counter (CNT) 315 circuit block is a circuit block that generates a counter used to track the bit positions of the weight vector for bitwise processing of weight values. The CNT 315's CNT pins include <0:K'> pins, where K' equals Roundup(log2K). The value at the output of the CNT 315 block changes based on a change in the CNTplus pin. When the CNTplus pin is activated, the CNT pin changes, resulting in an output value equal to the previous output plus 1. The RST pin is the pin that resets the CNT315 when activated, ensuring the CNT pin value is consistently zero.

[0074] The CNT logic 320 circuit block is similar to the state logic 310 circuit block. The CNT logic 320 circuit block has pins coupled to pins with the same names as the CNT 315 circuit block and provides a hot-swappable decoded output on pin CNT_d. The number of pins in CNT_d is <0:K>, where K is the number of bits in weight vector 215 as described above. When activated, the NEXT pin signals the CNT logic 320 to look for new inputs on CNT.

[0075] The State Flip-Flop (SFF) 325 circuit block is a circuit block containing flip-flops that store each value present on the decoded state pin from the state logic 310 circuit block. For example, the SFF 325 could contain D-type flip-flops for decoding each of the state pins. Alternatively, other flip-flop types can be used. The ST_dlat pin can transmit the latch signal to logic 330 for control signals.

[0076] The control signal circuit block logic 330 is a circuit with pins for ST_dlat, CNT_d, NEXT, RST, CNTplus, and pins for IN_LAT, W_LAT, ACC_LAT, ADD, SHIFT 255, SkipFF_LAT, SkipR_LAT, SkipSHIFT, Flow_Thru, and OUT_RDY. The signals for these pins with the same names are generated using logic gates according to the state diagram and state table described below.

[0077] The JMP logic 335 block is a circuit block with pins coupled to the CNT or CNT_d pin, pins coupled to the ST or ST_d pin, pins coupled to the RST pin, pins coupled to the NEXT pin, and pins coupled to the SKIP 250 pin. The JMP logic 335 also has a JMP0 pin coupled to the JMP0 pin of the FSM 305 and a JMP4 pin coupled to the JMP4 pin of the FSM 305. The JMP logic 335 uses the logic to activate the JMP0 pin when either of the following two conditions is true: first, when RST = 0 AND (AND) NEXT trigger AND CNT = 1, or second, when SKIP = 1. When RST = 0 AND NEXT trigger AND CNT! = 7, the JMP logic 335 uses the logic to activate the JMP4 pin. Optionally, the activation of the JMP0 or JMP4 pin can be restricted to the condition that the current state is 7. This can be decoded from the ST or ST_d pin or from a specific ST_d pin (e.g., ST_d pin). <7> (Pin) Acquisition. It should be understood that the JMP logic 335 circuit block is only one way to implement this logic. This logic can also be integrated as an additional state into logic 330 or a finite state machine.

[0078] Figure 17 A block diagram of a skip module 245 circuit block according to some embodiments is shown. The skip module 245 circuit block can be configured according to... Figure 7 Mathematical formulas Figures 8 to 12 Example or Figure 13 The flowchart is implemented using other logic / circuit blocks. Skip module 245 includes several sub-circuit blocks, including an input and flip-flop (IS_FF) 350, a shift register 355, a subtractor 360, a comparator 365, an absolute value (Abs) 370 circuit block, and a skip flip-flop (SkipFF) 375 circuit block. Skip module 245 includes SkipSHIFT pin, SkipSR_LAT pin, SkipFF_LAT pin, RST pin, input and pin groups, an accumulation and product and pin groups, and a SKIP 250 pin. These pins provide or receive signals of the same name and are coupled to the corresponding pins of the MAC skip circuit 200 as described above.

[0079] The IS_FF 350 circuit block is a circuit block that can use flip-flops to store the input sum from the IS pin. The SkipFF_LAT pin can provide a signal to latch the input sum into the IS_FF 350. The RST pin can provide a signal to reset the IS_FF 350 to its initial state. The number of bits in the IS_FF 350 can be equal to the number of bits N in the input vector 205 plus the logarithm (base 2) of the number of input vectors M (RoundUP(log2M)). The output pins of the IS_FF 350 can be coupled to shift register 355 and to subtractor 360 as subtrahends.

[0080] The shift register 355 circuit block is a circuit block that utilizes a register capable of performing a left shift. The SkipSHIFT pin, when activated, triggers the left shift. The SkipSR_LAT pin, when activated, causes the input sum (IS) from IS_FF 350 to be latched into shift register 355. The RST pin can provide a signal to reset shift register 355 to its initial state. The number of bits in shift register 355 can be equal to the number of bits N in input vector 205 plus the number of bits K in weight vector 215 plus the logarithm (base 2) of the number of input vectors M (RoundUP(log₂M)). The output pin of shift register 355 can be coupled to subtractor 360 as the minuend.

[0081] Subtractor 360 takes the value of IS_FF 350 and subtracts it from the value of shift register 355 to calculate output B. Subtractor 360 provides output B to comparator 365 via its output pin.

[0082] The Abs 370 circuit block is configured to acquire the accumulated product sum (APS), calculate its absolute value, and provide it as output A to comparator 365. The Abs 370 circuit block includes corresponding pins for these functions. The Abs 370 may optionally be configured to acquire a threshold TH value via a set of TH pins, which can be added to the absolute value of APS as a threshold offset to increase or decrease the likelihood of detecting a skipped condition. In such an embodiment, the Abs 370 may include an adder to add or subtract the threshold TH from output A before providing output A to comparator 365. (See below for reference.) Figure 23A and Figure 23B The threshold TH will be discussed in more detail.

[0083] The comparator 365 circuit block is a circuit block that compares output A with output B. If output A is greater than output B, a skip condition is found. The comparator includes corresponding pins for these values. The skip condition is provided on the Skip_LAT pin, which is coupled to the SkipFF 375 circuit block. The SkipFF 375 circuit block is a circuit block that includes a flip-flop that latches the SKIP 250 output pin when activated by a signal on the Skip_LAT pin.

[0084] Figure 15 The operation of the MAC skip circuit 200 can be referenced. Figure 15 , Figure 16 and Figure 17 The circuit block diagram in the middle is through Figures 18 to 22 The example in the text will be used to demonstrate. Figure 18 The diagram above illustrates the information about... Figure 8 The same input vector and weight vector are used. The input vector is latched in input FF 210. The first MSB bit of the weight vector is latched in weight FF 220. Figure 18 In this process, regardless of the weight values, the Flow_Thru pin in multiplier 225 is activated, and the input vectors are allowed to flow through multiplier 225. This is equivalent to multiplying each input vector by "1". The input vectors are then added by adder 230a, and the input sum 0101 0001 0001 (1297) is provided to skip module 245. The input sum is latched into IS_FF 350. As mentioned above, the number of bits N in IS_FF 350 is... IS =K + Roundup(log2M). The highest N of the input sum. IS -K+1 bits are latched into shift register 355. This is equivalent to dividing the input sum by 2. N IS -K+1 In this case, it is 2 7 This is done because the product of the MSBs of the input vector and the weight vector is not yet fully weighted. Each cycle accumulates the shifted portion of the product by one position. Similarly, each cycle shifts the register by one position. Therefore, the highest N of the input and weight vectors is utilized. IS –K+1 bits ensure that comparisons are performed within the same multiplier.

[0085] For example, the product of the partial accumulation of MSBs of the weighted bits is equal to ∑IN·W[7], and then it is multiplied by its position value 2. 7 The remaining unprocessed weights can be adjusted to ∑IN·W[6]·2 6 +∑IN·W[5]·2 5 +∑IN·W[4]·2 4 +∑IN·W[3]·23 +∑IN·W[2]·2 2 +∑IN·W[1]·2 1 +∑IN·W[0]·2 0 In the worst case, each of the weight bits is 1. Therefore, the above equation can be rewritten as IS·2. 6 +IS·2 5 +IS·2 4 +IS·2 3 +IS·2 2 +IS·2 1 +IS·2 0 IS equals all cases where the weight bits are 1. Decomposing IS yields IS, which is equivalent to IS·127. Since the position values ​​of the partially accumulated product sum have not yet been shifted, IS·127 divided by 128 can be obtained by calculating IS-IS / 128, for example,

[0086]

[0087] Therefore, skipping the threshold STH can be achieved by summing the inputs to the highest N. IS -K+1 bits are taken into shift register 355 for calculation, as described above, which is equivalent to dividing the input sum by 128. Then shift register 355 can be subtracted from the input sum to obtain STH.

[0088] exist Figure 19 In the first cycle, the first weight bit is processed. Each input is multiplied by the corresponding one in the MSB of the weight vector. These products are then summed to obtain 0011 0010 0101 (-805). Since the first bit is the sign bit, this value is subtracted from the initial accumulated sum of products (i.e., 0) (ADD=0) to convert the accumulated sum of products to 2s complement format. The skip threshold (STH) is calculated by subtracting the shift register 1010 (10) as the subtrahend from IS_FF 0101 0001 0001 (1297) as the minuend. To determine whether the skip condition has occurred, STH0101 0000 0111 (1287) is compared with the absolute value of the accumulated sum of products 0011 0010 0101 (805). Since the condition A>B is not met, the next weight bit is processed in the subsequent cycle.

[0089] exist Figure 20In the middle, the accumulated product is shifted left and the second weight bit is processed. Each input is multiplied by the corresponding bit in the next bit of the weight vector. Then these products are added together to get 0010 1111 0101 (757). This value is added to the shifted accumulated product to get 1111 1111 1100 1010 1011 (-853). Thus the accumulated product is equal to ∑IN·W[7]·2 1 +∑IN·W[6]. Shift register 355 is also shifted and subtracted again from IS_FF 350 to find the skip threshold STH 0000 0000 0100 1111 1101(1277). This is done because the worst case is where IS·2 5 +IS·2 4 +IS·2 3 +IS·2 2 +IS·2 1 +IS·2 0 This is equal to IS·63. Shift register 355 is the previous IS divided by 128, but because the accumulated product sum has been shifted once, it is also shifted once, making it a division by 64 instead. The value of IS·63 / 64 can be obtained by calculating IS – IS / 64. Optionally, the highest N of the input sum... IS – Bit K can be latched into shift register 355, which is equivalent to dividing IS by 64. To determine if a skip condition has occurred, STH 0000 0000 01001111 1101 (1277) is compared with the absolute value of the accumulated product sum 0011 0101 0101 (853). Since condition A>B is not met, the next weighted bit is processed in a subsequent cycle.

[0090] exist Figure 21 In the middle, the accumulated product is shifted left and the second weight bit is processed. Each input is multiplied by the corresponding bit in the next bit of the weight vector. Then these products are added together to get 0010 0001 1010 (538). This value is added to the shifted accumulated product sum to get 1111 1111 1011 0111 0000 (-1168). Thus the accumulated product sum is equal to ∑IN·W[7]·2 2 +∑IN·W[6]·2 1 +∑IN·W[5]·2 0 Shift register 355 is also shifted and subtracted again from IS_FF 350 to find the skip threshold STH 0000 0000 0100 1110 1001 (1257). This is done because the worst case is where IS·2 4 +IS·2 3 +IS·22 +IS·2 1 +IS·2 0 This is equal to IS·31. The shift register 355 is the previous IS divided by 64, but because the accumulated product sum has been shifted once, the shift register is also shifted once, making it a division by 32 instead. The value IS·31 / 32 can be obtained by calculating IS-IS / 32. Optionally, the highest N of the input sum... IS The -K-1 bit can be latched into shift register 355, which is equivalent to dividing IS by 32. To determine if a skip condition has occurred, STH00000000 0100 1110 1001 (1257) is compared with the absolute value of the accumulated product sum 0100 1001 0000 (1168). Since the condition A>B is not met, the next weighted bit is processed in a subsequent cycle.

[0091] exist Figure 22 In the middle, the accumulated product is shifted left and the third weight bit is processed. Each input is multiplied by the corresponding bit in the next bit of the weight vector. Then these products are added together to get 0010 0010 0110 (550). This value is added to the shifted accumulated product sum to get 1111 1111 1001 0000 0110 (-1786). Thus the accumulated product sum is equal to ∑IN·W[7]·2 3 +∑IN·W[6]·2 2 +∑IN·W[5]·2 1 +∑IN·W[4]·2 0 Shift register 355 is also shifted and subtracted again from IS_FF 350 to find the skip threshold STH 0000 0000 0100 1100 0001 (1217). This is done because the worst case is where IS·2 3 +IS·2 2 +IS·2 1 +IS·2 0 This is equal to IS·15. The shift register 355 is the previous IS divided by 32, but because the accumulated product sum has been shifted once, the shift register is also shifted once, making it a division by 16 instead. The value IS·15 / 16 can be obtained by calculating IS – IS / 16. Optionally, the highest N of the input sum... IS–K–2 bits can be latched into shift register 355, which is equivalent to dividing IS by 16. To determine if a skip condition has occurred, STH 0000 0000 0100 1110 1001 (10000 0000 0100 1100 0001 (1217)) is compared with the absolute value of the accumulated product sum 0110 1111 1010 (1786). Since the condition A>B is satisfied, the remaining processing of the weight bits is stopped, and the output can be provided to the circuitry that implements the activation function (such as the ReLU activation function).

[0092] If the threshold is not met, the processing will continue in almost the same way for the next cycle. Each input will be multiplied by the next bit of the weight vector. These products are then summed together to form the output of adder 230A. This value is then added to the shifted sum of accumulated products to obtain ∑IN·W[7]·2 4 +∑IN·W[6]·2 3 +∑IN·W[5]·2 2 +∑IN·W[4]·2 1 +∑IN·W[3]·2 0 Shift register 355 is also shifted and subtracted again from IS_FF 350 to find the skip threshold. This is done because the worst case is where IS·2 2 +IS·2 1 +IS·2 0 This is equal to IS·7. The shift register is the previous IS divided by 16, but because the accumulated product sum has been shifted once, the shift register is also shifted once, making it a division by 8 instead. The value IS·7 / 8 can be obtained by calculating IS–IS / 8. Optionally, the highest N of the input sum... IS The –K–3 bits can be latched into shift register 355, which is equivalent to dividing IS by 8. To determine if a skip condition has occurred, STH is compared with the absolute value of the accumulated product sum.

[0093] If the threshold is not met, the processing will continue in the same way for the next cycle. Each input will be multiplied by the next bit of the weight vector. These products are then added together to form the output of adder 230A. This value is then added to the shifted sum of accumulated products to obtain ∑IN·W[7]·2 5 +∑IN·W[6]·2 4 +∑IN·W[5]·2 3 +∑IN·W[4]·2 2 +∑IN·W[3]·2 1 +∑IN·W[2]·2 0Shift register 355 is also shifted and subtracted again from IS_FF 350 to find the skip threshold. This is done because the worst case is where IS·2 1 +IS·2 0 This is equal to IS·3. The shift register is the previous IS divided by 8, but because the accumulated product sum has been shifted once, the shift register is also shifted once, making it a division by 4 instead. The value IS·3 / 4 can be obtained by calculating IS–IS / 4. Optionally, the highest N of the input sum IS The –K –4 bits can be latched into shift register 355, which is equivalent to dividing IS by 4. To determine if a skip condition has occurred, STH is compared with the absolute value of the accumulated product sum.

[0094] If the threshold is not met, the processing will continue in the same way for the next cycle. Each input will be multiplied by the next bit of the weight vector. These products are then added together to form the output of adder 230A. This value is then added to the shifted sum of accumulated products to obtain ∑IN·W[7]·2 6 +∑IN·W[6]·2 5 +∑IN·W[5]·2 4 +∑IN·W[4]·2 3 +∑IN·W[3]·2 2 +∑IN·W[2]·2 1 +∑IN·W[1]·2 0 Shift register 355 is also shifted and subtracted again from IS_FF 350 to find the skip threshold. This is done because the worst case is where IS·2 0 This is equal to IS. The shift register is the previous IS divided by 4, but because the accumulated product sum has been shifted once, the shift register is also shifted once, making it a division by 2 instead. The value IS·1 / 2 can be obtained by calculating IS-IS / 2. Optionally, the highest N of the input sum... IS The –K–5 bits can be latched into shift register 355, which is equivalent to dividing IS by 2. To determine if a skip condition has occurred, STH is compared with the absolute value of the accumulated product sum.

[0095] If the threshold is not met, the processing will continue in the same way for the next cycle. If the number of weight bits K is equal to 8, as shown in this example, this is the last cycle because there will no longer be unknown weight values. Each input will be multiplied by the next bit of the weight vector. These products are then added together to form the output of adder 230A. This value is then added to the shifted sum of accumulated products to obtain ∑IN·W[7]·2 7 +∑IN·W[6]·2 6 +∑IN·W[5]·25 +∑IN·W[4]·2 4 +∑IN·W[3]·2 3 +∑IN·W[2]·2 2 +∑IN·W[1]·2 1 +∑IN·W[0]·2 0 Since all weight bits have been processed, the output is provided to the circuitry that implements the activation function (such as the ReLU activation function).

[0096] refer to Figure 23A and Figure 23B The flowchart is shown, illustrating the following: Figure 17 The use of the threshold TH is illustrated. As mentioned above, a skip condition is detected when Abs (the sum of accumulated products) > STH. The threshold TH can be an added value that can be used to relax the conditions required for skipping. For example, as mentioned above, the calculation of STH is based on the assumption of a worst-case scenario for the weight vector. Since it is unlikely that all bits of the weight vector are "1", relaxing the requirement by adding the threshold TH can result in less computational energy without introducing an unacceptable amount of error (e.g., an error could occur if adding the threshold TH causes a value that should not be skipped to be skipped).

[0097] exist Figure 23A In some embodiments, the threshold TH can be a predefined value based on some known statistical properties and error tolerance of the weight vector. For example, if the known error tolerance allows for leniency in the worst case, the threshold TH can be equal to the portion of the input sum divided by itself. For example, the threshold can be equal to the input sum divided by 4, 8, or 16. In other embodiments, the threshold TH can be a predefined value based on the accumulated sum of products (APS). For example, the threshold TH can be equal to the portion of Abs(APS) added back to itself, such as Abs(APS) + Abs(APS) / 8. In yet another embodiment, the threshold TH can be a constant, such as the value 128, or some other value. At element 405, the threshold TH is calculated based on predetermined conditions, such as any of the conditions described above. In such embodiments, the threshold TH is either a constant or calculated based on one of the available numerical values ​​(such as the input sum or APS). At element 410, TH is added to Abs(APS) before entering comparator 365. In effect, comparator 365 determines whether A + TH > B? or A > B - TH?. It should be noted that although TH is shown to be added to Abs (APS), it can be subtracted from the minuend (IS_FF) or added to the subtrahend (from shift register 355) before entering subtractor 360.

[0098] exist Figure 23BIn some embodiments, the threshold TH can be a dynamic value calculated based on some external factors, or it can be a predetermined component as described above plus the dynamic component discussed herein. For example, if the computing device has low resources (battery, power supply, memory, processing availability, etc.), the threshold can be increased. At element 455, a condition is detected, which includes some external factors such as low computing resources, high temperature, faulty cooling conditions, movement, etc. At 460, TH is increased to reduce computing power. If TH increases, the likelihood of the skipped condition being triggered also increases. In some embodiments, TH can be based on a predetermined TH as described above, but then TH is adjusted to further reduce computing power. Increasing TH has the potential negative impact of increasing error. In other embodiments, TH can be equal to 0. At 465, it is detected that the condition has recovered or changed to allow more computing resources to be used. At 470, TH is decreased to increase computing power and reduce error.

[0099] exist Figure 24A , Figure 24B and Figure 25 The documentation provides information about the state. It should be understood that this information may vary depending on the operating parameters and characteristics of the actual components used. For example, shift registers may use combinational logic to cause the latch signal to interact with the shift signal in a specific way. As another example, in some hardware, the multiplier 225 and adder 230a circuit blocks can begin calculations immediately without separate states. Figure 24A A state diagram is provided, and Figure 24B The basic processing performed during each state is described. For simplicity, CNT in this specification has values ​​between 0 and 7. It should be understood that 7 here is the value K-1 when the number of bits in each weight vector K = 8. If K is a different number, the value "7" can be replaced appropriately.

[0100] At state 0, MAC is idle, skipping idle, and memory is free. State 0 is the ready state. It is also a return state when calculations associated with other states are completed. It is also a return state when the RST signal is asserted during any other state. Figure 25 As shown, in state 0, the OUT_RDY signal is high, and the accumulated product sum can be used to activate the function. If RST = 0 and NEXT is triggered and START = 1, then state 1 is entered.

[0101] At state 1, the memory reads input vector 205 and provides these values ​​on the pins of the MAC skip circuit 200. Input vector 205 is latched into input FF 210. Therefore, the output of INPUT_FF 210 immediately makes its output available to the multiplier 225 circuit block. The Flow_Thru pin is activated, allowing input vector 205 to flow through the multiplier 225 circuit block to adder 230a. Adder 230a adds input vector 205 together and produces an input sum (IS) to skip module 245. Figure 25 As shown, in state 1, the IN_LAT pin is activated, and Flow_Thru is activated. If RST = 0 and NEXT is triggered, then state 2 is entered.

[0102] At state 2, the input and IS are latched to IS_FF 350 of skip module 245. For example... Figure 25 As shown, in state 2, the Flow_Thru pin is activated and the SkipFF_LAT pin is activated. If RST = 0 and NEXT is triggered, then state 3 is entered.

[0103] At state 3, the memory reads the MSB of weight vector 215 latched into weight FF 220. In skip module 245, the highest N of IS... IS -K+1 bits are latched into shift register 355. For example... Figure 25 As shown, in state 3, the W_LAT pin is activated and the SkipSR_LAT pin is activated. If RST = 0 and NEXT is triggered, then state 4 is entered.

[0104] At state 4, input vector 205 is provided by input FF 210, and the current weight bit is provided by weight FF 220 to the multiplier 225 circuit block, which multiplies input vector 205 by the current weight bit and provides an output to adder 230a. The product sum is provided to accumulator 230b. In some embodiments, the product sum can be provided in a previous state (e.g., if the latch operation immediately provides the input to multiplier 225). If CNT = 0, the ADD signal is invalid and the product sum is subtracted, since the product sum should be negative as it represents the product sum of the sign bits of weight vector 215. If CNT = 0, the partially accumulated product sum is 0. If CNT ! = 0, the partially accumulated product sum has been shifted from the previous state 7. After the accumulator adds the partially accumulated product sum APS from shift register 235 to the current product sum, the output pin of accumulator 230b is ready. The counter is then incremented to indicate that the current bit has been processed. Figure 25 As shown, the CNT_plus pin is activated in state 4. If RST = 0 and NEXT is triggered, then state 5 is entered.

[0105] At state 5, the newly accumulated product and APS are latched into shift register 235 and provided to skip module 245, which compares abs(APS) with STH via comparator 365 circuit block. Figure 25 As shown, in state 5, the ACC_LAT pin is activated to indicate the latched accumulated product and APS. If RST = 0 and NEXT is triggered, then state 6 is entered.

[0106] At state 6, the result of the skipped comparison is latched into SkipFF 375. For example... Figure 25 As shown, at state 6, the SKIP_LAT pin is activated (unless CNT = 7, because all bits have been processed by then). If RST = 0 and NEXT is triggered, then state 7 is entered.

[0107] At state 7, if CNT! = 7, shift register 235 shifts left to prepare for the next weight bit processing, memory reads the next bit of the weight vector, and the next bit of the weight vector is locked in weight FF 220. If RST = 0 and NEXT is triggered and CNT! = 7, then state 4 is entered. This indicates that there are still other weight bits to process and no assertion to skip them. If RST = 0 and NEXT is triggered and CNT = 7, indicating that all weight bits have been processed, state 0 is entered. Regardless of the CNT value, if SKIP = 1, state 0 is also entered, aborting further processing of the remaining weight bits. Figure 25 As shown, at state 7, (if CNT != 7) the W_LAT and ACC_SHIFT pins are activated to indicate shift register 235, and the SkipSHIFT pin is activated.

[0108] Figure 26This is a system diagram of system 500 using a MAC skip circuit. Memory 505 is coupled to MAC skip circuit 200 to provide input data and weights to MAC skip circuit 200. Memory 505 can be any suitable memory device and medium for storing non-transitory computer-readable data. Controller 510 (e.g., convolution controller 510) is coupled to MAC skip circuit 200 and memory 505. Controller 510 provides control signals and logic to instruct MAC skip circuit 200 to operate, such as START and NEXT control signals. MAC skip circuit 200 also provides status signals to controller 510, such as the OUT_READY signal. Controller 510 also instructs memory 505 which inputs and weights to provide to MAC skip circuit. Activation function 515 takes the output of MAC skip circuit 200 and applies an activation function (such as a ReLU activation function) to the output. The result can then be stored back in memory 505 and provided to controller 510.

[0109] Figure 27 The figure illustrates the results of a series of sample tests. It shows the energy required to run the accumulated partial sum for each data point. The dashed ellipse encloses the accumulated partial sums processed without the skip module. As shown, when the accumulated partial sum is greater than zero, the energy required to calculate the accumulated partial sum using the skip module is greater than without it. However, when the accumulated partial sum is only moderately less than zero, the energy requirement with the skip module is significantly lower. In this example, the average energy saving using the skip module is approximately 12.2%. Energy savings of 10% to 40% can be achieved.

[0110] The computational complexity of the MAC processing described above increases with the large number M of input and weight vectors. It is known that the ReLU activation function stores the MAC output value only when the output value is positive and zero when the output value is negative. The embodiments provide a processing method, circuit, and system that can optionally skip the remaining processing of the input data to save computational power. The embodiments iteratively compute the product sum. After each iteration, a skip function block evaluates a portion of the accumulated product sum. If the partially accumulated product sum is negative such that the remaining operations cannot make the result positive, the remaining computation is skipped. Skipping the remaining computation provides reduced computation cycles and reduced memory calls.

[0111] One embodiment is a method for product and accumulation, comprising: performing partial summation on the product of an input vector and a weight vector in a word-by-word input and bit-by-bit weighted manner to obtain a partially accumulated product sum; comparing the partially accumulated product sum with a threshold condition; and asserting a skip indicator to indicate skipping the remaining computation of the accumulation when the partially accumulated product sum satisfies the threshold condition.

[0112] In one embodiment, the method further includes: skipping partial sum accumulation of the product of the input vector and at least one bit position of the weight vector. In one embodiment, the threshold condition is a dynamically computed threshold condition. In one embodiment, an assertion skip indicator is made when the partially accumulated product sum is negative and the remaining set of unknown weight bits would not make the partially accumulated product sum positive. In one embodiment, the remaining set of unknown weight bits is assumed to be "1". In one embodiment, the method further includes: performing an activation function on the partially accumulated product sum and recording the output of the activation function as equal to 0 or the larger of the partially accumulated product sum. In one embodiment, the method further includes: when the partially accumulated product sum is less than the threshold condition: performing a second partial sum accumulation on the product of the input vector and the remaining part of the weight vector in a word-input and bit-weighted manner to obtain a second partial accumulated product sum.

[0113] Another embodiment relates to a computing device including a skip circuit block, comprising a multiply-accumulate circuit block configured to multiply a group of M input vectors by a first weight bit of a group of M weight vectors to compute a first group of M products, sum the first group of M products together to compute a first product sum, and provide the first product sum on a product sum output pin group. The device also includes an accumulator circuit block comprising an accumulator input pin group coupled to the product sum output pin group, configured to accumulate a first value by receiving a first value on the accumulator input pin group and adding the first value to an accumulated value in the accumulator circuit block to replace an alternative accumulated value, and providing the accumulated value on an accumulated value output pin group. The device further includes a skip circuit block comprising an accumulated value input pin group coupled to the accumulated value output pin group, configured to compare the accumulated value with a threshold, and assert a skip signal on a skip pin when the threshold is met. The device also includes a control circuit block configured to check if there is a skip signal on the skip pin, and when the skip signal is not asserted: cause the multiplication-accumulation circuit block to process the next weight bit of the group of M weight vectors, and when the skip signal is asserted: terminate the processing of the remaining part of the group of M weight vectors, assert the output ready signal, and provide the accumulated value to the activation circuit.

[0114] In one embodiment, the skip circuit block includes a subtraction circuit block configured to subtract a shifted input sum from the input sum to calculate a threshold, the shifted input and the number of shifts being based on the weight bit position of the group of M weight vectors being processed, and to compare the threshold with an accumulated value. In one embodiment, the threshold is a first threshold, wherein the subtraction circuit block is further configured to modify the accumulated value by adding a second threshold to the accumulated value before comparing the first threshold with the accumulated value. In one embodiment, the second threshold is a constant value, a derived value based on the input sum or the accumulated value, or a dynamic value based on environmental variables, including power state, battery state, hardware state, or resource state. In one embodiment, the accumulator circuit block includes an ADD pin, wherein the accumulator circuit block is configured to subtract a first value from the accumulated value when controlled by the ADD pin, wherein the control circuit block is further configured to control the accumulator circuit block to subtract the first value via the ADD pin when the first weight bit corresponds to the most significant bit of the group of M weight vectors. In one embodiment, the device further includes: an input latch circuit block configured to store a group of M input vectors; a weight bit latch circuit block configured to store a group of M weight bits; and a multiply-accumulate circuit block including a multiplier block and an adder tree to compute a first product sum.

[0115] Another embodiment relates to a computational system including a skip module, comprising a multiply-accumulate device configured to compute the product of the input node with a weight vector in the input in a bit-weighted manner, summing each product to compute a product sum for each bit, weighting the product sums according to the bit value position of each bit, and accumulating the product sums to an accumulated product sum. The system also includes a skip module configured to compare the product sum with a threshold and assert a skip signal when the threshold is met. The system further includes control logic configured to cause the multiply-accumulate device to process the next bit of the weight vector when the skip signal is not asserted, and to stop processing the remaining portion of the weight vector when the skip signal is asserted. The system also includes an activation device that determines the convolution value of the input node based on the value of the accumulated product sum.

[0116] In one embodiment, the skip module is configured to evaluate the product sum after processing each weight bit. In one embodiment, the threshold of the skip module is calculated based on the input sum of all inputs of the input node. In one embodiment, the threshold of the skip module is calculated by subtracting a bit-shifted input sum from the input sum, the bit-shifted input sum being shifted by the number of bits corresponding to the bit position of the currently evaluated weight bit. In one embodiment, the threshold is a first threshold, and the skip module is further configured to add a second threshold to the absolute value of the product sum and compare it with the first threshold, and assert a skip signal if it is greater than the first threshold. In one embodiment, the second threshold is a predetermined threshold based on the current value of the product sum, the input sum of all inputs of the input node, or a constant value. In one embodiment, the second threshold is a dynamic threshold based on environmental factors, including one or more of power state, battery level, computing resource availability, or hardware failure.

[0117] The foregoing outlines features of several embodiments to enable those skilled in the art to better understand various aspects of this disclosure. Those skilled in the art will understand that they can readily use this disclosure as a basis for designing or modifying other processes and structures to achieve the same purposes and / or advantages of the embodiments described herein. Those skilled in the art will also recognize that such equivalent structures do not depart from the spirit and scope of the invention, and that various changes, substitutions, and modifications can be made within the invention without departing from its spirit and scope.

Claims

1. A method of processing data using skip logic, comprising: performing partial sum accumulation of a product of an input vector and a weight vector in a word-by-word input and bit-by-bit weight manner to obtain a partially accumulated product sum; comparing, by a skip circuit block, the partially accumulated product sum to a threshold condition; and when the partially accumulated product sum satisfies the threshold condition, asserting, by the skip circuit block, a skip indicator to indicate a skip of remaining computation of sum accumulation, wherein the skip indicator is asserted when the partially accumulated product sum is negative and a remaining set of unknown weight bits would not be able to make the partially accumulated product sum positive.

2. The method of claim 1, further comprising: skipping partial sum accumulation of a product of the input vector and at least one bit position of the weight vector.

3. The method of claim 1, wherein, the threshold condition is a dynamically computed threshold condition.

4. The method of claim 1, wherein, performing partial sum accumulation of a product of the input vector and the weight vector by a multiply-accumulate device to obtain the partially accumulated product sum.

5. The method of claim 1, wherein, the remaining set of unknown weight bits are each assumed to be one.

6. The method of claim 1, further comprising: performing an activation function on the partially accumulated product sum and recording an output of the activation function as equal to zero or a larger of the partially accumulated product sum.

7. The method of claim 1, further comprising, when the partially accumulated product sum is less than the threshold condition: performing a second partial sum accumulation of a product of the input vector and a remaining portion of the weight vector in a word-by-word input and bit-by-bit weight manner to obtain a second partially accumulated product sum.

8. A computing device comprising a skip circuit block, comprising: a multiply-accumulate circuit block configured to multiply a group of M input vectors by a first weight bit of a group of M weight vectors to compute a first group of M products, add the first group of M products together to compute a first product sum, and provide the first product sum on a group of product sum output pins; an accumulator circuit block comprising a group of accumulator input pins coupled to the group of product sum output pins, the accumulator circuit block configured to accumulate a first value by receiving the first value on the group of accumulator input pins, and adding the first value to an accumulated value of the accumulator circuit block to replace the accumulated value, and provide the accumulated value on a group of accumulated value output pins; a skip circuit block comprising a group of accumulated value input pins coupled to the group of accumulated value output pins, the skip circuit block configured to compare the accumulated value to a threshold, and when the threshold is satisfied, assert a skip signal on a skip pin, wherein the skip signal is asserted when the accumulated value is negative and a remaining set of unknown weight bits would not be able to make the accumulated value positive, and a skip circuit block comprising a group of accumulated value input pins coupled to the group of accumulated value output pins, the skip circuit block configured to compare the accumulated value to a threshold, and when the threshold is satisfied, assert a skip signal on a skip pin, wherein the skip signal is asserted when the accumulated value is negative and a remaining set of unknown weight bits would not be able to make the accumulated value positive, and a control circuit block configured to check whether the skip pin has the skip signal and, when the skip signal is not asserted: cause the multiply-accumulate circuit block to process a next weight bit of the group of M weight vectors and, when the skip signal is asserted: terminate processing of a remaining portion of the group of M weight vectors, assert an output ready signal and provide the accumulated value to an activation circuit.

9. The computing device of claim 8, wherein, The skip circuit block includes: a subtraction circuit block configured to subtract a shifted input sum from an input sum to compute the threshold, the shifted input sum being shifted a number of times based on a weight bit position of the group of M weight vectors being processed, and compare the threshold to the accumulated value.

10. The computing device of claim 9, wherein, The threshold is a first threshold, wherein the subtraction circuit block is further configured to change the accumulated value by adding a second threshold to the accumulated value before comparing the first threshold to the accumulated value.

11. The computing device of claim 10, wherein, The second threshold is a constant value, a derived value based on the input sum or the accumulated value, or a dynamic value based on an environmental variable including a power state, a battery state, a hardware state, or a resource state.

12. The computing device of claim 8, wherein, The accumulator circuit block includes an ADD pin, wherein the accumulator circuit block is configured to subtract the first value from the accumulated value when controlled by the ADD pin, wherein the control circuit block is further configured to control the accumulator circuit block to subtract the first value via the ADD pin when the first weight bit corresponds to a most significant bit of the group of M weight vectors.

13. The computing device of claim 8, further comprising: an input latch circuit block configured to store the group of M input vectors; a weight bit latch circuit block configured to store the group of M weight bits; and a multiply-accumulate circuit block including a multiplication block and an adder tree to compute the first product sum.

14. A computing system including a skip module, comprising: a multiply-accumulate device configured to compute products of input nodes with weight vectors in an input in a bit-wise weighted manner and sum each of the products to compute a product sum for each bit, the product sum being weighted according to a bit value position of each bit, and accumulate the product sums as an accumulated product sum; a skip module configured to compare the product sum to a threshold and assert a skip signal when the threshold is satisfied, wherein the skip signal is asserted when the product sum is negative and a remaining set of unknown weight bits will not be able to make the product sum positive; a control logic device configured to cause the multiply-accumulate device to process a next bit of the weight vectors when the skip signal is not asserted and cause the multiply-accumulate device to abort processing of a remaining portion of the weight vectors when the skip signal is asserted; and an activation device that determines a convolution value of the input nodes based on a value of the accumulated product sum.

15. The computing system of claim 14, wherein, The skip module is configured to evaluate the product sum after processing each weight bit.

16. The computing system of claim 14, wherein, The threshold of the skip module is computed based on an input sum of all inputs of the input nodes. The accumulator circuit block includes an ADD pin, wherein the accumulator circuit block is configured to subtract the first value from the accumulated value when controlled by the ADD pin, wherein the control circuit block is further configured to control the accumulator circuit block to subtract the first value via the ADD pin when the first weight bit corresponds to a most significant bit of the group of M weight vectors.

17. The computing system of claim 16, wherein, The threshold of the skip module is computed by subtracting a bit-shifted input sum from the input sum, the bit-shifted input sum being shifted by a number of bits corresponding to a bit position of a currently evaluated weight bit.

18. The computing system of claim 14, wherein, The threshold is a first threshold, the skip module being further configured to add a second threshold to an absolute value of the product sum and compare it to the first threshold, and if greater than the first threshold, assert the skip signal.

19. The computing system of claim 18, wherein, The second threshold is a predetermined threshold based on a current value of the product sum, an input sum of all inputs of the input node, or a constant value.

20. The computing system of claim 18, wherein, The second threshold is a dynamic threshold based on environmental factors including one or more of a power state, a battery level, a computational resource availability, or a hardware failure. The second threshold is a dynamic threshold based on environmental factors including one or more of a power state, a battery level, a computational resource availability, or a hardware failure.

Images