Computational circuit with layered accumulators

Abstract

Methods and systems related to computational circuits are disclosed herein. A computational circuit includes a math circuit, a first accumulator, and a second accumulator. The first accumulator includes a first memory. The second accumulator includes a second memory. The first accumulator is communicatively coupled to the math circuit and accumulates values ​​from the math circuit in the first memory. The second accumulator is communicatively coupled to the first memory and accumulates values ​​from the first memory in the second memory. The first memory is faster and smaller than the second memory.

Description

Technical Field

[0001] The present invention relates to the field of computers, and more particularly to methods and systems related to computer processing architectures and circuits. Background Art

[0002] Computational circuits, such as arithmetic logic units (ALUs), floating-point logic units (FPUs), and other related combinational digital circuits, are fundamental components of computing systems, performing arithmetic and logical operations on data to perform calculations. While such combinational digital circuits are discrete at the scale of the entire central processing unit (CPU), each circuit comprises multiple individual logic circuits to perform various operations on the data provided to the overall digital circuit. In addition to basic AND and OR blocks, these individual logic circuits can include more complex circuits, such as accumulators, which include memory to store the intermediate results of step-by-step functions. For example, a payroll program designed to calculate the total number of hours worked by a given employee might store a running total of hours in such memory and iteratively receive the number of hours worked each day and add it to a value in memory, replacing the value in memory with the output of the function. As used herein, the term "accumulator" refers to a logic circuit that, in addition to a memory to store values, also performs the related iterative read, write, and calculation operations.

[0003] One particularly important application of accumulators is for large matrix multiplications or convolutions. Matrix multiplication in certain applications, such as when implementing artificial neural networks (ANNs), requires a large number of sum-of-products computations. Because the data structures associated with standard ANNs can contain millions or even billions of elements, matrix multiplications performed on such data structures can also be computationally intensive. Computational circuits dedicated to performing large matrix multiplications can include an accumulator configured to store the matrix multiplication output, where the accumulator includes multiple sectors individually associated with respective portions of the matrix multiplication output. By passing input data to a set of multiplication circuits and accumulating their outputs into a sector of the accumulator, the sum of products for each portion of the matrix multiplication output can be calculated. While this can result in efficient execution of complex calculations, the size of the accumulator can be prohibitively expensive. Mathematical circuits can be designed to perform calculations so quickly that only register memory can keep pace with the circuits. However, register memory is relatively expensive compared to slower memory such as static random access memory or dynamic random access memory. Furthermore, the output of a large matrix multiplication or other complex calculation can be a huge matrix. Therefore, if sectors of the accumulator using register memory need to be dedicated to each part of such a large matrix, then the accumulator can be very expensive. Summary of the Invention

[0004] Methods and systems are disclosed in the fields of computer processing architecture and circuits. The disclosed systems include computational circuits having hierarchical accumulators. The disclosed systems include hierarchical accumulators having two or more levels, each level including an accumulator; wherein each level includes an accumulator of increased size, reduced cost per bit, and reduced speed relative to the accumulator of the level above it in the level. These disclosed hierarchical accumulators result in fast, low-cost accumulators with enhanced performance in complex calculations.

[0005] In a specific embodiment of the present invention, at least a first level of a hierarchical accumulator is divided into different sectors, wherein at least one sector remains idle while another sector is used to accumulate output values, and a subsequent level of the accumulator is configured to read from the idle sector while the first level participates in accumulating output values ​​in the active sectors of the first level. In a specific embodiment, at least the first level consists of two sectors, one of which is idle and the other is active, or vice versa. In a specific embodiment, each subsequent level of the accumulator can read all values ​​in the idle sectors of the previous level before the idle sectors of the previous level are activated again.

[0006] In a specific embodiment of the present invention, each level of the accumulator is configured so that each level accumulates at a rate of data elements per second that is at least the same as the previous level. In a specific embodiment of the present invention, the speed of the accumulator is set by the speed of the accumulation operation of that accumulator level (expressed in writes per second) divided by the number of accumulation operations required to calculate the data elements (expressed in writes per data element). Therefore, the speed at which each level generates a final value in memory is the speed of the accumulation operation divided by the number of accumulation operations required to calculate the data elements. Therefore, even though the accumulators at a higher level may have a higher accumulation operation speed, if the ratio of the relative number of accumulation operations required to calculate data elements between the higher level and the next level is set to be equal to the ratio of the relative accumulation operation speeds between the next level and the higher level, the hierarchical accumulator can continue to generate values ​​at the fastest speed that the computational circuitry can execute without creating any bottlenecks in the various levels of the hierarchy.

[0007] In specific embodiments of the present invention, hierarchical accumulators are used as part of a computational unit in a network of compute nodes. In these embodiments, the use of hierarchical accumulators can alleviate network pressure by increasing data reuse of local data before additional data is required from the network, while eliminating the need for large and expensive fast accumulators. In specific embodiments of the present invention, a computation circuit including the disclosed hierarchical accumulator operates on operands retrieved from local memory (i.e., local data located on the same substrate as the controller and computation circuitry) and remote memory (i.e., remote data accessible to the controller and / or computation circuitry over the network). Operands can be routed from memory to the network in blocks of a given size, and the hierarchical accumulator can be configured to include a memory capable of storing blocks of a given size. Advantageously, this block size can be large, which increases the number of computations that can be performed on a given block before another block must be passed across the network. This benefit exceeds linear improvement, as both the local and remote data blocks are larger, resulting in a significant increase in the number of operations that can be performed with a single remote data block.

[0008] In a specific embodiment of the present invention, a computation circuit is provided. The computation circuit includes a math circuit. The computation circuit also includes a first accumulator, the first accumulator being communicatively connected to the math circuit, the first accumulator having a first memory, and the first accumulator accumulating values ​​from the math circuit in the first memory. The computation circuit also includes a second accumulator, the second accumulator being communicatively connected to the first memory, the second accumulator having a second memory, and the second accumulator accumulating values ​​from the first memory in the second memory. The first memory is faster and smaller than the second memory.

[0009] In a specific embodiment of the present invention, a method is provided. The method includes accumulating values ​​from a math circuit in the first memory using a first accumulator having a first memory. The method also includes accumulating values ​​from the first memory in the second memory using a second accumulator having a second memory. The first memory is faster and smaller than the second memory.

[0010] In a specific embodiment of the present invention, a computing circuit is provided. The computing circuit includes a matrix multiplier array, a register memory, and a static random access memory. The computing circuit also includes a first accumulator that accumulates values ​​from the matrix multiplier array in the register memory. The computing circuit also includes a second accumulator that accumulates values ​​from the register memory in the static random access memory. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 A block diagram of a computing circuit with a hierarchical accumulator and a set of flow charts of methods according to specific embodiments of the present invention disclosed herein are shown.

[0012] Figure 2 Shown Figure 1 A block diagram of a hierarchical accumulator in FIG. 1 is annotated to illustrate the interoperation between the hierarchical accumulator levels according to a specific embodiment of the invention disclosed herein.

[0013] Figure 3 A block diagram of a matrix multiplication circuit with a layered accumulator is shown according to a specific embodiment of the invention disclosed herein.

[0014] Figure 4 A block diagram of a system is shown of a matrix multiplication circuit having an accumulator according to a specific embodiment of the invention disclosed herein, wherein the matrix multiplication circuit receives a set of data values ​​from a remote memory over a network and another set of data values ​​from a local memory.

[0015] Figure 5 A flow chart is shown of a set of methods for operating a layered accumulator according to specific embodiments of the invention disclosed herein. DETAILED DESCRIPTION

[0016] Disclosed herein in detail are methods and systems related to the field of computing circuits in accordance with the above-described invention. The methods and systems disclosed in this section are non-limiting embodiments of the invention and are intended for explanatory purposes only and should not be used to limit the full scope of the invention. It should be understood that the disclosed embodiments may or may not overlap with each other. Therefore, a portion of one embodiment or a specific embodiment thereof may or may not fall within the scope of another or a specific embodiment thereof, and vice versa. Different embodiments from different aspects may be combined or implemented separately. Many different combinations and sub-combinations of the representative embodiments shown within the broad framework of the invention may be obvious to those skilled in the art, but not explicitly shown or described should not be construed as being excluded.

[0017] Figure 1A block diagram and a flowchart 150 of a computation circuit 100 with a hierarchical accumulator 110 are shown, along with a set of methods, according to specific embodiments of the present invention disclosed herein. Computation circuit 100 includes math circuit 101 and hierarchical accumulator 110. The math circuit can be a multiplication array capable of accommodating multiple operands, multiplying them in pairs, and outputting paired products. The math circuit can operate on its input operands and produce an output in a single clock cycle, or generate a serial output in a set of clock cycles. The output can be a set of products required to compute all elements of an output matrix. Computation circuit 100 also includes two input source registers in the form of source A register 120 and source B register 121. Computation circuit 100 can accumulate values ​​for a composite computation in a pipelined manner, providing multiple pairs of input values ​​to source A register 120 and source B register 121 to perform multi-component computations of the composite computation using math circuit 101, and accumulating the outputs of the composite computations using hierarchical accumulator 110.

[0018] Computation circuit 100 performs complex computations on multiple pairs of input values ​​provided to source A register 120 and source B register 121. While computation circuit 100 accepts a single pair of inputs, alternative computation circuits according to the present invention may accept more than two inputs. By orchestrating the execution of complex computations, inputs may be provided to the input source registers in a pipelined manner. The values ​​in the source registers are provided to math circuit 101 as operands for computation performed by math circuit 101. The input values ​​stored in source A register 120 and source B register 121 may be scalar data elements of a data type compatible with math circuit 101 (e.g., 8-bit integer, 16-bit floating point, etc.). The input values ​​stored in source A register 120 and source B register 121 may also be vectors or multidimensional tensors containing a set of data elements of the same or different data types (e.g., each input may be a 16x16 tensor of a single data element, each data element being a 16-bit floating point data type). Accordingly, math circuit 101 may include an array of discrete computation circuits 102 to operate in parallel on the various data elements of the math circuit inputs. For example, computation circuit 100 may be a matrix multiplication circuit, and math circuit 101 may have an array of discrete multiplication circuits, each of which extracts two values ​​from an input data value (e.g., extracts two 16-bit floating point data element values ​​from two 16x16 tensor input data values) and outputs the product of these values.

[0019] The control system responsible for feeding values ​​to math circuit 101 can be designed to provide new values ​​to source A register 120 and source B register 121 in synchronization with the speed of math circuit 101 to perform complex calculations (such as large matrix multiplications) while keeping math circuit 101 at full capacity. Math circuit 101 can perform all the component calculations required for a set of inputs in a single clock cycle. For example, if math circuit 101 includes 256 discrete multiplication circuits, each of which calculates in a single clock cycle, then math circuit 101 can perform all the component multiplications required for the complex calculation in the form of matrix multiplications of two 16x16 tensors in a single clock cycle. However, if the math circuit does not have too many discrete multiplication circuits, or the discrete multiplication circuits do not calculate at the speed of one clock cycle, more than one clock cycle will be required to perform all the component multiplications.

[0020] The hierarchical accumulator 110 accumulates the output values ​​from the math circuit 101. The hierarchical accumulator 110 includes a first accumulator 111 and a second accumulator 112. The hierarchical accumulator 110 includes two levels, each of which includes an accumulator having a memory for storing accumulated values, and logic circuitry for reading input values, adding the input values ​​to the accumulated values ​​stored in the memory, and storing the output of the addition in the memory. The logic circuitry may include various logic gates, latches, flip-flops, and other digital logic or analog logic components. Although the hierarchical accumulator 110 includes two levels for ease of explanation, the hierarchical accumulator according to the present invention may have more than two levels.

[0021] A first accumulator 111 is communicatively coupled to math circuit 101. The first accumulator 111 includes a first memory 113, and the first accumulator 111 accumulates values ​​from math circuit 101 in the first memory 113. Thus, the first accumulator 111 can read a value from memory 113, retrieve a value from math circuit 101, perform an addition operation on the two values, and write the result of the addition operation back to memory 113. This operation, illustrated in step 151 of flowchart 150, involves using the first accumulator 111 with the first memory 113 to accumulate values ​​from math circuit 101 in the first memory 113. The speed at which these operations are performed can be referred to as the accumulation operation speed of the accumulator 111 and can be expressed in units of data elements written per second. The first memory 113 can include a first memory sector 115 and a second memory sector 116. Each memory sector can contain multiple addresses for storing data elements. The data elements can be of various data types. However, the data type generally matches the output format of math circuit 101. For example, first accumulator 111 may retrieve a value from memory sector 115, obtain a value from math circuit 101, sum the two values, and then write the sum back to memory sector 115. First memory 113 may be register memory, but it may also be any type of memory, including static random access memory, dynamic random access memory, stripe memory, phase change memory, or any type of readable and rewritable memory.

[0022] Second accumulator 112 is communicatively connected to first memory 113. Second accumulator 112 includes second memory 114, and second accumulator 112 accumulates values ​​from second memory 113 in second memory 114. Therefore, second accumulator 112 can read a value from memory 114, retrieve a value from memory 113, perform an addition operation on these two values, and write the result of the addition operation back to memory 114. This operation, illustrated in step 152 of flowchart 150, involves accumulating the values ​​from first memory 113 in second memory 114 using second accumulator 112 and second memory 114. The speed at which these operations are performed can be referred to as the accumulation operation speed of accumulator 112 and can be expressed in units of data elements written per second. Second memory 114 can include a first memory sector 117 and a set of additional memory sectors 118. Each memory sector can contain multiple addresses for storing data elements. The data elements can be of various data types. However, the data type generally matches the format of the data elements in memory 113. Second memory 112 can be static random access memory. However, it may also be a register memory, a dynamic random access memory, a stripe memory, a phase change memory or any type of readable and rewritable memory.

[0023] In certain embodiments of the present invention, each level of the hierarchical accumulator is slower than the previous level but has more memory than the previous level. For example, if computation circuit 100 is a matrix multiplier and math circuit 101 is a multiplier array, memory 113 can be as large as the output matrix generated by math circuit 101 in response to all products generated by a pair of operand inputs, and memory 114 can be as large as the output matrix generated from a set of operand inputs (e.g., set 130) and provided to the source register. Furthermore, each level of the hierarchical accumulator may have a slower accumulation operation rate per data element written per second than the previous level due to fewer logic circuits, slower logic circuits, slower memory, or both. For example, math circuit 101 may output a value at an output rate of 256 data elements per nanosecond, and first accumulator 111 may accumulate the value from math circuit 101 in first memory 113 at a first accumulation operation rate. The value may be a data element, and the first accumulation operation rate may be a data element accumulation operation rate. Continuing with this example, the first accumulation operation speed can be at least as fast as the output speed of 256 data elements per nanosecond. The second accumulator 112 can accumulate values ​​from the first memory 113 in the second memory 114 at a second accumulation operation speed. The values ​​can be data elements, and the second accumulation operation speed can be a data element accumulation operation speed. To remain consistent with the above example, the second accumulation operation speed can be slower than the first accumulation operation speed (i.e., 256 data elements per nanosecond), thereby allowing the use of a relatively slower memory because the second accumulator does not need to accumulate values ​​every nanosecond.

[0024] The speed differences between different levels of the hierarchy can be caused by a variety of factors. For example, the logic circuitry of an accumulator at one level of the hierarchy may be slower than that of the accumulator at the previous level to perform the summation required by that level of the hierarchy. In an alternative or combination, each level of the hierarchical accumulator can utilize memory that is slower to read and write than the previous level. Since there is generally an inverse relationship between the speed and size / cost of different types of memory, this configuration allows each level of the hierarchy to be slower than the previous level, but larger / cheaper. Applying this configuration to Figure 1 , the first memory 113 will be faster and smaller than the second memory 114. For example, the first memory 113 may be a smaller 4096-bit register memory and the second memory 114 may be a larger 131072-bit static random access memory.

[0025] Figure 2 Shows a specific embodiment of the invention disclosed herein Figure 12 is a block diagram 200 of a hierarchical accumulator 110 annotated to illustrate the interoperability of the levels of the hierarchical accumulator. As shown, the accumulator 111 includes a memory divided into a memory sector 115 and a memory sector 116. The accumulator 111 receives an upstream output 201 (which may be the output of the math circuit 101) and accumulates it into the memory 113. As shown, the output generated by the math circuit (e.g., output 201) can have an output size, and the sectors of the first memory (e.g., memory sector 115 and memory sector 116) can be at least as large as the output size. Accordingly, each sector of the memory can hold the entire output of the math circuit for a given input. In a specific embodiment of the present invention, as Figure 2 As shown, the first memory includes a first sector and a second sector, both of which have sizes equal to the output size.

[0026] In a specific embodiment of the present invention, the memory of each level of the hierarchy is divided into at least two sectors, which are in an idle or active state relative to one level and in an alternate state relative to an adjacent level. For example, in a hierarchical accumulator having two levels, the first level has a first accumulator with a first memory, and the second level has a second accumulator with a second memory. The first memory can be divided into a first sector and a second sector having this characteristic. Consistent with this example, the first accumulator can accumulate in the first sector of the first memory while the second accumulator accumulates from the second sector of the first memory, and the first accumulator can accumulate in the second sector of the first memory while the second accumulator accumulates from the first sector of the first memory.

[0027] like Figure 2 As shown, the staged accumulator 110 can switch the accumulator 111 between utilizing the memory sector 116 and the memory sector 115 as additional upstream outputs (e.g., upstream output 201) are received. The sector in which the accumulator 111 is currently accumulating values, e.g. Figure 2 The memory sectors 116 in the memory may be referred to as active sectors, and any sectors in the accumulator 11 that are not currently accumulating values, such as Figure 2 The memory sectors 115 in the tier may be referred to as idle sectors. The currently active sectors may change with each upstream output received by the tier, or may change less frequently. Figure 1 In the example above, assuming the inputs are matrices of equal size being multiplied, the active sector changes each time a new set of inputs is provided to the source registers.

[0028] In a specific embodiment of the present invention, the lower levels of the accumulator are configured to read from the idle sectors of the higher levels of the accumulator, while the higher levels participate in quickly accumulating the output values ​​in the active sectors of the higher levels so that all values ​​are accumulated from the idle sectors before the sector becomes active again relative to the higher level. For example, as shown in the figure, before accumulator 111 writes to memory sector 115, accumulator 112 can read and accumulate all values ​​in memory sector 115 of memory 113 into memory sector 117 of accumulator 112. Although accumulator 111 has only two memory sectors, accumulator 112 can read and accumulate idle sectors in the same manner before additional upstream outputs need to be accumulated, thereby utilizing more sectors. Each level of the hierarchy can match this feature.

[0029] In a specific embodiment of the present invention, the various levels of the hierarchical accumulator are configured such that each level accumulates at a bit-per-second rate that is at least as fast as the previous level, regardless of whether the accumulation operation of lower levels of the accumulator is slower than that of higher levels. As previously described, the accumulation operation of each accumulator may be slower than that of the previous level. However, the overall operation of the hierarchical accumulator can be such that, while each level accumulates slower than the next higher level, each level can be as fast or faster than the next higher level in terms of bits accumulated per second. For example, in response to a set of operands applied to a math circuit, a first accumulator can perform several accumulation operations to store output data values ​​in a first memory, and in response to a set of operands applied to the math circuit, a second accumulator can perform several accumulation operations to store output data values ​​in a second memory, and the number of accumulation operations performed by the first accumulator can be greater than the number of accumulation operations performed by the second accumulator. For example, to store a single bit of output in first memory 113, a set of operands 130 can be provided to math circuit 101. This requires math circuit 101 to perform eight operations using eight different sets of operand inputs. For each set of operand inputs, accumulator 111 must perform at least one accumulation operation. However, to store a single bit of the output in memory 114, only a single accumulation may be required. Therefore, as long as the accumulation speed of accumulator 111 is less than eight times that of accumulator 112, accumulator 112 can maintain its required speed in the pipeline.

[0030] exist Figure 2In the example, the speed of the accumulator 111 is set by the speed of the accumulation operation of the accumulator 111 (expressed in the number of writes per second) divided by the number of accumulation operations "A" required to calculate the bit (expressed in the number of writes per bit) (i.e., the number of accumulated bits per second = X / A). Continuing with this example, the speed of the accumulator 112 is set by the speed of the accumulation operation of the accumulator 122 (expressed in the number of writes per second) divided by the number of accumulation operations "B" required to calculate the bit (expressed in the number of writes per bit) (i.e., the number of accumulated bits per second = Y / B). In this example, X can be greater than Y, and the ratio of Y / B can be greater than or equal to X / A. Although Figure 2 The hierarchical accumulators for provide a two-level hierarchy, but multiple levels of hierarchy can be used as long as the writes per second and writes per second ratio remain constant across multiple levels. Again, refer to Figure 2 , but for the specific number of variables in the previous paragraph, each bit in accumulator 111 may require 16 upstream outputs 201 to be accumulated to calculate the bit, while each bit in accumulator 112 may require a single bit in memory 113 to calculate the bit. Figure 2 In this case, memory 113 may be a register memory, and the value of X may be written once per nanosecond. In this example, the upstream outputs may be in the form of 16 8-bit integers, all of which can be written in a single cycle of a 1 GHz clock. Continuing with this example, the computation circuitry may require 16 upstream outputs 201 to accumulate 1 portion of memory sector 116, where the portion of memory sector 116 is the same size as a single upstream output. Therefore, the first level of the hierarchy will accumulate at a rate of 1 write / ns divided by 16 writes / value, providing a rate of 1 / 16 value / ns. Even if memory 114 is a slower memory (e.g., static random access memory), the second level accumulator may have a faster accumulation rate, such that the accumulator requires 16 nanoseconds to accumulate. If the computation circuitry only requires 1 value from memory 113, the second level of the hierarchy will accumulate at a rate of 1 / 16 write / ns divided by 1 write / value, providing a rate of 1 / 16 value / ns. Accordingly, the two levels of accumulators will be synchronized with the pipeline providing the upstream output, and the capacity of the two levels will be maximized. In a specific embodiment of the present invention, the operating speed (in bits per second) of the accumulators at different levels can also be different based on the number of logic circuits in each level, rather than the difference in memory speed, where the logic circuits operate on a block of data in a step-by-step manner in a set of clock cycles (e.g., 16 logic circuits accumulate 32 values ​​in two clock cycles).

[0031] Figure 3A block diagram of a matrix multiplication circuit 300 according to a specific embodiment of the present invention disclosed herein is shown, having a hierarchical accumulator 310. Hierarchical accumulator 310 includes a first accumulator 311, which may exhibit the characteristics of first accumulator 111 described above. Hierarchical accumulator 310 also includes a second accumulator 312, which may exhibit the characteristics of second accumulator 112 described above. First accumulator 311 includes register memory 313. Second accumulator 312 includes static random access memory 314. Matrix multiplication circuit 300 is designed to multiply a first input matrix 303 by a second input matrix 304 to produce an output matrix. The output matrix may be stored in static random access memory 314. Matrix multiplication array 301 includes an array of discrete multiplication computation units 302, which accept two operands, multiply them, and output the product. The overall matrix multiplication of the first input matrix 303 and the second input matrix 304 may include a multiplication array 301 that performs all product operations and a first accumulator 311 and a second accumulator 321 that performs all sum operations of component product and sum calculations required for the matrix multiplication.

[0032] according to Figure 3 In a specific embodiment of the present invention, the output matrix of the matrix multiplication circuit will include portions that uniquely correspond to memory sectors of the layered accumulator. For example, the sectors of memory 314 will have a one-to-one correspondence with portions of the output matrix produced by multiplying the first input matrix 303 and the second input matrix 304. Figure 3 The matrix is ​​divided into separate parts, such as part 303. These separate parts are applied to source A register 320 and source B register 321 in a pipelined manner to perform the matrix multiplication. The individual parts of the matrix can be scalar data elements of a data type compatible with the matrix multiplication array circuit 301 (e.g., 8-bit integer, 16-bit floating point, etc.). These parts can also be vectors or multi-dimensional tensors that have a set of data elements of the same or different data types (e.g., each input can be a 16x16 tensor with each data element in a 16-bit floating point data type). Accordingly, the multiplication array 301 can include an array of discrete computation circuits 302 to operate on the various data elements of the input to the mathematical circuit in parallel.

[0033] exist Figure 3In the example of , each part of the input matrix is ​​an array of 16x16 data elements. Therefore, the sectors of the static random access memory 312 and the register memory 313 (e.g., sector 315) are memories that can store 16x16 data elements. The multiplier array 301 can perform matrix multiplication on two input data values ​​in the form of two 16x16 data element arrays within 4 clock cycles, outputting a new value in each clock cycle. Therefore, the accumulator 311 needs to have an accumulation operation time as fast as a single clock cycle. However, calculating the output part of the output matrix in the memory sector (e.g., sector 316) of the register 313 requires multiple writes. This is because each 16x16 matrix in the column 305 of the input matrix 304 must be multiplied with each 16x16 matrix in the row 306 of the input matrix 303 to calculate the output matrix portion corresponding to the sector 315 of the static random access memory 314. The first accumulator 311 accumulates the values ​​from all eight matrix multiplications performed by the matrix multiplier array 301 in the register memory 313. Accordingly, the memory sector 316 needs to remain active so that the accumulator 311 accumulates the values ​​for 8x4 clock cycles (8 matrix multiplications multiplied by 4 clock cycles / matrix multiplication). This allows the second accumulator 312 to accumulate the values ​​from the register memory 313 in sector 316 to the static random access memory 314 in sector 315 in 32 clock cycles. Therefore, the accumulation operation speed of the accumulator 312 is 32 clock cycles per 16x16 data elements to keep pace with the higher-level accumulators. If the clock cycle is 1ns, then these 32ns provide enough time to read and write to the static random access memory.

[0034] If the matrix multiplication circuit 300 is designed only for matrix inputs of the size of the first input matrix 303 and the second input matrix 304, the accumulator 312 can be replaced by a static random access memory and a circuit that can read from the register 313 and write to the static random access memory 314 separately (i.e., it does not require a summing circuit). However, the matrix multiplication circuit 300 can be used to perform large matrix multiplication operations, where Figure 3 The 128x128 matrices shown are just blocks of larger matrices. In a specific embodiment, the internal dimensions of these larger matrices are equal to the memory size of the hierarchical accumulators. Figure 3In the example of , this would be a matrix of 128x128 data elements so that the correspondence between the various portions of the output matrix and the sectors of memory 314 can be preserved (i.e., memory 314 is large enough to store the entire output matrix). In these embodiments, the various blocks can be provided to the matrix multiplication circuit 300 in the same manner as the portions of the illustrated matrix are provided to the source registers in a super-cycle fashion. Thus, the accumulations that occur in accumulator 312 will be used to calculate the final value of the output matrix, with sectors (such as sector 315) being accumulated once per super-cycle. In these embodiments, the layered accumulator can have another layer that is slower than accumulator 312 (e.g., the memory can be dynamic random access memory or EEPROM), and the size of memory 314 can be reduced in proportion to the speed at which the next layer can be written.

[0035] In a specific embodiment of the present invention, the matrix blocks are routed as a unit from the main memory to the processing pipeline of the processor. In a specific embodiment of the present invention, the computational circuit disclosed herein is located in the pipeline of the processor in the processing core network, and the blocks are routed through the network as a unit. These blocks can be routed by a controller. The controller can also orchestrate the pipeline on the computational circuit. For example, if the computational circuit is a matrix multiplication circuit, the controller can be programmed to use the matrix multiplication circuit to multiply a first matrix and a second matrix to generate an output matrix, provide the first matrix to the multiplier array in the first series of blocks, and provide the second matrix to the matrix multiplication circuit in the second series of blocks. In these embodiments, the second memory of the hierarchical accumulator (e.g., memory 313) can be as large as the output matrix.

[0036] Regardless of the type of embodiment involved, routing large blocks of data through a processor or processing core network can take up valuable hardware and power resources. Therefore, performing matrix multiplication in a manner that maximizes data reuse, and the time interval between additional data blocks required by the slowest link in the pipeline, can create significant benefits. Therefore, the controller can be programmed to retrieve a block from the slowest link only once (i.e., retrieve a block from the slowest link and perform all calculations involved in that block before retrieving another block). In the case of matrix multiplication, this would require retrieving alternative blocks (i.e., blocks obtained from faster links) from memory multiple times.

[0037] Figure 44 is a block diagram of a system 400 having a matrix multiplication circuit 401 with an accumulator 402, wherein the matrix multiplication circuit 401 receives a set of data values ​​from a remote memory 403 and another set of data values ​​from a local memory 405 via a network 404. The data values ​​from the remote memory 403 may be blocks of a first matrix. The data values ​​from the local memory 405 may be blocks of a second matrix. The internal dimensions of the first and second matrices set the dimensions of an output matrix 410. The local memory 405 may be the main memory of a processor. The remote memory 403 may be the main memory of another processor, where the network 404 connects the two processors so that they can operate as multiple cores in a multi-core processor. The accumulator may be a hierarchical accumulator, such as hierarchical accumulator 312. The accumulator includes memory for storing the output matrix 410. The matrix multiplication circuit 401 operates on a first input matrix block 412 retrieved from a set of first input matrix blocks 413 via the network 404 and a second input matrix block 411 retrieved from a set of second input matrix blocks 414 from the local memory 405. Each time a new block is retrieved from remote memory 403 or local memory 405, Figure 3 The super loop mentioned in the preceding text can be orchestrated by the controller 420, which is programmed to use the matrix multiplication circuit 401 to multiply a first matrix (e.g., composed of a group of first input matrix blocks 413) and a second matrix (e.g., composed of a group of second input matrix blocks 414), provide the first matrix (e.g., a series formed by a group of first input matrix blocks 413) to the matrix multiplication circuit 401 in the first series of blocks, and provide the second matrix (e.g., a series formed by a group of second input matrix blocks 414) to the matrix multiplication circuit 401 in the second series of blocks.

[0038] In certain embodiments of the present invention, the layered accumulators and block sizes used by system 400 may be selected to minimize the stress placed on network 404. Figure 4 In the example, the system can be designed to loop through all the values ​​in the local memory 405 while performing a super loop using a single input matrix block from the set of first input matrix blocks 413. Thus, in the case shown, it is assumed that the super loop has a reference Figure 3Given the features of the specific example provided, and each matrix is ​​divided into four blocks, the network will have at least 32x16x16x4 clock cycles to pass another block therein. Therefore, the size of the accumulator 402 has a direct impact on reducing the pressure on the network. If the accumulator is just a single fast accumulator (for example, the accumulator's accumulation operation speed is required to be 1 clock cycle), then the accumulator will be very expensive. Therefore, using the hierarchical accumulator disclosed herein provides significant benefits in these types of applications because the output matrix can be accumulated in a slightly cheaper memory that can be large enough to support large block sizes. This benefit is even more significant if both input matrices contain blocks transferred from remote memory.

[0039] Figure 5 A flowchart 500 is shown of a set of methods for operating a hierarchical accumulator according to a specific embodiment of the present invention disclosed herein. The flowchart 500 includes Figure 1 , which are nested within step 501 where the first matrix and the second matrix are multiplied to generate an output matrix. Flowchart 500 also includes step 502 where the first matrix and the second matrix are provided to a computational circuit as a set of blocks. This step is shown as part of a super loop because the remaining steps can be executed multiple times before another block is needed. These blocks can be, for example, Figure 4 412 and 411 in the flowchart 500. The flowchart 500 also includes step 501, which is to use a calculation circuit to accept a set of blocks as operands. This step is shown as part of a loop because the remaining steps can be performed multiple times when different parts of the blocks are fed to the calculation circuit. The calculation circuit can be Figure 1 The calculation circuit 100 or Figure 3 300 in the calculation circuit. The calculation circuit may include a math circuit, such as math circuit 101. Second memory 114 may be at least as large as the output matrix. The math circuit may be a multiplier array, such as multiplier array 301. Thus, the portion of the block provided as an operand to the calculation circuit may be a sub-block from a set of operand inputs, such as set 130. The value from the math circuit used in step 151 may be generated using this set of blocks as operands and may include a set of products, and the first memory used in step 151 may be at least twice as large as the set of products. A related step is shown as step 502 of outputting the value of the first accumulator using the math circuit. Step 502 may involve generating the set of products required to calculate the output matrix.

[0040] According to the above method, example steps 151 and 152 of flowchart 500 may include substeps in which different layers of the hierarchical accumulator utilize different sectors of the first memory and the second memory. As shown, step 151 includes substep 510, which accumulates a value from the mathematical circuit into the first sector of the first memory using a first accumulator having a first memory, and substep 511, which is performed simultaneously. Substep 510 accumulates a value from the mathematical circuit into the first sector of the first memory using a first accumulator having a first memory, and substep 511, which accumulates a value from the second sector of the first memory into the second memory using a second accumulator having a second memory. As further shown, step 152 includes substep 512, which accumulates a value from the mathematical circuit into the second sector of the first memory using the first accumulator having a first memory, and substep 513, which is performed simultaneously. Substep 512 accumulates a value from the mathematical circuit into the second sector of the first memory using the first accumulator having a first memory, and substep 513 accumulates a value from the first sector of the first memory into the second memory using the second accumulator having a second memory.

[0041] The relative memory size and speed of executing the steps in an implementation of flowchart 500 may be as described in detail in the accompanying drawings. Figure 1-4 . For example, in a specific embodiment of the present invention, the first sector of the first memory will be at least as large as the output size of the output generated by the math circuit in each cycle of step 502. As another example, in a specific embodiment of the present invention, the size of the first sector of the first memory used in step 510 is equal to the output size of the output generated in step 502, and the size of the second sector of the first memory used in step 511 is also the same as the output size of the output generated in step 502. As another example, in a specific embodiment of the present invention, step 502 of outputting a value for the first accumulator using the math circuit is performed at an output speed, step 151 is performed at a first per value accumulation speed that is at least as fast as the output speed, and step 152 is performed at a second per value accumulation speed that is slower than the first per value accumulation speed.

[0042] The hierarchical accumulators disclosed herein can be part of a processing pipeline of a processor. A processor can include one or more hierarchical accumulators. The processor can come in various forms. A processor can process cores in a multi-core processor or a standalone processor. The processor can be implemented as a single-chip system, including a wafer-level single-chip system, a multi-chip single-package system, or in a multi-chip multi-package system, where the chips are typically connected to a common substrate, such as a printed circuit board (PCB), an intermediate layer, or a silicon mesh. The processor can be part of a processor network. The network can be a network on chip (NoC). The processor according to the present invention can also be part of a network that includes chips on multiple substrates connected together by a higher-level common substrate, such as in the case of multiple PCBs, each PCB having a set of chips, where the multiple PCBs are fixed to a common backplane. Processors consistent with the present invention can also be implemented in a chip-based system. For example, in a specific embodiment of the present invention, one or more processors can be packaged or implemented by one or more networked chips, such as connected by an intermediary.

[0043] The processor according to the present invention may include at least one non-transitory computer-readable medium. The at least one processor may include at least one computing node in a network of computing nodes. The medium may include cache memory on the processor. The medium may also include shared memory that is independent of the unique computing node. The medium may be a shared memory, a shared random access memory, or a DDR DRAM. The shared memory may be accessed through multiple channels. The non-transitory computer-readable medium may store data required to perform any method disclosed herein, instruction data disclosed herein, and / or operand data disclosed herein. The computer-readable medium may also store instructions that, when executed by the system, cause the system to perform the method disclosed herein. The concept of executing instructions is used here to describe the operation of a device that performs any logical or data movement operation, even if the "instruction" is completely specified in hardware (for example, an AND gate executes an "AND" instruction). The term is not meant to attribute programmable capabilities to the device.

[0044] A processor according to the present invention may include at least one logic circuit as described above. The logic circuit may include active and passive devices and may operate with one or more logic levels. The logic circuit may use Boolean logic operations, and the logic circuit may be a synchronous clock circuit or an asynchronous circuit. The logic circuit may include logic gates, flip-flops, latches, registers, and other basic circuit components that are required to generate digital logic outputs in response to digital logic inputs. The logic circuit may be implemented directly in hardware such that a logical OR operation is performed by a physical collection of transistors that implement an OR gate, and the storage of data elements involves the physical state of at least one flip-flop, delay line, or other physical storage element.

[0045] Although the description has been described in detail with respect to specific embodiments of the present invention, it should be understood that those skilled in the art, after understanding the foregoing, can easily envision changes, variants, and equivalents to these embodiments. Any method disclosed herein can be performed by a processor in combination with a computer-readable medium that stores instructions for the method in combination with the other hardware elements described above. Those skilled in the art can implement these and other modifications and variations of the present invention without departing from the scope of the present invention, the scope of the present invention being set forth in more detail in the appended claims.

Claims

1. A computing circuit comprising: Mathematical Circuits; a first accumulator communicatively connected to the math circuit, the first accumulator having a first memory and accumulating values ​​from the math circuit in the first memory; a second accumulator communicatively connected to the first memory, the second accumulator having a second memory and accumulating values ​​from the first memory in the second memory; and The first memory is faster and smaller than the second memory.

2. The computing circuit according to claim 1 , further comprising: a first sector of the first memory; a second sector of the first memory; and wherein: (i) when the second accumulator accumulates from the second sector of the first memory, the first accumulator accumulates in the first sector of the first memory; and (ii) the first accumulator accumulates at the second sector of the first memory while the second accumulator accumulates from the first sector of the first memory.

3. The computing circuit according to claim 2, wherein: The math circuit generates an output having an output size; and The first sector of the first memory is at least as large as the output size.

4. The computing circuit according to claim 3, wherein: The size of the first sector of the first memory is equal to the output size; and The size of the second sector of the first memory is equal to the output size.

5. The computing circuit according to claim 1 , wherein: The mathematical circuit outputs a value at an output rate; the first accumulator accumulating values ​​from the math circuit in the first memory at a first accumulation operation speed; The first accumulation operation speed is at least as fast as the output speed; the second accumulator accumulating values ​​from the first memory in the second memory at a second accumulation operation speed; and The second accumulation operation speed is slower than the first accumulation operation speed.

6. The computing circuit according to claim 5, wherein: the first accumulator performs a number of accumulation operations to store output data values ​​in the first memory in response to a set of operands applied to the math circuit; the second accumulator performing a number of accumulation operations to store the output data value in the second memory in response to the set of operands applied to the math circuit; and The number of accumulation operations for storing the output data value in the first memory is greater than the number of accumulation operations for storing the output data value in the second memory.

7. The computing circuit according to claim 1, wherein: The mathematical circuit is a multiplier array; The mathematical circuit generates a set of products required to calculate the output matrix; and The first memory is at least as large as the output matrix.

8. The computing circuit according to claim 1, wherein: The first memory is a register memory; and The second memory is a static random access memory.

9. A system using the computing circuit of claim 1, further comprising: Controller; Wherein, the computing circuit is a matrix multiplication circuit; wherein the controller is programmed to: (i) multiply a first matrix and a second matrix using the matrix multiplication circuit to generate an output matrix; (ii) provide the first matrix to the matrix multiplication circuit in a first series of blocks; and (iii) provide the second matrix to the matrix multiplication circuit in a second series of blocks; and Wherein, the second memory is at least as large as the output matrix.

10. The system of claim 9, further comprising: network; a remote memory accessible via the network; a local memory, the local memory being located on a substrate having the computing circuit; wherein the first series of blocks is provided from the local memory; and wherein the second series of blocks is provided from the remote memory.

11. A method comprising: using a first accumulator having a first memory, accumulating values ​​from the math circuit in the first memory; using a second accumulator having a second memory, accumulating values ​​from the first memory in the second memory; and The first memory is faster and smaller than the second memory.

12. The method according to claim 11, further comprising: accumulating values ​​from the math circuit into a first sector of the first memory using the first accumulator with the first memory, while accumulating values ​​from a second sector of the first memory into the second memory using the second accumulator with the second memory; and The values ​​from the math circuit are accumulated into the second sector of the first memory using the first accumulator with the first memory, while the values ​​from the first sector of the first memory are accumulated into the second memory using the second accumulator with the second memory.

13. The method according to claim 12, further comprising: generating an output having an output size using the mathematical circuit; and The first sector of the first memory is at least as large as the output size.

14. The method according to claim 13, wherein: The size of the first sector of the first memory is equal to the output size; and The size of the second sector of the first memory is equal to the output size.

15. The method according to claim 12, further comprising: outputting the value of the first accumulator at an output rate using the math circuit; wherein the accumulation is performed using the first accumulator at a first value-by-value accumulation speed, the first value-by-value accumulation speed being at least as fast as the output speed; and The second accumulator is used to perform accumulation at a second value-by-value accumulation speed, and the second value-by-value accumulation speed is slower than the first value-by-value accumulation speed.

16. The method according to claim 15, wherein: the first accumulator performs a number of accumulation operations to store output data values ​​in the first memory in response to a set of operands applied to the math circuit; the second accumulator performing a number of accumulation operations to store the output data value in the second memory in response to the set of operands applied to the math circuit; and The number of accumulation operations for storing the output data value in the first memory is greater than the number of accumulation operations for storing the output data value in the second memory.

17. The method of claim 11, wherein: The mathematical circuit is a multiplier array; The mathematical circuit generates a set of products required to compute an output; and The first memory is at least twice as large as the set of products.

18. The method according to claim 11, further comprising: multiplying the first matrix and the second matrix to produce an output matrix; providing the first matrix and the second matrix as a set of blocks to a computation circuit including the mathematical circuit; accepting the set of blocks as operands using the computational circuitry; wherein a value from the mathematical circuit is generated using the set of blocks as operands; and Wherein, the second memory is at least as large as the output matrix.

19. The method of claim 11, wherein: The first memory is a register memory; and The second memory is a static random access memory.

Images