Balanced binary tree structures for stream reducing operations

Abstract

To provide methods, systems and other embodiments for incorporating a balanced binary tree into multiplication modules of a tensor processor so as to execute sequences of instructions more efficiently for stream-reducing operations; where this abstract and the independent claims are concise signifiers of embodiments of the claimed inventions, and the abstract does not limit the scope of the claimed inventions.SOLUTION: In one or more ECINs disclosed herein, balanced binary tree structures are disclosed that enable a compiler to locally optimize an aggregation operation for the timing / area as needed without having an impact on the overall architecture.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] This disclosure is of general significance in the field of architectures for high performance processors, and of particular significance to the subject of advantageous interconnection of functional blocks in high performance processors. This information is for limited use in searching the prior art. [Background technology]

[0002] This patent document may be reproduced exactly as it appears in the U.S. Patent and Trademark Office application, but the assignee otherwise reserves all rights in any subset of the author's work contained herein protected by Section 102(a) of the Copyright Law, 17 U.S.C.

[0003] In the Background, Summary, and Detailed Description that follow, paragraph headings are signifiers that do not limit the scope of the claimed embodiments of the invention (ECIN). Citation or identification of any publication does not indicate relevance or use in the prior art. Paragraphs in all italic font (represented herein by underlines) indicate text that is present in one or more patent specifications filed by the assignee.

[0004] Statements enclosed in double angle brackets ("") indicate an exact copy of the statement appearing in the copyrighted work. Signifiers such as words or phrases enclosed in angle brackets ("") indicate terms that are not currently defined and do not have a meaning to be evaluated, or that will not have a meaning in that particular usage until defined (e.g., when the cited term "module" is first used).

[0005] This application claims the benefit of priority to U.S. Provisional Application No. 63 / 584,353, entitled "BALANCED BINARY TREE STRUCTURES FOR STREAM REDUCING OPERATIONS," filed September 21, 2023, the entire contents of which are expressly incorporated by reference herein.

[0006] Tensors are a family of mathematical structures that includes vectors, matrices, and higher dimensional arrays. Tensors are used in many areas of science and engineering, and large tensors with millions to billions of elements are used in numerical computations such as machine learning. One operation - multiplication - requires a huge amount of processing power for large tensors, and specialized processors have been developed in recent years.

[0007] One type of tensor processor is deterministic (the time and location of all instruction execution is known prior to execution), such as the Tensor Streaming Processor (TSP) sold by Groq Incorporated. These types of deterministic processors comprise a two-dimensional mesh of processor cores, where data flows across lanes and instructions flow across slices. This is generally shown in FIG. 3.

[0008] In this organization of tensor processors, each computational element implements a specific function and is stacked vertically in a specific "function slice" in one dimension (e.g., the Y dimension) of the two-dimensional on-chip mesh. Each function slice is independently controlled by a set of instructions specific to its on-chip role. For example, a MEM (memory) function slice supports read and write, but not necessarily Add (addition) or Mul (multiplication), which are commonly implemented in arithmetic function slices (e.g., Vector Execution Module (VXM) and Matrix Execution Module (MXM) function slices) for some typical machine learning (ML) algorithms, such as linear regression algorithms. In the X dimension, each function row comprises a complete set of different types of function cores, e.g., MEM, VXM, MXM, SXM (exchange and permute module), etc. Each function row is referred to herein as a superlane.

[0009] In the TSP architecture, data flows in the X dimension and instructions flow in the Y dimension. Function slices operate on data streams in a single instruction multiple data (SIMD) manner, where different data streams are processed independently in different superlanes of the same function slice using the same instruction sequence. Physically, it takes time for an instruction to move from one superlane to the next. As a result, data streams in different superlanes need to arrive at the function slice with a staggering delay. For example, with a staggering delay of one clock cycle, a data stream arrives at superlane 0 at time t, superlane 1 at time t+1, and so on. If the staggering delay is 2, the data stream arrives at superlane 0 at time t, superlane 1 at time t+2, superlane 2 at time t+4, and so on.

[0010] The stagger delay of a data stream is dictated by the need to arrive in time to correctly execute its intended instructions at each function slice. The stagger delay of a data stream, in turn, is determined by the traversal delay of the instruction sequence that produces the data stream. For example, a sequence of memory read instructions traversing through a MEM slice with a delay of one clock cycle moving from one superlane to the next will produce a data stream with a stagger delay of one clock cycle. Naturally, in order for different function slices to be compatible with each other, the stagger delay needs to be the same.

[0011] In traditional machine learning (ML) and high performance computing (HPC) applications, there are specialized operations that perform aggregation on multiple data streams and produce fewer streams or a single result stream. We call such operations that consume multiple input streams and produce fewer output streams streams stream reduction. Unlike typical SIMD operations where each function core operates on x input streams and produces x output streams in the same dimension (dimension X), stream reduction operations need to change the direction of data flow from dimension X to dimension Y for aggregation, and then change it back from dimension Y to dimension X for writing back the single result stream.

[0012] For example, in a matrix multiplication C=A·B, the element cij in the result matrix C is the sum of s multiplications between the elements of the i-th row of matrix A and the j-th column of matrix B, e.g.

[0013]

number

[0014] where s is the number of columns in matrix A or the number of rows in matrix B. To compute a matrix multiplication on the TSP, the operation (aix × bxj) is performed in parallel in s / x different superlanes, where each superlane handles the multiplication of x elements. Each result element cij is computed by aggregation of the results of the multiplication from the s / x superlanes and then written back into a single result stream present in the i / x superlanes.

[0015] Stream-reduced operations are essential for both ML and HPC applications. Due to their differences from typical SIMD operations, others have failed to implement these operations efficiently in deterministic streaming processors, and therefore cannot provide solutions to at least the following three problems:

[0016] 1) Perfectly managing dependencies between the completion times of operations in the X and Y dimensions. The stream reduction operations in each superlane require the input streams from the X and Y dimensions to be synchronized with each other. This requires effectively staggering the input data streams with a delay determined by the instruction traversal time through the superlane plus the partial aggregation logic in each superlane.

[0017] 2) Different stream reduction operations can incur different delays determined by the aggregation logic required, which in turn determines the shear delay of the data streams, thus adding additional challenges to support multiple stream reduction operations at the architecture level in a unified manner.

[0018] 3) Stream reduce increases the wiring demands on the slice through which the instructions flow. This is due to the need to pass partial aggregation results from one superlane to its adjacent superlanes. Furthermore, the final result of the stream reduce operation needs to be written back to the target superlane based on the index.

[0019] What others have failed to provide is a way to more efficiently generate sequences of instructions that optimize stream reduction for tensor processors. Summary of the Invention [Problem to be solved by the invention]

[0020] This Summary is a brief set of signifiers for at least one ECIN (which may be discovery, see 35 U.S.C. §100(a), see 35 U.S.C. §100(j)) for use in a transaction where the specification and drawings, together with any claims, satisfy 35 U.S.C. §112.

[0021] In one or more ECINs disclosed herein, a balanced binary tree structure is disclosed and incorporated into a multiplication module of a tensor processor to address previously unsolved problems of efficiently executing instructions for stream reduction operations. In particular, a balanced binary tree structure is disclosed that can be advantageously used to erase delay dependencies in aggregation logic, eliminating the need to further shift the data stream according to the aggregation delay. Thus, different stream reduction operations can be unified. [Means for solving the problem]

[0022] One or more ECINs disclosed in this specification disclose a balanced binary tree structure that allows a compiler to locally optimize aggregation operations for timing / area as needed without having an impact on the overall architecture. Effect of the Invention

[0023] The one or more ECINs disclosed herein allow for a balanced binary tree structure with the same unit logic design for all superlanes, thereby allowing for more efficient design scalability of the TSP.

[0024] The summary of the present invention does not completely depict any ECIN. Although the summary of the present invention may depict at least one essential element of an ECIN enabled by the present specification and drawings, the summary of the present invention does not depict any limitation on the scope of any ECIN. [Brief description of the drawings]

[0025] The following detailed description, drawings, and claims illustrate one or more applications of ECIN and the advances enabled thereby. All drawings are used only to provide knowledge and understanding and do not limit the scope of any ECIN. Such drawings are not necessarily drawn to scale.

[0026] The drawings may have the same or similar reference signifiers in the form of labels (such as alphanumeric symbols, e.g., reference numbers) and may indicate the same or equivalent function or use. Furthermore, reference signifiers of the same type may be distinguished by appending to the reference label a dash and a second label that distinguishes the similar signifiers. If only a first label is used in this specification, its use applies to any similar component having the same label regardless of any other reference labels. A brief list of the drawings is below.

[0027] [Figure 1] FIG. 1 illustrates a system for compiling a program to be executed on a tensor processor, according to one embodiment. [Figure 2A] FIG. 2 illustrates an instruction and data flow in a processor having a function slice architecture in accordance with some embodiments. [Figure 2B] FIG. 2 illustrates an instruction and data flow in a processor having a function slice architecture in accordance with some embodiments. [Diagram 3] FIG. 2 illustrates an instruction and data flow in a processor having a function slice architecture in accordance with some embodiments. [Figure 4] FIG. 2 illustrates an MXM slice and a daisy-chain circuit incorporated into the MXM slice according to some embodiments. [Diagram 5] FIG. 2 illustrates an MXM slice and a balanced binary tree circuit incorporated in the MXM slice according to some embodiments. [Figure 6] A diagram showing one unit of a balanced binary tree circuit according to some embodiments. [Figure 7] FIG. 2 illustrates an unbalanced binary tree circuit according to some embodiments. [Figure 8A] A diagram illustrating a balanced binary tree with four superlanes in accordance with some embodiments. [Figure 8B] A diagram illustrating a balanced binary tree with four superlanes in accordance with some embodiments. [Figure 8C] A diagram illustrating a balanced binary tree with four superlanes in accordance with some embodiments. [Figure 9] A diagram illustrating a balanced binary tree with eight superlanes in accordance with some embodiments. [Figure 10] A diagram illustrating a balanced binary tree with 16 superlanes in accordance with some embodiments. [Figure 11] FIG. 1 illustrates a computer system suitable for enabling embodiments of the claimed invention.

[0028] In the figures, reference numbers may be omitted consistent with accepted engineering practice, but those of ordinary skill in the art will understand the components illustrated in the context of the figure as a whole, the context of the accompanying description regarding such figure, and the context of the embodiment of the claimed invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0029] The drawings and the description show at least one ECIN for the purpose of providing knowledge and understanding only. In order to minimize the length of the description, various features, structures or characteristics may be described together in a single embodiment, but they may also be used in other embodiments without being described. Variations of any of these elements, as well as modules, processes, machines, systems, manufactures, or compositions disclosed by such embodiments and / or examples, are easily used in trade. The drawings and the description show, implicitly or explicitly, advantages and improvements of at least one ECIN for use in trade.

[0030] In the drawings and the description, numerous specific details may be described to enable at least one ECIN. Any embodiment disclosed herein shows a tangible form of the claimed invention. In order not to reduce the significance of the embodiments and / or examples in this description, some elements known to those skilled in the art may be combined for presentation and illustration and not specified in detail. In order not to reduce the significance of these embodiments and / or examples, some well-known processes, machines, systems, manufactures, or compositions are not described in detail. However, those skilled in the art can use these embodiments and / or examples in their trade without these specific details or their equivalents. Thus, the description focuses on enabling elements of the invention of any ECIN. When this description refers to several elements in the singular, more than one element may be shown in the figures, and similar elements are labeled with similar numbers.

[0031] FIG. 1 illustrates a system 100 for compiling a program to be executed on a tensor processor, according to one embodiment. The system may also be utilized to generate power usage information for the compiled program, according to some embodiments. System 100 includes a user device 102, a server 110, and a processor 120. Each of these components and their subcomponents (if any) are described in more detail below. Although a particular configuration of components is described herein, in other embodiments, system 100 has different components that perform the functions of system 100 in a different order or using different mechanisms. For example, while FIG. 1 illustrates a single server 110, in other embodiments, compilation, assembly, and power usage functions are performed on different devices. For example, in some embodiments, at least a portion of the functions performed by server 110 are performed by user device 102.

[0032] The user device 102 comprises any electronic computing device, such as a personal computer, laptop, or workstation, that uses an application program interface (API) 104 to build a program to be executed on the processor 120. The server 110 receives a program specified by a user at the user device 102 and compiles the program to generate a compiled program 114. In some embodiments, the compiled program 114 enables a data model for prediction that processes input data and makes a prediction from the input data. Examples of prediction are category classification or prediction of time series values ​​made with a classifier. In some embodiments, the predictive model describes a machine learning model that includes nodes, tensors, and weights. In one embodiment, the predictive model is specified as a TensorFlow model, the compiler 112 is a TensorFlow compiler, and the processor 120 is a tensor processor. In another embodiment, the predictive model is specified as a PyTorch model, and the compiler is a PyTorch compiler. In other embodiments, other machine learning specification languages ​​and compilers are used. For example, in some embodiments, the predictive model defines nodes that represent operators (e.g., arithmetic operators, matrix transform operators, Boolean operators, etc.), tensors that represent operands (e.g., values ​​that the operators modify, such as scalar values, vector values, and matrix values ​​that may be represented in integer or floating-point format), and weight values ​​that are generated after training and stored in the model. In some embodiments, if the processor 120 is a tensor processor with a function slice architecture, the compiler 112 translates the program into a set of operations to be executed by the processor 120 and generates an explicit plan for how the processor will execute the program by specifying when each instruction will be executed, which function slices will perform the work, and which stream registers will hold the operands. This type of scheduling is known as "deterministic scheduling."This explicit plan for execution includes information about an explicit stream reduction process with a balanced binary tree architecture.

[0033] The assembler 116 receives the compiled program 114 generated by the compiler 112 and performs the final compilation and linking of the scheduled instructions to generate a compiled binary. In some embodiments, the assembler 116 maps the scheduled instructions indicated in the compiled program 114 to the hardware of the server 110 and then determines the exact component queue in which to place each instruction.

[0034] The processor 120 is, for example, a hardware device with a large number of matrix multiplier units that accepts a compiled set of instruction bits (e.g., a "binary") assembled by the assembler 116 and executes the instructions contained in the compiled binary. The processor 120 may include one or more blocks of circuitry for matrix arithmetic, numerical conversion, vector calculations, short-term memory, and data substitution / exchange. Once such a processor 120 is a tensor processor with a function slice architecture. In some embodiments, the processor 120 comprises multiple tensor processors connected together to function as a single deterministic entity. Exemplary Processor 2A and 2B show instruction and data flow in a processor having a function slice architecture according to some embodiments. One implementation possibility of processor 200 is as an application specific integrated circuit (ASIC) and corresponds to processor 120 shown in FIG.

[0035] The functional units of the processor 200 (also called "functional tiles") are aggregated into multiple functional processing units 205 (hereinafter referred to as "slices"), each corresponding to a particular function type, in some embodiments. For example, different functional slices of the processor correspond to processing units for MEM (memory), VXM (vector execution module), MXM (matrix execution module), NIM (numerical interpretation module), and SXM (exchange and substitution module). In other embodiments, each tile may include an aggregation of functional units, such as a tile having both a MEM unit and an execution unit, as an example. As shown in Figures 2A and 2B, each slice corresponds to a column of N functional units that extend in a direction different (e.g., perpendicular) from the direction of data flow. The functional units of each slice may share an instruction queue (not shown) that stores instructions and an instruction control unit (ICU) 210 that controls the execution flow of the instructions. Instructions in each instruction queue are executed only by the functional units in the associated slice of the queue and not by other slices of the processor. In other embodiments, each functional unit has an associated ICU that controls the execution flow of the instructions.

[0036] Processor 200 also includes communication lanes for transporting data between functional units of different slices. Each communication lane connects to each of slices 205 of processor 200. In some embodiments, communication lanes 220 connecting rows of functional units of adjacent slices are referred to as "superlanes" and comprise multiple data lanes or "streams," each configured to transport data values ​​along a particular direction. For example, in some embodiments, each functional unit of processor 200 is connected to a corresponding functional unit on an adjacent slice by a superlane that is comprised of multiple lanes. In other embodiments, processor 200 includes a communication device, such as a router, to transport data between adjacent functional units.

[0037] By arranging the functional units of the processor 200 into different function slices 205, the on-chip instruction and control flow of the processor 200 is separated from the data flow. Since many types of data are acted upon by the same set of instructions, for visualization purposes, it is important to visualize the flow of instructions rather than the flow of data. In some embodiments, FIG. 2A illustrates the flow of instructions within the processor architecture, while FIG. 2B illustrates the flow of data within the processor architecture. As shown in FIGS. 2A and 2B, instructions and control signals flow in a first direction across the functional units of the processor 200 (e.g., along the length of the function slices 205), while data flow flows in a second direction across the functional units of the processor 200 (e.g., across the function slices) that is not parallel to the first direction via communication lanes (e.g., superlanes) connecting the slices.

[0038] In some embodiments, functional units in the same slice execute instructions in a "staggered" fashion, where instructions are issued for each tile in the slice over a period of N cycles. For example, the ICU for a given slice issues instructions to the first tile of the slice (e.g., the bottom tile of the slice shown in FIG. 2B closest to the ICU of the slice) during a first clock cycle, which are passed to subsequent functional units of the slice over subsequent cycles. That is, each row of functional units (corresponding to functional units along a particular superlane) of processor 200 executes the same set of instructions, but offset in time with respect to functional units in adjacent rows.

[0039] The function slices of the processor are configured such that operand data read from a memory slice is intercepted by a different function slice as the data moves across the chip, and the results flow in the opposite direction, where they are written back to the memory. For example, a first data flow from a first memory slice flows in a first direction (e.g., toward the right), where it is intercepted by a VXM slice that performs a vector operation on the received data. The data flow then continues to an MXM slice that performs a matrix operation on the received data. The processed data then flows in a second direction opposite to the first direction (e.g., toward the left), where it is again intercepted by a VXM slice to perform an accumulation operation, and then written back to the memory slice.

[0040] In some embodiments, the function slices of the processor are configured such that data flow between the memory and the function slices occurs in both a first direction and a second direction. For example, a second data flow originating from a second memory slice travels in a second direction toward a second slice, where the data is intercepted and processed by a VXM slice before proceeding to a second MXM slice. The results of the matrix operation performed by the second MXM slice then flow back in the first direction toward the second memory slice.

[0041] In some embodiments, stream registers are located along the superlanes of the processor according to some embodiments. The stream registers are located between the function slices of the processor to facilitate the transport of data (e.g., operands and results) along each superlane. For example, within the memory domain of the processor, the stream registers are located between a set of four MEM units. The stream registers are architecturally visible to the compiler and serve as a primary hardware structure with visibility into the execution of the program. Each functional unit of the set includes a stream circuit configured to enable the functional unit to read or write the stream register in either direction of the superlane. In some embodiments, each stream register is implemented as a collection of registers corresponding to each stream of the superlane and is sized based on the base data type used by the processor (e.g., if the base data type of the TSP is INT8, then each register may be 8 bits wide). In some embodiments, to support larger operands (e.g., FP16 or INT32), multiple registers are collectively treated as one operand, where the operands are transmitted over multiple streams of the superlane.

[0042] The superlanes of functional units that characterize these functions, the slicing of instruction flows, and the processing of different types of integers and floating point numbers, all of which occur trillions of times per second, create complex power flows and possible disruptive power variations that can adversely affect the performance of the processor. However, given the deterministic nature of execution by the processor, any disruptive power variations (such as voltage droop) can be determined prior to program execution, and information regarding such variations (such as processor instructions and timings for such instructions) is supplied by the compiler to the processor for use by the processor during program execution to mitigate the variations. Stream Reduction The stream reduce instruction can be considered to have two sub-operations ("sub-ops"): distributed Y dimension aggregation and Y dimension result writeback. Distributed Y dimension aggregation performs a partial aggregation in each superlane. This sub-op has timing dependencies on both the X and Y dimension inputs, meaning that the operands from the X and Y dimensions need to be correctly related to each other to produce the correct partial aggregation result. Y dimension result writeback writes the final aggregation result to the target superlane through a bus dedicated to the Y dimension that arrives on all superlanes.

[0043] As explained above, the main challenge of stream reduce operations arises from aggregation in the Y dimension. A simple implementation daisy-chains n superlanes in order from 0 to n-1. An example of a daisy-chain built into an MXM slice is shown in FIG. 4. Advantages of such an implementation include: 1) The partial aggregation logic in superlane i only needs to look at superlanes i and (i-1) for input operands; the logic is independent of superlane number / ID and therefore only needs to be implemented once and replicated for all superlanes. 2) The output of the partial aggregation logic only needs to be routed to the next superlane. There is no need for partial results to traverse multiple superlanes. This reduces the need for routing resources for the Y dimension.

[0044] However, the following drawbacks have a more significant impact on the TSP architecture: 1) the delay of the partial aggregation logic needs to match the data stream's skew delay d. Thus, the partial aggregation logic needs to be completed within a delay of d or the data stream's skew delay needs to be adjusted according to the needs of each stream reduction instruction along with the instruction propagation path of the function slice, which can be costly for the entire TSP chip; 2) the input data needs to go through a longer chain of aggregation logic, which can lead to precision loss. For example, the input from superlane 0 needs to go through a whole chain of n partial aggregation logic elements. Balanced Binary Tree FIG. 5 shows an MXM slice for 16 superlanes and a balanced binary tree circuit incorporated in the MXM slice. In the daisy-chain structure, the arrival time of the operand in superlane 1 must match the partial aggregation output from superlane 0, which is determined by the delay of the partial aggregation logic d. In other words, the operand in superlane i must arrive at time i*d. Since the delay d plays a role in determining the arrival time of the operand, this means that the architecture has a dependency on the implementation of the partial aggregation logic, e.g., when the implementation of the partial aggregation logic changes, other parts of the design must also change. Or, if the architecture remains the same, the partial aggregation logic must meet the delay constraint imposed by the stagger delay of the data stream.

[0045] In contrast, the delay of partial aggregation logic is eliminated in a balanced binary tree structure. For 16 superlanes, there are four layers of partial aggregation logic from the leaf nodes to the root node, where the root node generates the final aggregation result. In the first layer, close to the leaf nodes, the arrival time of the operands has no dependency on the aggregation logic. Assuming that data arrives with a staggered delay of one cycle, e.g., data in superlane i arrives during cycle i, the partial aggregation logic in the first layer only needs to insert one cycle of delay for the operand coming from the lower superlane. In the second layer, the arrival times of the two operands become 2*j+1+d and 2*j+3+d, respectively, for the jth partial aggregation logic in the second layer. Thus, the aggregation logic only needs to insert two cycles of delay for the lower operand. Note that the delay d of the partial aggregation logic from the previous layer is eliminated between two operands feeding the same partial aggregation logic, and the number of delay buffers required is determined solely by the shear delay of the initial data stream. This continues as the operands are processed by the balanced binary tree structure, and the partial aggregation logic needs to insert a four cycle delay for its lower operand at the third layer, an eight cycle delay at the fourth layer, and so on.

[0046] The delay per layer is the cumulative delay from the previous layer plus the current layer. Hence, the delay buffering added at each layer does not increase as the data moves further down in the layer. Furthermore, the delay at each layer in a binary tree will be the same since it is a balanced binary tree.

[0047] The advantages of the balanced binary tree structure are: The delay of the partial aggregation logic does not affect other parts of the design or dictate the data stream shear delay. In other words, there are fewer constraints on how the aggregation logic must be implemented. Multiple stream reduction instructions can be supported without significant impact on the overall architecture or data shear delay. Data traverses fewer stages of partial aggregation to reach the final result, which can lead to better accuracy. It requires less partial aggregation logic than a daisy-chained structure when the number of superlanes is greater than two.

[0048] The disadvantages are: It becomes difficult to partition the balanced binary tree into different superlanes and make them the same. For TSP, it is important to keep all superlanes the same to reduce both logical and physical design costs. The output of the partial aggregation logic may need to be routed to non-adjacent superlanes depending on its location in the binary tree. This increases the routing traffic between different superlanes. The number of superlanes is preferably a power of 2, otherwise the delay of the partial aggregation logic cannot be completely eliminated. If the number of superlanes is not a power of 2, the additional buffering capacity required increases by log2 depending on the depth of the tree.

[0049] Figure 6 shows one unit of a balanced binary tree circuit similarly built into each superlane. In this configurable logic unit design, a set of multiplexers is added at each position, where the design can vary depending on the location of the superlane. The modified design can use different signal routing or different delay paths. The select signals of those multiplexers are controlled by a group of configuration registers, which are set by software at initialization time.

[0050] FIG. 7 shows an unbalanced binary tree circuit using five superlanes as an example. The aggregation of superlanes 0 to 3 is well balanced, while the aggregation of superlane 4 with partial results from 0 to 3 is not balanced in terms of branch depth or delay. Superlane 4 therefore needs to insert a buffer with a latency of 2d-1 (assuming d≧1) to match the arrival times of the partial results from superlanes 0 to 3. The latency of the inserted buffer in turn has a dependency on the delay d of the partial aggregation logic. Therefore, for a balanced binary tree structure to completely cancel the effect of the partial aggregation logic delay, the number of superlanes should advantageously be a power of two.

[0051] Figures 8A-8C show a balanced binary tree with four superlanes. In the balanced binary tree structure, it is difficult to partition the whole aggregation tree into different superlanes while keeping the logic the same in each superlane. To address this issue, a unit design for each superlane and a connection scheme between different superlanes are used as shown in Figures 8A-8C.

[0052] 8A shows a unit design for partial aggregation logic, which consists of aggregation logic, a four-entry delay buffer, three routing channels (indicated by the shaded rectangles), and three other routing channels (indicated by the dotted rectangles). Local input data is represented as black circles.

[0053] FIG. 8B shows how two adjacent superlanes (of a set of superlanes) must be connected. Assume that the input data of each superlane is staggered with a delay of one cycle. Local inputs from a higher superlane are routed to a lower superlane. Note that we assume that routing across a superlane boundary in the Y dimension adds one clock cycle of latency. To match the input latency, local inputs from a lower superlane must go through a two-cycle buffer before feeding into the aggregation logic. A partial aggregation result is produced at time 2+d. Note that the aggregation logic in the upper superlane was left unused. This connection scheme applies to all pairs of two adjacent superlanes.

[0054] FIG. 8C shows how partial aggregation results for four superlanes (e.g., a set of superlanes) should be connected. A partial result (e.g., a first partial result of the partial aggregation results) from an upper pair of superlanes (e.g., a first set of superlanes of the set of superlanes) is routed to a partial result (e.g., a second partial result of the partial aggregation results) from a lower pair of superlanes (e.g., a second set of superlanes of the set of superlanes). The two partial results are aggregated in superlane 1, where the partial result from the lower pair is delayed by two cycles (e.g., a defined number of cycles that may be more or less than two cycles) to match the latency for the input from the upper pair. The final partial results for the four superlanes are generated at time 5+2d. This connection scheme is applied to all pairs of four adjacent superlanes. Note that there is one unused aggregation logic block in superlane 3 in this scheme.

[0055] FIG. 9 shows a balanced binary tree with eight superlanes, which shows how the connection scheme should be expanded to go from four superlanes to eight superlanes. For eight superlane connections, two four-superlanes are stacked together and the partial results of the four superlanes are routed from superlane 5 to superlane 3. Similarly, the partial results from superlane 1 are routed to superlane 3. A four-cycle delay buffer is inserted for the lower input in superlane 3, since the partial results from the top four superlanes are four clock cycles later than those from the bottom four superlanes.

[0056] FIG. 10 shows a balanced binary tree with 16 superlanes, which shows how the connection scheme should be expanded to go from 8 superlanes to 16 superlanes. The 16 superlane connection can be completed by stacking two connected 8 superlanes and routing the partial results to either superlane 7 or 15. If the total number of superlanes in the TSP is 16, the partial results must be routed to superlane 15, so that the final aggregation results can be distributed back to each superlane using a single bus, as shown in FIG. 6. Note that no additional delay buffers are required in superlane 15 for the lower input operands of the partial aggregation logic. The delay is matched when the partial results from the lower 8 superlanes meet the partial results from the upper 8 superlanes.

[0057] Provided herein is a balanced binary tree structure for stream reduction operations. An embodiment relates to a method including incorporating a balanced binary tree structure in a multiplication module of a tensor processor to more efficiently execute a sequence of instructions for stream reduction operations. In one implementation, the balanced binary tree structure may include at least four superlanes. However, in other implementations, the balanced binary tree structure may include any number of superlanes.

[0058] In some implementations, incorporating a balanced binary tree structure into a multiplication module of a tensor processor can include canceling delays of partial aggregation logic in the balanced binary tree structure. In some implementations, the incorporating can include canceling delay dependencies in the aggregation logic. For example, canceling delay dependencies can include eliminating the need to stagger the data stream according to the aggregation delay. In another example, the delay dependencies can include unifying stream reduction operations.

[0059] According to some implementations, incorporating a balanced binary tree structure can include enabling a compiler to locally optimize aggregation operations. In some implementations, incorporating a balanced binary tree structure can include enabling a balanced binary tree structure with the same unit logic design for a set of superlanes. In addition to these implementations, for example, enabling a balanced binary tree structure can include operatively connecting two adjacent superlanes of the set of superlanes.

[0060] In another example, enabling the balanced binary tree structure can include operatively connecting the partial aggregation results of the set of superlanes. Further to this example, the method can include routing a first partial result of the partial aggregation results for a first set of superlanes of the set of superlanes toward a second partial result of the partial aggregation results for a second set of superlanes of the set of superlanes. In some implementations, the method can include delaying the second partial result by a defined number of cycles determined to match the latency for the first partial result.

[0061] Another embodiment relates to a system including a compiler that incorporates a balanced binary tree into a multiplication module of a tensor processor to more efficiently execute sequences of instructions for stream reduction operations. In one example, the compiler adds a set of multiplexers at a location determined based on the location of the superlanes. Further to this example, the selection signals of the set of multiplexers may be controlled by a group of configuration registers.

[0062] In another example, the compiler can include a unit design for the partial aggregation logic. Further to this example, the unit design for the partial aggregation logic can include a multi-entry delay buffer, a first set of routing channels, and a second set of routing channels.

[0063] According to one implementation, the compiler enables a balanced binary tree with the same unit logic design for a set of superlanes. In addition to this implementation, the compiler operatively connects two adjacent superlanes of the set of superlanes. In one implementation, the balanced binary tree can include at least four superlanes. However, in other implementations, the balanced binary tree can include three or fewer superlanes, four superlanes, or five or more superlanes. Detailed Description of the Invention-Technical Support From data / instructions to processors / programs Data and Information. Although "data" and "information" are often used interchangeably (e.g., "data processing" and "information processing"), the term "datum" (plural "data") generally denotes a representation of a factual value (e.g., a measurement of a physical quantity such as the current in a wire or the price of gold) or an answer to a question (e.g., "yes" or "no"), whereas the term "information" generally denotes a set of data with a structure (often denoted by a "data structure"). Data structures are used in the trade to convert electronic devices for use as specific machines as manufactured goods (see In re Lowry, 32 F.3d 1579 [CAFC, 1994]). Data and information are physical objects, e.g., binary data ("bits" usually denoted by "0" and "1") enabled by two levels of voltage in digital circuits or electronic components. For example, data may be enabled as electrical, magnetic, optical, or acoustic signals or states, quantum states such as particle spins enabling "qubits," or physical states of atoms or molecules. All such data and information, when enabled, may be stored, accessed, transferred, combined, compared, or acted upon in actions that require and cost energy.

[0064] The term "processing" as used herein refers to a man-made, finite, ordered set of physical actions ("actions" also denoted by "operations" or "steps") to produce at least one result. Some types of actions include transformation and transport. An action is a technical application of one or more natural laws of science or man-made laws of technology. An action often changes the physical state of a machine, of a data and information structure, or of a composition. Two or more actions may occur at approximately the same time, or one action may occur before or after another action, if the processing produces the same result. Descriptions of the physical actions and / or transformations that comprise processing are often indicated with a set of gerund phrases (or their semantic equivalents), typically preceded by the signifier "step of" (e.g., "processing comprising steps of measuring, transforming, partitioning, and then distributing ..."). The signifiers "algorithm," "method," "procedure," "(sub)routine," "protocol," "recipe," and "technique" are often used interchangeably with "process," and 35 U.S.C. §100 defines a "process" as a type of process that is always patentable under 35 U.S.C. §101 by legislative statute. As used herein, the term "thread" refers to a subset of the entire process. A process may be partitioned into multiple threads that may be used simultaneously or nearly simultaneously.

[0065] The term "component" as used herein (and also indicated by "portion" and generally by "element" when described in patent text or figures) refers to a physical object used in combination with other components to enable processing. For example, electronic components are used in processing to affect the physical state of one or more electromagnetic or quantum particles / waves (e.g., electrons, photons) or quasiparticles (e.g., electron holes, phonons, magnetic domains) and their associated fields or signals. Electronic components generally have at least two connection points that are attached to a conductive component, typically a conductive wire or line or optical fiber, with one conductive component end attached to the component and the other end connected to another component as part of a circuit through which current or photons flow. There are at least three types of electrical components: passive, active, and electromechanical. Passive electronic components generally do not introduce energy into a circuit, and such components include resistors, memristors, capacitors, magnetic inductors, crystals, Josephson junctions, transducers, sensors, antennas, waveguides, and the like. Active electronic components require a source of energy and can inject energy into a circuit, such components include semiconductors (e.g., diodes, transistors, optoelectronic devices), vacuum tubes, batteries, power supplies, displays (e.g., LEDs, LCDs, lamps, CRTs, plasma displays), etc. Electromechanical components use mechanical force and structure to affect the flow of electrical current, such components include switches, relays, protection devices (e.g., fuses, circuit breakers), heat sinks, fans, cables, wires, terminals, connectors, and printed circuit boards.

[0066] One of the most important components as a commodity in commerce is the integrated circuit and its abstractions. The term "integrated circuit" as used herein refers to a set of connected electronic components on a small substrate of semiconductor material such as silicon or gallium arsenide with the components fabricated on one or more layers (hence the use of the signifier "chip"). Other signifiers for "integrated circuit" include "monolithic integrated circuit", "IC", "chip", "microchip", and "system on chip" ("SoC"). Examples of types of integrated circuits include gate / logic arrays, processors, memories, interface chips, power controllers, and operational amplifiers. The term "cell" as used in electronic circuit design refers to the specification of one or more components, e.g., a set of transistors connected to function as a logic gate. Cells are typically stored in a database that is to be accessed by the circuit designer and the design process.

[0067] The term "module" as used herein refers to a tangible structure for operating on data and information. For example, the term "module" can refer to a process that transforms data and information, e.g., a process comprising a computer program (defined below). The term "module" can also refer to one or more interconnected electronic components, such as digital logic devices. A process comprising a module can also be translated into a specification for a structure of electronic components that transforms data and information that produces the same result as the process, when specified in a programming language (defined below), such as System C or Verilog. This last sentence is derived from a modified Church-Turing thesis, which is expressed as "whatever can be transformed by a (patentable) process and a processor can be transformed by a (patentable) equivalent set of modules," as opposed to the doublethink of simply removing just one of the "(patentable)".

[0068] A module may be permanently structured (e.g., a circuit with immutable connections), temporarily structured (e.g., a circuit or process that is changeable with a set of data), or a combination of the two forms of structuring. For example, a permanently structured module may be manufactured using application specific integrated circuits ("ASICs") such as, for example, arithmetic logic units ("ALUs"), programmable logic arrays ("PLAs"), or read only memories ("ROMs"), all of which are typically structured during manufacturing. For example, a permanently structured module comprises an integrated circuit. A temporary structured module may be manufactured using, for example, field programmable gate arrays (e.g., FPGAs sold by Xilinx or Intel's Altera), random access memories (RAMs), or microprocessors. For example, data and information are transformed using the data as addresses in a RAM or ROM memory that stores output data and information. A temporary structured module may be embedded in a permanently structured module (e.g., an FPGA embedded in an ASIC).

[0069] A temporarily structured module may be structured in multiple time periods. For example, a processor with one or more modules may have its modules initially structured by a manufacturer in a factory and then further structured by a user when used in trade. The processor may comprise one or more sets of modules during a first time period and then be reconfigured to comprise one or a different set of modules during a second time period. The decision to manufacture or implement a module in a permanently structured form, a temporarily structured form, or a combination of the two forms depends on trade considerations such as cost, time considerations, resource constraints, tariffs, maintenance needs, national intellectual property laws, and / or specific design objectives [FACT]. How a module is used and its functions are largely independent of the physical form in which it is manufactured or enabled. This last sentence also follows from a modified Church-Turing thesis.

[0070] The term "processor" as used herein refers to a processing machine for tangible data and information for use in a transaction that physically transforms, transfers, and / or transmits data and information using at least one process. A processor consists of one or more modules, such as a central processing unit ("CPU") module, an input / output ("I / O") module, a memory control module, a network control module, and / or other modules. The term "processor" can also refer to one or more processors, or one or more processors with multiple computational cores / CPUs, specialized processors (e.g., graphics processors or signal processors), and combinations thereof. When two or more processors interact, one or more of the processors may be located remotely relative to the location of the other processors. If the term "processor" is used in another context, such as "chemical processor," it will be shown and defined in that context.

[0071] The processor may comprise, for example, digital logic circuits (e.g., binary logic gates) and / or analog circuits (e.g., operational amplifiers). The processor may also use a combination of technologies such as optical signal processing, DNA conversion, quantum computing, microfluidic logic processing, or optoelectronic processors. For data and information structured with binary data, any processor capable of transforming data and information using AND, OR, and logic operations (and their derivatives such as NAND, NOR, and XOR operations) may also transform data and information using any function of Boolean logic. Processors such as analog processors, such as artificial neural networks, may also transform data and information. There is no scientific evidence that any of these technology processors process, store, and retrieve data and information using any processing or structure equivalent to the bioelectrical structure and processing of the human brain.

[0072] One or more processors may also use processing in a "cloud computing" or "time sharing" environment, where the time and resources of multiple remote computers are shared by multiple users or processors communicating with the computers. For example, a group of processors may use at least one process available in a distributed or remote system, which use a communications network (e.g., the Internet or Ethernet) and use one or more designated network interfaces ("interfaces" defined below) (e.g., application program interfaces ("APIs") that expose functions and data structures for communicating with remote processes).

[0073] The terms "computer" and "computer system" as used herein (as further defined below) include at least one processor that performs operations such as (but not limited to) Boolean logic operations on data and information using electronic gates that may comprise transistors, with the addition of, for example, memory (e.g., memory structured with flip-flops using NOT-AND or NOT-OR operations). Any processor that can perform logical AND, OR, and NOT operations (or their equivalents) is Turing complete and computationally universal [FACT]. A computer may have a simple structure, for example, with an input / output module, a CPU module, and a memory, that performs processing to input signals, transform signals, and output signals without human intervention.

[0074] The term "programming language" as used herein refers to a structured grammar for specifying a set of operations and data for use by modules, processors and computers. Programming languages ​​include any of source or object code written in any combination of assembler instructions, instruction set architecture instructions, machine language instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or one or more higher level languages, such as the C programming language and similar general programming languages ​​(such as Fortran, Basic, Javascript, PHP, Python, C++), knowledge programming languages ​​(such as LISP, Smalltalk, Prolog, or CycL), electronic structure programming languages ​​(such as VHDL, Verilog, SPICE, or SystemC), textual programming languages ​​(such as SGML, HTML, or XML), or audiovisual programming languages ​​(such as SVG, MathML, X3D / VRML, or MIDI), and any future equivalent programming languages. The term "source code" as used herein refers to a set of instructions and data specified in textual form using a programming language. A large amount of source code for use in enabling any of the claimed inventions is available on the internet, including from source code libraries such as Github.

[0075] The term "program" (also called "application program") as used herein refers to one or more process and data structures that structure a module, processor, or computer for use as a "particular machine" (see In re Alappat, 33 F3d 1526 [CAFC, 1991]). One use of a program is to structure one or more computers, e.g., standalone, client, or server computers, or one or more modules, or a system of one or more such computers or modules. The term "computer application" as used herein refers to a program that has a particular use, e.g., enabling a text processing operation or enabling the encryption of a set of data. The term "firmware" as used herein generally refers to a type of program that structures a processor or computer, where firmware is smaller in size and generally less accessible and modifiable to a computer user than a typical application program. Computer programs and firmware are often specified using source code written in a programming language such as the C module, and circuits, processors, programs, and computers may be specified at multiple levels of abstraction using, for example, the SystemC programming language, and may have value as articles in commerce as taxable goods under the U.S. Uniform Commercial Code (see UCC Title 2, Article 1).

[0076] The program is transferred from a data and information device or storage system to one or more memories of a computer or computer system. The computer system generally has a device for reading the storage medium used to transfer the program and / or has an interface device for receiving the program over a network. This transfer is described in the general computer description section. Detailed Description of the Invention-Technical Support General Computer Description FIG. 11 illustrates a computer system suitable for enabling embodiments of the claimed invention.

[0077] In FIG. 11, the structure of computer system 500 generally includes at least one computer that communicates with peripheral devices via a bus subsystem 504. Generally, the computer includes a processor 502 (e.g., a microprocessor, a graphics processing unit, or a digital signal processor) or its electronic processing equivalents such as an application specific integrated circuit ("ASIC") or a field programmable gate array ("FPGA"). Generally, the peripheral devices include a storage subsystem 506 with a memory subsystem 508 and a file storage subsystem 510, a user interface input device 512, a user interface output device 514, and / or a network interface subsystem 516. The input / output devices enable direct and remote user interaction with the computer system 500. The computer system enables significant post-processing activity using at least one output device and / or a network interface subsystem.

[0078] A computer system may be structured as a server, client, workstation, mainframe, personal computer (PC), tablet PC, set-top box (STB), personal digital assistant (PDA), cellular phone, smartphone, web appliance, rack-mounted "blade", kiosk, television, game station, network router, switch or bridge, or any data processing machine with instructions that specify actions to be performed by the machine. As used herein, the term "server" generally refers to a computer or processor that performs processing for and sends data and information to another computer or processor.

[0079] A computer system is typically structured in part with at least one operating system program, such as Microsoft's Windows, Sun Microsystems' Solaris, Apple Computer's MacOs and iOS, Google's Android, Linux, and / or Unix. A computer system typically includes a basic input / output system (BIOS) and processor firmware. The operating system, BIOS, and firmware are used by the processor to structure and control any subsystems and interfaces connected to the processor. Typical processors that enable these operating systems include Pentium, Itanium, and Xeon processors from Intel, Opteron and Athlon processors from Advanced Micro Devices, Graviton processors from Amazon, POWER processors from IBM, SPARC processors from Oracle, and ARM processors from ARM Holdings.

[0080] Any ECIN is not limited to electronic digital logic computers or electronically programmable devices structured with a program. For example, the claimed invention may use optical computers, quantum computers, analog computers, and the like. Furthermore, where only one computer system or single machine is referred to, the use of the singular form of such terms may also refer to any structure of computer systems or machines that use processing individually or jointly. Due to the ever-changing nature of computers and networks, the description of computer system 500 shown in FIG. 11 is only an example. Many other structures of computer system 500 have more or fewer components than the computer system shown in FIG. 11.

[0081] The network interface subsystem 516 provides an interface to external networks, including an interface to a communications network 518, and is coupled via the communications network 518 to corresponding interface devices in other computer systems or machines. The communications network 518 comprises many interconnected computer systems, machines, and physical communications connections (denoted by "links"). These communications links may be wireline links, optical links, wireless links (e.g., using WiFi or Bluetooth protocols), or any other physical device for communication of information. The communications network 518 may be any suitable computer network, for example, a wide area network such as the Internet and / or a local to wide area network such as Ethernet. The communications network may be wired and / or wireless, and many communications networks use encryption and decryption processes such as are available along with virtual private networks. The communications network uses one or more communications interfaces, which receive data from and transmit data to other systems. Embodiments of the communications interfaces generally include Ethernet cards, modems (e.g., telephone, satellite, cable, or ISDN), (asynchronous) digital subscriber line (DSL) units, Firewire interfaces, USB interfaces, and the like. Communications algorithms ("protocols") may be specified using one or more communications languages, such as HTTP, TCP / IP, RTP / RTSP, IPX, and / or UDP.

[0082] The user interface input devices 512 may include alphanumeric keyboards, keypads, pointing devices such as mice, trackballs, toggle switches, touchpads, styluses, graphics tablets, optical scanners such as barcode readers, touch screen electronics for display devices, audio input devices such as voice recognition systems or microphones, eye gaze recognition, brainwave pattern recognition, optical character recognition systems, and other types of input devices. Such devices are connected to the computer system by wires or wirelessly. In general, the term "input device" refers to all possible types of devices and processes for transferring data and information to the computer system 500 or over the communications network 518. User interface input devices generally allow a user to select objects, icons, text, etc. that appear on some type of user interface output device, e.g., a display subsystem.

[0083] The user interface output devices 514 may include a display subsystem, a printer, a fax machine, or non-visual communication devices such as audio and tactile devices. The display subsystem may include a flat panel device such as a cathode ray tube (CRT), a liquid crystal display (LCD), an image projection device, or some other device for creating visual stimuli such as a virtual reality system. The display subsystem may also provide non-visual stimuli such as through audio output, aroma generation, or haptic / tactile output (e.g., vibration and force) devices. In general, the term "output device" refers to all possible types of devices and processes for transferring data and information from within the computer system 500 to a user or another machine or computer system. Such devices are connected to the computer system by wires or wirelessly. Note: Some devices transfer data and information both into and out of the computer, for example, haptic devices generate vibrations and forces on the user's hand and also incorporate sensors for measuring the location and movement of the hand. Technical applications of the science of ergonomics and semiotics are used to improve the efficiency of user interaction with any process and computer disclosed herein, such as any interaction related to the design and manufacture of circuits, that uses any of the input or output devices listed above.

[0084] The memory subsystem 508 typically includes several memories, including a main random access memory ("RAM") 520 (or other volatile storage device) for storage of instructions and data during execution of the program, and a read-only memory ("ROM") 522 in which fixed instructions are stored. The file storage subsystem 510 provides persistent storage for program and data files and may include a hard disk drive, a floppy disk drive with associated removable media, a CD-ROM drive, an optical drive, a flash memory such as a USB drive, or a removable media cartridge. If the computer system 500 includes an input device that performs optical character recognition, text and symbols printed on paper may be used as a device for storage of program and data files. Databases and modules used by some embodiments may be stored by the file storage subsystem 510.

[0085] Bus subsystem 504 provides a device for transmitting data and information between various components and subsystems of computer system 500. Although bus subsystem 504 is shown as a single bus, alternative embodiments of the bus subsystem may use multiple buses. For example, main memory using RAM may communicate directly with a file storage system using a direct memory access ("DMA") system. Detailed Description-Semantic Support The signifier "commercial solution" simply denotes, for the following paragraphs, a technology domain-specific (hence non-preemptive--see Bilski) electronic structure, process for a specified machine, manufacturable circuit (and its Church-Turing equivalent), or composition of matter that applies science and / or technology for use in the trade to solve an unsatisfied need in the art.

[0086] The signifier "abstract" (when used in a patent claim for any operative embodiment disclosed herein for a new commercial solution that is the scientific use of one or more laws of nature (see Benson), that solves a problem in the art for use in commerce (see Diehr) or that improves an existing solution used in commerce (see Diehr)) is precisely defined by the inventor as follows (see MPEP2111.01 (9th edition, Rev.08.2017)):

[0087] a) a new commercial solution is "abstract" if it is not novel (e.g., it is well known in the equivalent prior art (see Alice) and / or the use of the equivalent prior art solution has long been widespread in science, engineering or trade (see Bilski)) and therefore unpatentable under 35 U.S.C. §102, for example, because it is "difficult to understand" how the commercial solution differs from the equivalent prior art solution (see Merriam-Webster definition of "abstract"), or b) A new commercial solution is "abstract" and therefore unpatentable under 35 U.S.C. § 103 if the existing prior art contains at least one similar prior art solution (see KSR) or at least two prior art disclosures that could be combined by a person skilled in the art (often called "PHOSITA", see MPEP 2141-2144 (9th edition, Rev. 08.2017)) to become equivalent to the new commercial solution (see Alice), for example, because it is "difficult to understand" how the new commercial solution differs from the PHOSITA combination / application of the existing prior art; or c) A new commercial solution is "abstract" if it is not disclosed with a description enabling its practice, either because there is insufficient guidance in the description or because only a general implementation is described without specifying components, parameters, or functions (see Mayo), and thus PHOSITA is unable to instantiate, for example, an embodiment of the new solution for use in commerce without the need for special programming (see Kaz) (or, for example, circuit design) to be performed by PHOSITA, and thus is unpatentable under 35 U.S.C. § 112, because, for example, it is "difficult to understand" how any embodiment of the new commercial solution should be used in commerce. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS - CONCLUSION The detailed description refers to each individual feature, structure, function, or characteristic described herein, or any combination thereof, without limitation to the claims, to the extent that such feature, structure, function, or characteristic, or combination thereof, solves any problem disclosed herein, to the extent that such feature, structure, function, or characteristic, or combination thereof, is enabled by the detailed description as a whole in light of the knowledge and understanding of a person skilled in the art. When an ECIN comprises a particular feature, structure, function, or characteristic, it is within the knowledge and understanding of a person skilled in the art to use such feature, structure, function, or characteristic with respect to another ECIN, for example, as a substitute for another feature, structure, function, or characteristic, whether or not explicitly described.

[0088] In view of the detailed description, one skilled in the art will understand that many variations of any ECIN may be possible, such as the function and structure of elements described herein, while still being as useful as the ECIN. One or more elements of an ECIN may be replaced by one or more elements in another ECIN, as would be understood by one skilled in the art. The description of any ECIN indicates its use in a transaction, thereby enabling one skilled in the art to use this ECIN in a transaction as well.

[0089] The detailed description of the invention has been adequately written to provide knowledge and understanding. It is not exhaustive or limited to the exact structure described, but should be given the widest scope consistent with the principles and features disclosed. Without limitation, any and all equivalents described, shown, or incorporated by reference (or explicitly incorporated) in this patent application are specifically incorporated in the detailed description. Furthermore, any and all variations described, shown, or incorporated with respect to any one ECIN may also be included with any other ECIN. Any such variations include currently known variations as well as future variations, for example, any element used for enablement includes future equivalent elements that provide the same function, regardless of the structure of the future equivalent elements.

Claims

1. Incorporating a balanced binary tree structure into a multiplication module of a tensor processor to more efficiently execute sequences of instructions for stream reduction operations. The method includes:

2. The method of claim 1 , wherein the incorporating step comprises canceling delays of partial aggregation logic in the balanced binary tree structure.

3. The method of claim 1 , wherein the incorporating step comprises canceling delay dependencies in aggregation logic.

4. 4. The method of claim 3, wherein canceling the delay dependency comprises eliminating the need to stagger data streams according to an aggregation delay.

5. The method of claim 3 , wherein the step of canceling the delay dependencies comprises the step of unifying the stream reduction operations.

6. 2. The method of claim 1, wherein the step of incorporating the balanced binary tree structure comprises enabling a compiler to locally optimize aggregation operations.

7. The method of claim 1 , wherein the step of incorporating the balanced binary tree structure comprises the step of enabling the balanced binary tree structure with the same unit logic design for a set of superlanes.

8. The method of claim 7 , wherein the step of enabling the balanced binary tree structure comprises operatively connecting two adjacent superlanes of the set of superlanes.

9. The method of claim 7 , wherein the enabling of the balanced binary tree structure comprises operatively connecting partial aggregation results of the set of superlanes.

10. Routing a first partial result of the partial aggregation results for a first set of superlanes of the set of superlanes towards a second partial result of the partial aggregation results for a second set of superlanes of the set of superlanes. The method of claim 9 further comprising:

11. delaying the second partial result by a defined number of cycles determined to match the latency for the first partial result. The method of claim 10 further comprising:

12. The method of claim 1 , wherein the balanced binary tree structure includes at least four superlanes.

13. A compiler that incorporates a balanced binary tree into the multiplication module of a tensor processor to more efficiently execute sequences of instructions for stream reduction operations. A system comprising:

14. 14. The system of claim 13, wherein the compiler adds a set of multiplexers at locations determined based on the location of the superlanes.

15. 15. The system of claim 14, wherein select signals for the set of multiplexers are controlled by a group of configuration registers.

16. The system of claim 13 , wherein the compiler provides a unit design for partial aggregation logic.

17. 20. The system of claim 16, wherein the unit design for the partial aggregation logic comprises a multi-entry delay buffer, a first set of routing channels, and a second set of routing channels.

18. 14. The system of claim 13, wherein the compiler enables the balanced binary trees with the same unit logic design for a set of superlanes.

19. 20. The system of claim 18, wherein the compiler operatively connects two adjacent superlanes of the set of superlanes.

20. The system of claim 13 , wherein the balanced binary tree comprises at least four superlanes.

Images