Balanced binary tree structure for stream reduction operations

Abstract

To provide methods, systems and other embodiments for incorporating a balanced binary tree into multiplication modules of a tensor processor so as to execute sequences of instructions more efficiently for stream-reducing operations; where this abstract and the independent claims are concise signifiers of embodiments of the claimed inventions, and the abstract does not limit the scope of the claimed inventions.SOLUTION: In one or more ECINs disclosed herein, balanced binary tree structures are disclosed that enable a compiler to locally optimize an aggregation operation for the timing / area as needed without having an impact on the overall architecture.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] This disclosure has general significance in the field of architectures for high-performance processors, and in particular has significance with respect to the subject of advantageous interconnections of functional blocks in high-performance processors. This information is limited to use in the search of the prior art.

Background Art

[0002] This patent document may be reproduced exactly as it appears in the application of the United States Patent and Trademark Office, but otherwise, the assignee reserves all rights in any subset of the author's works contained in this specification that are protected by Section 102(a) of Title 17 of the United States Code, the United States Copyright Law.

[0003] In the following background art, summary, and modes for carrying out the invention, paragraph headings are signifiers that do not limit the scope of the claimed embodiments (ECINs). Any publication citation or identification does not indicate relevance to or use of prior art. Paragraphs in all italics (represented by underlining herein) indicate text present in one or more patent specifications filed by the assignee.

[0004] Statements enclosed in double quotation marks ('') indicate an exact copy of the statement as it appears in the copyrighted work. Signifiers such as words or phrases enclosed in single quotation marks ("") indicate terms that are not yet defined, do not have meaning to be evaluated, or will not have meaning in their particular use (for example, when the quoted term "module" is first used) until they are defined.

[0005] This application claims the benefit of priority of U.S. Provisional Application No. 63 / 584,353, filed on September 21, 2023, entitled "BALANCED BINARY TREE STRUCTURES FOR STREAM REDUCING OPERATIONS", which is hereby incorporated by reference in its entirety.

[0006] A tensor is a family of mathematical structures that includes vectors, matrices, and higher-dimensional arrays. Tensors are used in many fields of science and engineering, and huge tensors with millions to billions of elements are used in numerical calculations such as machine learning. One operation - multiplication requires a huge amount of processing power for large tensors, and special processors have been developed in recent years.

[0007] One type of tensor processor is deterministic (where the time and location of execution for every instruction are known before execution), such as the Tensor Streaming Processor (TSP) sold by Groq Incorporated. These types of deterministic processors have a two-dimensional mesh of processor cores, where data flows across lanes and instructions flow across slices. This is generally illustrated in Figure 3.

[0008] In this tensor processor configuration, each computational element implements a specific function and is stacked vertically in a specific "function slice" in one dimension (e.g., Y dimension) of a two-dimensional on-chip mesh. Each function slice is independently controlled by a set of instructions specific to its on-chip role. For example, a MEM (memory) function slice supports reading and writing, but does not necessarily support Add (addition) or Mul (multiplication), which are generally implemented in arithmetic function slices (e.g., vector execution module (VXM) and matrix execution module (MXM) function slices) for several typical machine learning (ML) algorithms such as linear regression algorithms. In dimension X, each function row comprises a complete set of different types of function cores, such as MEM, VXM, MXM, SXM (exchange and substitution modules). Each function row is referred to herein as a superlane.

[0009] In the TSP architecture, data flows in the X dimension and instructions flow in the Y dimension. Function slices act on data streams using a single-instruction multiplexed data (SIMD) scheme, where different data streams are processed independently in different superlanes of the same function slice using the same instruction sequence. Physically, it takes time for an instruction to move from one superlane to the next. As a result, data streams in different superlanes must arrive at the function slice with a staggering delay. For example, with a staggering delay of one clock cycle, the data stream arrives at superlane 0 at time t, at superlane 1 at time t+1, and so on. If the staggering delay is 2, the data stream arrives at superlane 0 at time t, at superlane 1 at time t+2, at superlane 2 at time t+4, and so on.

[0010] The data stream's lag delay is determined by the need for it to arrive in time to execute its intended instructions precisely within each function slice. Conversely, the data stream's lag delay is determined by the traverse delay of the instruction sequence that generates the data stream. For example, a sequence of memory read instructions traversing a MEM slice with a one-clock-cycle delay moving from one superlane to the next will generate a data stream with a one-clock-cycle lag delay. Naturally, different function slices must have the same lag delay to fit together.

[0011] Traditional machine learning (ML) and high-performance computing (HPC) applications involve specific operations that perform aggregation on multiple data streams to produce fewer streams or a single result stream. We refer to such operations that consume multiple input streams and produce fewer output streams as stream reduction operations. Unlike typical SIMD operations where each function core operates on x input streams and produces x output streams in the same dimension (dimension X), stream reduction operations require changing the direction of data flow from dimension X to dimension Y for aggregation, and then changing it back from dimension Y to dimension X for writing back a single result stream.

[0012] For example, in matrix multiplication C = A·B, the element cij in the resulting matrix C is the sum of s multiplications between the elements of the i-th row of matrix A and the j-th column of matrix B, for example,

[0013]

number

[0014] Here, s is the number of columns in matrix A or the number of rows in matrix B. To compute matrix multiplication on the TSP, the operation (aix × bxj) is performed in parallel across s / x different superlanes, where each superlane handles the multiplication of x elements. Each resulting element cij is computed by the aggregation of the results of the multiplications from s / x superlanes and then written back to a single result stream that exists across i / x superlanes.

[0015] Stream reduction operations are essential for both ML and HPC applications. Due to their differences from typical SIMD operations, others fail to efficiently implement these operations in deterministic streaming processors and therefore cannot provide solutions to at least the following three problems.

[0016] 1) Completely manage the dependencies between the completion times of operations in the X and Y dimensions. Stream reduction operations in each superlane require input streams from the X and Y dimensions to synchronize with each other. This requires efficiently shifting the input data streams with a delay determined by the partial aggregation logic in each superlane, in addition to the instruction crossover time through the superlanes.

[0017] 2) Different stream reduction operations can result in different delays, determined by the required aggregation logic, which in turn determines the data stream's misalignment delay. Therefore, this adds an additional challenge to support multiple stream reduction operations at the architecture level in a unified manner.

[0018] 3) Stream reduction increases the wiring requirements for the slice through which the instruction flows. This is because the partial aggregation result needs to be passed from one superlane to its adjacent superlanes. Furthermore, the final result of the stream reduction operation needs to be written back to the target superlane based on the index.

[0019] What others have failed to provide is a more efficient way to generate a sequence of instructions that optimize stream reduction for tensor processors. [Overview of the project] [Problems that the invention aims to solve]

[0020] The outline of the present invention, along with any claims, is a simple set of signifiers for at least one ECIN (which may be a discovery; see 35 § 100(a) and 35 § 100(j)) for use in a transaction satisfying 35 U.S.C. § 112.

[0021] In one or more ECINs disclosed herein, a balanced binary tree structure is disclosed and incorporated into the multiplication module of a tensor processor to address previously unsolved problems in efficiently executing instructions for stream reduction operations. In particular, a balanced binary tree structure is disclosed that can be used, advantageously, to eliminate delay dependencies in the aggregation logic and remove the need to further shift the data stream according to the aggregation delay. Thus, different stream reduction operations can be unified.

Means for Solving the Problems

[0022] In one or more ECINs disclosed herein, a balanced binary tree structure is disclosed that allows a compiler to locally optimize the aggregation operation for timing / area as needed without affecting the entire architecture.

Advantages of the Invention

[0023] In one or more ECINs disclosed herein, a balanced binary tree structure with the same unit logic design for all super lanes is enabled, thereby enabling more efficient design scalability of the TSP.

[0024] The summary of this invention does not fully represent any ECIN. The summary of this invention may represent at least one essential element of an ECIN made possible by this specification and drawings, but the summary of this invention does not represent any scope or limitation of any ECIN.

Brief Description of the Drawings

[0025] The following embodiments, drawings, and claims illustrate the use of one or more ECINs and the advances enabled by them. All drawings are for knowledge and understanding purposes only and do not limit the scope of any ECIN. Such drawings are not necessarily drawn to a specific scale.

[0026] Drawings may have the same or similar reference signifiers in the form of labels (alphanumeric symbols, such as reference numbers) that can indicate equivalent or equivalent functions or uses. Furthermore, reference signifiers of the same type may be distinguished by attaching a dash and a second label to the reference label to distinguish similar signifiers. Where only the first label is used herein, its use applies to any similar component having the same label, regardless of any other reference labels. A brief list of drawings is provided below.

[0027] [Figure 1] FIG. shows a system for compiling a program to be executed on a tensor processor according to one embodiment. [Figure 2A] FIG. shows the instructions and data flow in a processor having a function slice architecture according to some embodiments. [Figure 2B] This figure shows the instruction and data flow in a processor having a function slice architecture, according to several embodiments. [Figure 3] This figure shows the instruction and data flow in a processor having a function slice architecture, according to several embodiments. [Figure 4] This figure shows MXM slices and daisy-chain circuits incorporated into MXM slices according to several embodiments. [Figure 5] This figure shows an MXM slice and a balanced binary tree circuit incorporated into the MXM slice, according to several embodiments. [Figure 6] This figure shows one unit of a balanced binary tree circuit according to several embodiments. [Figure 7] This figure shows an unbalanced binary tree circuit according to several embodiments. [Figure 8A] This figure shows a balanced binary tree with four superlanes, according to several embodiments. [Figure 8B] This figure shows a balanced binary tree with four superlanes, according to several embodiments. [Figure 8C] This figure shows a balanced binary tree with four superlanes, according to several embodiments. [Figure 9] This figure shows a balanced binary tree with eight superlanes, according to several embodiments. [Figure 10] This figure shows a balanced binary tree with 16 superlanes, according to several embodiments. [Figure 11] This figure shows a computer system suitable for enabling embodiments of the claimed invention.

[0028] In the figures, reference numerals may be omitted to conform to acceptable engineering practice, but those skilled in the art will understand that the components illustrated are to be understood in the context of the figure as a whole, the context of the accompanying description relating to such figure, and the context of the claimed embodiments of the invention. [Modes for carrying out the invention]

[0029] The drawings and embodiments for carrying out the invention show at least one ECIN for the sole purpose of providing knowledge and understanding. To minimize the length of the embodiments for carrying out the invention, various features, structures or properties may be described together in a single embodiment, but they may still be used in other embodiments without being described. Modifications of any of these elements, as well as modules, processes, machines, systems, manufactures, or compositions disclosed by such embodiments and / or examples, may be readily used in trade. The drawings and embodiments for carrying out the invention implicitly or explicitly indicate the advantages and improvements of at least one ECIN for use in trade.

[0030] In the drawings and modes for carrying out the invention, numerous specific details may be described to enable at least one ECIN. Any embodiment disclosed herein represents a tangible form of the claimed invention. In order not to diminish the significance of the embodiments and / or examples in the modes for carrying out the invention, some elements known to those skilled in the art may be combined for presentation and illustrative purposes and not specified in detail. In order not to diminish the significance of these embodiments and / or examples, some well-known processes, machines, systems, manufactures, or compositions are not described in detail. However, those skilled in the art may use these embodiments and / or examples in trade without these specific details or their equivalents. Thus, the modes for carrying out the invention focus on enabling elements of the invention in some ECIN. Where a mode for carrying out the invention refers to several elements in a single form, two or more elements may be shown in the drawings, and similar elements will be marked with similar numbers.

[0031] Figure 1 shows a system 100 for compiling a program to be executed on a tensor processor, according to one embodiment. The system may also be used, according to several embodiments, to generate power usage information for the compiled program. System 100 includes a user device 102, a server 110, and a processor 120. Each of these components and their subcomponents (if any) are described in more detail below. While this specification describes specific configurations of the components, in other embodiments, system 100 may have different components, and these components may perform the functions of system 100 in a different order or using different mechanisms. For example, Figure 1 shows a single server 110, but in other embodiments, the compilation, assembly, and power usage functions may be performed on different devices. For example, in some embodiments, at least a portion of the functions performed by the server 110 may be performed by the user device 102.

[0032] The user device 102 comprises some electronic computing device, such as a personal computer, laptop, or workstation, which uses an Application Programming Interface (API) 104 to construct a program that will run on the processor 120. The server 110 receives a program specified by the user on the user device 102 and compiles the program to produce a compiled program 114. In some embodiments, the compiled program 114 enables a data model for prediction that processes input data and makes predictions from the input data. Examples of predictions are categorical classification or prediction of time series values ​​performed using a classifier. In some embodiments, the prediction model describes a machine learning model that includes nodes, tensors, and weights. In one embodiment, the prediction model is specified as a TensorFlow model, the compiler 112 is a TensorFlow compiler, and the processor 120 is a tensor processor. In another embodiment, the prediction model is specified as a PyTorch model, and the compiler is a PyTorch compiler. In other embodiments, other machine learning specification languages ​​and compilers are used. For example, in some embodiments, a predictive model defines nodes representing operators (e.g., arithmetic operators, matrix transformation operators, Boolean operators, etc.), tensors representing operands (e.g., values ​​modified by the operators, such as scalar values, vector values, and matrix values, which can be represented in integer or floating-point form), and weight values ​​generated after training and stored in the model. In some embodiments, if the processor 120 is a tensor processor with a function slice architecture, the compiler 112 translates the program into a set of operations to be performed by the processor 120 and generates an explicit plan of how the processor will execute the program by specifying when each instruction will be executed, which function slices will perform the work, and which stream registers will hold the operands. This type of scheduling is known as "deterministic scheduling."This explicit plan for execution includes information about explicit stream reduction processing using a balanced binary tree architecture.

[0033] The assembler 116 receives the compiled program 114 generated by the compiler 112, performs the final compilation and linking of the scheduled instructions, and produces a compiled binary. In some embodiments, the assembler 116 maps the scheduled instructions shown in the compiled program 114 to the server 110 hardware and then determines the exact component queue in which each instruction will be placed.

[0034] The processor 120 is a hardware device having a large number of matrix multiplier units that accept a compiled set of instruction bits (e.g., a "binary") assembled by, for example, assembler 116, and execute the instructions contained in the compiled binary. The processor 120 may include one or more blocks of circuitry for matrix arithmetic, numerical transformations, vector calculations, short-term memory, and data substitution / exchange. Once such a processor 120 is a tensor processor having a function slice architecture. In some embodiments, the processor 120 comprises multiple tensor processors connected together to function as a single deterministic component. Exemplary processor Figures 2A and 2B show instruction and data flows in a processor having a function slice architecture according to several embodiments. One feasibility of processor 200 is as an application-specific integrated circuit (ASIC), corresponding to processor 120 shown in Figure 1.

[0035] The function units of the processor 200 (also called "function tiles") are aggregated in some embodiments into multiple function process units 205, each corresponding to a specific function type (hereinafter referred to as "slice"). For example, different function slices of the processor correspond to processing units for MEM (memory), VXM (vector execution module), MXM (matrix execution module), NIM (numerical interpretation module), and SXM (exchange and substitution module). In other embodiments, each tile may include an aggregation of function units, such as a tile having both MEM units and execution units, for example. As shown in Figures 2A and 2B, each slice corresponds to a column of N function units extending in a direction different from the direction of data flow (e.g., orthogonal). The function units of each slice may share an instruction queue (not shown) for storing instructions and an instruction control unit (ICU) 210 for controlling the instruction execution flow. Instructions in each instruction queue are executed only by the function units in the associated slice of the queue and not by other slices of the processor. In other embodiments, each function unit has an associated ICU for controlling the instruction execution flow.

[0036] The processor 200 also includes communication lanes for transporting data between function units in different slices. Each communication lane connects to each of the slices 205 of the processor 200. In some embodiments, a communication lane 220 connecting rows of function units in adjacent slices is called a “superlane” and comprises multiple data lanes or “streams” configured to transport data values ​​along a specific direction. For example, in some embodiments, each function unit of the processor 200 is connected to a corresponding function unit on an adjacent slice by a superlane consisting of multiple lanes. In other embodiments, the processor 200 includes a communication device, such as a router, for transporting data between adjacent function units.

[0037] By configuring the function units of the processor 200 in different function slices 205, the on-chip instruction and control flow of the processor 200 is separated from the data flow. Since many types of data are acted upon by the same set of instructions, it is important for visualization to visualize the instruction flow rather than the data flow. In some embodiments, Figure 2A shows the instruction flow within the processor architecture, while Figure 2B shows the data flow within the processor architecture. As shown in Figures 2A and 2B, instructions and control signals flow in a first direction across the function units of the processor 200 (e.g., along the length of the function slice 205), while the data flow flows in a second direction across the function units of the processor 200 (e.g., across the function slices) that are not parallel to the first direction, via communication lanes (e.g., superlanes) connecting the slices.

[0038] In some embodiments, function units within the same slice execute instructions in a "staggered" manner, where instructions are issued tile by tile within the slice over a period of N cycles. For example, the ICU for a given slice issues an instruction during a first clock cycle to the first tile of the slice (e.g., the tile at the bottom of the slice shown in Figure 2B, closest to the ICU of the slice), which is passed to subsequent function units of the slice over subsequent cycles. That is, each row of function units in the processor 200 (corresponding to function units along a particular superlane) executes the same set of instructions, but is time-offset with respect to the function units in adjacent rows.

[0039] The processor's function slices are configured such that operand data read from memory slices is intercepted by different function slices as data moves across the chip, and the results flow in the opposite direction, where they are written back to memory. For example, a first data flow from a first memory slice flows in a first direction (e.g., to the right), where it is intercepted by a VXM slice that performs a vector operation on the received data. The data flow then follows an MXM slice that performs a matrix operation on the received data. The processed data then flows in a second direction opposite to the first direction (e.g., to the left), where it is again intercepted by a VXM slice to perform an accumulation operation, and then written back to the memory slice.

[0040] In some embodiments, the processor's function slices are configured such that data flow between memory and the function slice occurs in both a first and a second direction. For example, a second data flow originating from a second memory slice moves in a second direction toward the second slice, where the data is intercepted and processed by the VXM slice before proceeding to the second MXM slice. The result of the matrix operation performed by the second MXM slice then flows back toward the second memory slice in the first direction.

[0041] In some embodiments, stream registers are located along the processor's superlanes, according to some embodiments. Stream registers are located between the processor's function slices to facilitate the transport of data (e.g., operands and results) along each superlane. For example, within the processor's memory area, stream registers are located between sets of four MEM units. Stream registers act as primary hardware structures that are structurally visible to the compiler and have a view to program execution. Each function unit in the set includes a stream circuit configured to allow the function unit to read or write stream registers in either direction of the superlane. In some embodiments, each stream register is implemented as a set of registers corresponding to each stream in the superlane, sized based on the underlying data type used by the processor (e.g., if the underlying data type of the TSP is INT8, each register may be 8 bits wide). In some embodiments, to support larger operands (e.g., FP16 or INT32), multiple registers are treated collectively as a single operand, where the operand is transmitted across multiple streams in the superlane.

[0042] The function features—superlanes of function units, slices of instruction flow, and processing of different types of integers and floating-point numbers—all occurring trillions of times per second, create complex power flows and possible destructive power fluctuations that can negatively impact processor performance. However, assuming a deterministic nature to execution by the processor, any destructive power fluctuations (such as voltage drops) can be determined before program execution, and information about such fluctuations (such as processor instructions and the timing for such instructions) is supplied to the processor by the compiler for use during program execution to mitigate the fluctuations. Stream reduction A stream reduction instruction can be considered to have two sub-operations ('sub-ops'): distributed Y-dimensional aggregation and Y-dimensional result write-back. Distributed Y-dimensional aggregation performs partial aggregation in each superlane. This sub-op has timing dependencies on both the X and Y-dimensional inputs, meaning that the operands from the X and Y dimensions must be precisely related to each other to produce the correct partial aggregation result. Y-dimensional result write-back writes the final aggregation result to the target superlane via a dedicated bus for the Y dimensions arriving in all superlanes.

[0043] As explained above, the main challenge in stream reduction operations arises from aggregation into the Y dimension. A simple implementation involves daisy-chaining n superlanes sequentially from 0 to n-1. An example of a daisy-chain incorporated into an MXM slice is shown in Figure 4. The advantages of such an implementation include: 1) The partial aggregation logic in superlane i only needs to consider superlanes i and (i-1) for the input operands, the logic is independent of the superlane number / ID and therefore only needs to be implemented once and replicated for all superlanes. 2) The output of the partial aggregation logic only needs to be routed to the next superlane. The partial result does not need to traverse multiple superlanes. This reduces the need for routing resources for the Y dimension.

[0044] However, the following drawbacks have a more significant impact on the TSP architecture: 1) The delay of the partial aggregation logic must match the data stream misalignment delay d. Therefore, the partial aggregation logic must be completed within a delay of d, or the data stream misalignment delay must be adjusted according to the needs of each stream reduction instruction, along with the instruction propagation path of the function slice, which can be costly for the entire TSP chip; 2) Input data must pass through a longer chain of aggregation logic, which can lead to loss of precision. For example, input from super lane 0 must pass through the entire chain of n partial aggregation logic elements. Balanced binary tree Figure 5 shows an MXM slice and a balanced binary tree circuit incorporated into the MXM slice for the case of 16 superlanes. In a daisy-chain structure, the arrival time of an operand in superlane 1 must match the partial aggregation output from superlane 0, which is determined by the delay of the partial aggregation logic d. In other words, an operand in superlane i must arrive at time i*d. Since the delay d is involved in determining the arrival time of the operand, this means that the architecture is dependent on the implementation form of the partial aggregation logic; for example, when the implementation form of the partial aggregation logic changes, other parts of the design must also change. Or, if the architecture remains the same, the partial aggregation logic must satisfy delay constraints imposed by the data stream drift delay.

[0045] In contrast, the delay of partial aggregation logic is eliminated in a balanced binary tree structure. For 16 superlanes, there are four layers of partial aggregation logic from the leaf node to the root node, where the root node generates the final aggregation result. In the first layer adjacent to the leaf node, the arrival times of the operands are independent of the aggregation logic. Assuming that the data arrives with a one-cycle delay, for example, that the data in superlane i arrives in cycle i, the partial aggregation logic in the first layer only needs to insert a one-cycle delay for the operand coming from the lower superlane. In the second layer, the arrival times of the two operands are 2*j+1+d and 2*j+3+d, respectively, for the j-th partial aggregation logic in the second layer. Thus, the aggregation logic only needs to insert a two-cycle delay for the lower operand. It should be noted that the delay d of the partial aggregation logic from the previous layer is erased between the two operands that feed into the same partial aggregation logic, and the number of delay buffers required is simply determined by the lag delay of the initial data stream. This continues as the operands are processed by a balanced binary tree structure, and the partial aggregation logic needs to insert a 4-cycle delay for its lower operands in the third layer and an 8-cycle delay in the fourth layer.

[0046] The delay per layer is the cumulative delay from the previous layer plus the delay from the current layer. Therefore, the delay buffering added to each layer does not increase as the data moves further down through the layers. Furthermore, the delay at each layer in the binary tree is the same because it is a balanced binary tree.

[0047] The advantages of a balanced binary tree structure are as follows: The delay of partial aggregation logic does not affect other parts of the design or impose data stream misalignment delays. In other words, there are fewer constraints on how the aggregation logic must be implemented. Multiple stream reduction instructions can be supported without a significant impact on the overall architecture or data misalignment delay. The data traverses fewer stages of partial aggregation to reach the final result, which can lead to better accuracy. It requires less partial aggregation logic than daisy-chained structures when the number of superlanes is greater than 2.

[0048] The drawbacks are as follows: It becomes difficult to divide a balanced binary tree into different superlanes and make them identical. In the case of TSP, it is important to keep all superlanes identical to reduce both logical and physical design costs. The output of partial aggregation logic may need to be routed to a non-adjacent superlane depending on its location in the binary tree. This increases routing traffic between different superlanes. The number of superlanes is preferably a power of 2; otherwise, the delay of partial aggregation logic cannot be completely eliminated. If the number of superlanes is not a power of 2, the additional buffering capacity required increases by log2 depending on the depth of the tree.

[0049] Figure 6 shows one unit of a balanced binary tree circuit that is similarly incorporated into each superlane. In this configurable logic unit design, a set of multiplexers is added at each location, where the design may differ depending on the location of the superlane. The modified design can use different signal routing or different delay paths. The selection signals of those multiplexers are controlled by a group of configuration registers, which are set by software at initialization time.

[0050] Figure 7 shows an example of an unbalanced binary tree circuit using five superlanes. The aggregation of superlanes 0-3 is well balanced, while the aggregation of superlane 4 with the partial results from 0-3 is not balanced with respect to branch depth or delay. Therefore, superlane 4 needs to insert a buffer with a latency of 2d-1 (assuming d≧1) to match the arrival times of the partial results from superlanes 0-3. The latency of the inserted buffer now depends on the delay d of the partial aggregation logic. Thus, the number of superlanes must be a power of 2 in favor of the balanced binary tree structure to completely cancel out the effects of the partial aggregation logic delay.

[0051] Figures 8A to 8C show a balanced binary tree with four superlanes. In a balanced binary tree structure, it is difficult to divide the entire aggregation tree into different superlanes while maintaining the same logic in each superlane. To address this problem, a unit design is used for each superlane and for the connection scheme between different superlanes, as shown in Figures 8A to 8C.

[0052] Figure 8A shows a unit design for partial aggregation logic, which consists of aggregation logic, a four-entry delay buffer, three routing channels (indicated by diagonal rectangles), and three other routing channels (indicated by dotted rectangles). Local input data is represented as black circles.

[0053] Figure 8B shows how two adjacent superlanes (of a set of superlanes) must be connected. Assume that the input data of each superlane is shifted by a one-cycle delay. Local inputs from the higher superlane are routed to the lower superlane. Note that routing across the superlane boundary in the Y dimension adds a latency of one clock cycle. To match the input latency, local inputs from the lower superlane must pass through a two-cycle buffer before being fed into the aggregation logic. The partial aggregation result is generated at time 2+d. Note that the aggregation logic in the upper superlane remains unused. This connection scheme applies to all pairs of two adjacent superlanes.

[0054] Figure 8C illustrates how partial aggregation results for four superlanes (e.g., a set of superlanes) should be connected. Partial results from the upper pair of superlanes (e.g., the first set of superlanes in a set of superlanes) are routed to partial results from the lower pair of superlanes (e.g., the second set of superlanes in a set of superlanes) (e.g., the second partial result of the partial aggregation results). The two partial results are aggregated in superlane 1, where the partial results from the lower pair are delayed by two cycles (e.g., a defined number of cycles, which may be more or less than two) to match the latency for the input from the upper pair. The final partial results for the four superlanes are generated at time 5+2d. This connection scheme applies to all pairs of four adjacent superlanes. Note that this scheme leaves one unused aggregation logic block in superlane 3.

[0055] Figure 9 shows a balanced binary tree with eight superlanes, illustrating how the connection scheme should be extended to progress from four superlanes to eight. In the case of eight superlane connections, two sets of four superlanes are stacked together, and partial results from the four superlanes are routed from superlane 5 to superlane 3. Similarly, partial results from superlane 1 are routed to superlane 3. Since the partial results from the upper four superlanes are four clock cycles later than those from the lower four superlanes, a four-cycle delay buffer is inserted for the lower inputs in superlane 3.

[0056] Figure 10 shows a balanced binary tree with 16 superlanes, illustrating how the connection scheme should be extended to progress from 8 superlanes to 16 superlanes. The 16 superlane connection can be achieved by stacking two connected 8 superlanes and routing the partial results to either superlane 7 or 15. If the total number of superlanes in the TSP is 16, the partial results must be routed to superlane 15, and thus the final aggregation results can be distributed back to each superlane using a single bus, as shown in Figure 6. Note that no additional delay buffer is required in superlane 15 for lower input operands of the partial aggregation logic. The delay coincides when the partial results from the lower 8 superlanes satisfy the partial results from the upper 8 superlanes.

[0057] This specification provides a balanced binary tree structure for stream reduction operations. Embodiments relate to a method for incorporating the balanced binary tree structure into a multiplication module of a tensor processor to execute a sequence of instructions more efficiently for stream reduction operations. In one implementation, the balanced binary tree structure may include at least four superlanes. However, in other implementations, the balanced binary tree structure may include any number of superlanes.

[0058] In some implementations, incorporating a balanced binary tree structure into the tensor processor's multiplication module may include canceling the delay in the partial aggregation logic within the balanced binary tree structure. In some implementations, this may also include canceling the delay dependency in the aggregation logic. For example, canceling the delay dependency may eliminate the need to shift the data stream according to the aggregation delay. In another example, the delay dependency may include unifying stream reduction operations.

[0059] In some implementations, incorporating a balanced binary tree structure may include allowing the compiler to locally optimize aggregation operations. In some implementations, incorporating a balanced binary tree structure may include making a balanced binary tree structure with the same unit logic design available for a set of superlanes. In addition to these implementations, for example, making a balanced binary tree structure available may include making two adjacent superlanes in a set operable.

[0060] In another example, enabling a balanced binary tree structure could involve operationally connecting the partial aggregation results of a set of superlanes. In addition to this example, the method could involve routing the first partial aggregation result for a first set of superlanes out of the set of superlanes to the second partial aggregation result for a second set of superlanes out of the set of superlanes. In some implementations, the method could involve delaying the second partial result by a defined number of cycles determined to match the latency for the first partial result.

[0061] Another embodiment relates to a system including a compiler that incorporates a balanced binary tree into the multiplication module of a tensor processor to execute a sequence of instructions more efficiently for stream reduction operations. In one example, the compiler adds a set of multiplexers at positions determined based on the location of the superlanes. In addition to this example, the selection signals for the set of multiplexers may be controlled by a group of configuration registers.

[0062] In another example, the compiler may include a unit design for partial aggregation logic. In addition to this example, the unit design for partial aggregation logic may include a multi-entry delayed buffer, a first set of routing channels, and a second set of routing channels.

[0063] In one implementation, the compiler makes available a balanced binary tree with the same unit logic design for a set of superlanes. In addition to this implementation, the compiler connects two adjacent superlanes from the set in a functional way. In one implementation, the balanced binary tree can contain at least four superlanes. However, in other implementations, the balanced binary tree can contain three or fewer superlanes, four superlanes, or five or more superlanes. Forms for carrying out the invention - Technical support From data / instructions to processor / program Data and Information. While “data” and “information” are often used interchangeably (e.g., “data processing” and “information processing”), the term “datum” (plural “data”) generally refers to a representation of a factual value (e.g., a measurement of a physical quantity such as electric current in a wire or the price of gold) or an answer to a question (e.g., “yes” or “no”), while the term “information” generally refers to a set of data that has structure (often indicated by “data structure”). Data structures are used in trade to convert electronic devices into manufactured goods for use as specific machines (see re Lowry, 32 F.3d 1579 [CAFC, 1994]). Data and information are binary data (usually represented by “0” and “1” “bits”) enabled by two levels of voltage in physical objects, such as digital circuits or electronic components. For example, data can be enabled as electrical, magnetic, optical, or acoustic signals or states, quantum states such as particle spins enabling “qubits,” or physical states of atoms or molecules. All such data and information, when made possible, will be stored, accessed, transferred, combined, compared, or acted upon for actions that require and incur energy.

[0064] As used herein, the term “process” refers to a finite, ordered set of artificial physical actions (also indicated by “actions” or “steps”) for producing at least one result. Some types of actions include transformation and transport. An action is an example of a technical application of one or more natural laws of science or artificial laws of technology. An action often alters the physical state of a machine, a structure of data and information, or a composition. If a process produces the same result, two or more actions may occur at approximately the same time, or one action may occur before or after another. Descriptions of physical actions and / or transformations comprising a process are often indicated by a set of gerund phrases (or their semantic equivalents) generally preceded by the signifier “steps to do” (e.g., “a process comprising the steps of measuring, transforming, partitioning, and then distributing…”). The signifiers “algorithm,” “method,” “procedure,” “(sub)routine,” “protocol,” “recipe,” and “technique” are often used interchangeably with “process,” and 35 U.S.C. § 100 defines “method” by legislative statute as one type of process that is always patentable under 35 U.S.C. § 101. The term “thread” as used herein refers to a subset of an entire process. A process can be divided into multiple threads that may be used simultaneously or nearly simultaneously.

[0065] As used herein (and also indicated by “Part” and generally by “Element” when described in the patent text or figures), the term “Component” refers to a physical object used to enable processing in combination with other components. For example, electronic components are used in processing that affect the physical state of one or more electromagnetic or quantum particles / waves (e.g., electrons, photons) or quasiparticles (e.g., electron holes, phonons, magnetic domains) and their associated fields or signals. Electronic components generally have at least two connection points attached to conductive components, generally conductive wires or lines or optical fibers, with one end of one conductive component attached to the component and the other end connected to another component, as part of a circuit through which current or photons flow. There are at least three types of electrical components: passive, active, and electromechanical. Passive electronic components generally do not introduce energy into a circuit, and such components include resistors, memristors, capacitors, magnetic inductors, crystals, Josephson junctions, transducers, sensors, antennas, waveguides, etc. Active electronic components require an energy source and can inject energy into a circuit; such components include semiconductors (e.g., diodes, transistors, optoelectronic devices), vacuum tubes, batteries, power supplies, and displays (e.g., LEDs, LCDs, lamps, CRTs, plasma displays). Electromechanical components use mechanical force and structure to influence the flow of current; such components include switches, relays, protective devices (e.g., fuses, circuit breakers), heat sinks, fans, cables, wires, terminals, connectors, and printed circuit boards.

[0066] One of the most important components of a commodity in a transaction is the integrated circuit, and its abstractions. As used herein, the term “integrated circuit” refers to a set of connected electronic components on a small substrate of semiconductor material such as silicon or gallium arsenide, with components fabricated on one or more layers (hence the use of the signifier “chip”). Other signifiers for “integrated circuit” include “monolithic integrated circuit,” “IC,” “chip,” “microchip,” and “system on a chip” (“SoC”). Examples of types of integrated circuits include gate / logic arrays, processors, memory, interface chips, power controllers, and operational amplifiers. In electronic circuit design, the term “cell” refers to the specifications of one or more components, for example, a set of transistors connected to function as logic gates. Cells are typically stored in a database that will be accessed by the circuit designer and the design process.

[0067] As used herein, the term “module” refers to a tangible structure for acting on data and information. For example, the term “module” can refer to a process that transforms data and information, such as a computer program (as defined below). The term “module” can also refer to one or more interconnected electronic components, such as a digital logic device. A process comprising a module can also be translated into a specification for the structure of electronic components that transform data and information producing the same results as the process, when specified in a programming language (as defined below), such as System C or Verilog. This last sentence is derived from the modified Church-Turing thesis, which, contrary to the doublethink of simply removing the one “(patentable),” can be expressed as “Anything that can be transformed by a (patentable) process and processor can be transformed by a set of (patentable) equivalents of modules.”

[0068] A module can be permanently structured (e.g., a circuit using immutable connections), temporarily structured (e.g., a circuit or process that can be modified using a set of data), or a combination of both forms of structuring. For example, a permanently structured module may be manufactured using application-specific integrated circuits ("ASICs"), such as arithmetic logic units ("ALUs"), programmable logic arrays ("PLAs"), or read-only memory ("ROMs"), all of which are generally structured during manufacturing. For example, a permanently structured module comprises an integrated circuit. A temporarily structured module may be manufactured using, for example, field-programmable gate arrays (e.g., FPGAs sold by Xilinx or Intel's Altera), random access memory (RAM), or a microprocessor. For example, data and information are translated using data as addresses in RAM or ROM memory that store output data and information. A temporarily structured module can be embedded within a permanently structured module (e.g., an FPGA embedded in an ASIC).

[0069] Modules that are temporarily structured may be structured over multiple time periods. For example, a processor with one or more modules may have modules of its own that are initially structured by the manufacturer in the factory and then further structured by the user when used in a transaction. A processor may have a set of one or more modules in a first time period and then be reconfigured to have one or a different set of modules in a second time period. The decision to manufacture or implement modules in a permanently structured form, a temporarily structured form, or a combination of the two forms depends on transactional issues such as cost, time considerations, resource constraints, tariffs, maintenance needs, national intellectual property laws, and / or specific design purposes [FACT]. How a module is used and what it does to function is largely independent of the physical form in which it is manufactured or made available. This last sentence is also derived from a modified Church-Turing thesis.

[0070] As used herein, the term “processor” refers to a tangible data and information processing machine for use in transactions that physically transform, transfer, and / or transmit data and information using at least one process. A processor consists of one or more modules, for example, a central processing unit ("CPU") module, an input / output ("I / O") module, a memory control module, a network control module, and / or other modules. The term “processor” can also refer to one or more processors, one or more processors with multiple compute cores / CPUs, a specialized processor (for example, a graphics processor or a signal processor), and combinations thereof. When two or more processors interact, one or more of the processors may be located remotely from the locations of the other processors. When the term “processor” is used in another context, such as “chemical processor,” it will be indicated and defined in that context.

[0071] A processor may comprise, for example, digital logic circuits (e.g., binary logic gates) and / or analog circuits (e.g., operational amplifiers). A processor may also utilize a combination of technologies such as optical signal processing, DNA transmutation, quantum computing, microfluidic logic processing, or optoelectronic processors. Any processor that can transform data and information using AND, OR, and logical operations (as well as their derivatives such as NAND, NOR, and XOR operations) for data and information structured with binary data may also transform data and information using any function of Boolean logic. Processors such as analog processors, including artificial neural networks, can also transform data and information. There is no scientific evidence that any of these technological processors process, store, and retrieve data and information using any processing or structure equivalent to the bioelectric structure and processing of the human brain.

[0072] One or more processors may also be used for processing in a “cloud computing” or “time-sharing” environment, where the time and resources of multiple remote computers are shared by multiple users or processors communicating with the computer. For example, a group of processors may use at least one processing available in a distributed or remote system, and these processors may use a communication network (e.g., the Internet or Ethernet) and one or more designated network interfaces (as defined below, “interfaces”) (e.g., application programming interfaces (“APIs”) that provide functions and data structures for communicating with remote processing.

[0073] As used herein (and further defined below), the terms “computer” and “computer system” include, for example, at least one processor that performs operations on data and information using electronic gates that may include transistors, such as Boolean logical operations, by adding memory (for example, memory structured with flip-flops that use NOT-AND or NOT-OR operations). Any processor that can perform logical AND, OR and NOT operations (or their equivalents) is Turing complete and computationally universal [FACT]. A computer can have a simple structure, for example, comprising an input / output module, a CPU module and memory, which perform operations that take signals in, transform signals and output signals without human intervention.

[0074] As used herein, the term “programming language” refers to a structured grammar for specifying a set of operations and data for use by modules, processors, and computers. A programming language includes either source code or object code written in any combination of assembler instructions, instruction set architecture instructions, machine language instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or one or more higher-level languages, such as the C programming language and similar common programming languages ​​(Fortran, Basic, Javascript, PHP, Python, C++, etc.), knowledge programming languages ​​(LISP, Smalltalk, Prolog, or CyclL, etc.), electronic structure programming languages ​​(VHDL, Verilog, SPICE, or SystemC, etc.), text programming languages ​​(SGML, HTML, or XML, etc.), or audiovisual programming languages ​​(SVG, MathML, X3D / VRML, or MIDI, etc.), and any future equivalent programming languages. As used herein, the term “source code” refers to a set of instructions and data specified in text form using a programming language. A large amount of source code to enable any of the claimed inventions is available on the internet, such as from source code libraries like GitHub.

[0075] As used herein, the term “program” (also called “application program”) refers to one or more processes and data structures that structure a module, processor, or computer to be used as a “specific machine” (see In re Alappat, 33 F3d 1526 [CAFC, 1991]). One use of a program is to structure one or more computers, for example, a standalone, client, or server computer, or one or more modules, or a system of one or more such computers or modules. As used herein, the term “computer application” refers to a program that has a specific use, for example, to enable text processing operations or to encrypt a set of data. As used herein, the term “firmware” generally refers to a type of program that structures a processor or computer, where firmware is smaller in size than a typical application program, generally less accessible to the computer user, and not modifiable. Computer programs and firmware are often specified using source code written in programming languages ​​such as C modules, and circuits, processors, programs, and computers can be specified at multiple levels of abstraction using, for example, the SystemC programming language, and may have value as a product in trade as a taxable item under the Uniform Commercial Code (see UCC Title 2, Section 1).

[0076] A program is transferred from a data and information device or storage system to one or more memories of a computer or computer system. A computer system generally has devices for reading the storage medium used to transfer programs, and / or interface devices for receiving programs over a network. This transfer is described in the general computer description section. Forms for carrying out the invention - Technical support General computer description Figure 11 shows a computer system suitable for enabling embodiments of the claimed invention.

[0077] In Figure 11, the structure of the computer system 500 generally includes at least one computer that communicates with peripheral devices via a bus subsystem 504. Generally, the computer includes a processor 502 (e.g., a microprocessor, graphics processing unit, or digital signal processor), or an electronic processing equivalent thereof such as an application-specific integrated circuit ("ASIC") or a field-programmable gate array ("FPGA"). Generally, the peripheral devices include a storage subsystem 506 comprising a memory subsystem 508 and a file storage subsystem 510, a user interface input device 512, a user interface output device 514, and / or a network interface subsystem 516. The input / output devices enable direct and remote user interaction with the computer system 500. The computer system enables significant post-processing activities using at least one output device and / or the network interface subsystem.

[0078] A computer system can be structured as a server, client, workstation, mainframe, personal computer (PC), tablet PC, set-top box (STB), personal digital assistant (PDA), cellular phone, smartphone, web device, rack-mount “blade,” kiosk, television, game station, network router, switch or bridge, or any data processing machine having instructions that specify the actions to be performed by such a machine. As used herein, the term “server” generally refers to a computer or processor that performs processing for another computer or processor and sends data and information to it.

[0079] Computer systems are generally structured using at least one operating system program, such as Microsoft's Windows, Sun Microsystems' Solaris, Apple Computer's MacOs and iOS, Google's Android, Linux®, and / or Unix. Computer systems generally include a Basic Input / Output System (BIOS) and processor firmware. The operating system, BIOS, and firmware are used by the processor to structure and control any subsystems and interfaces connected to it. Typical processors that enable these operating systems include Intel's Pentium, Itanium, and Xeon processors, Advanced Micro Devices' Opteron and Athlon processors, Amazon's Graviton processors, IBM's POWER processors, Oracle's SPARC processors, and ARM Holdings' ARM processors.

[0080] Any ECIN is not limited to electronically programmable devices or electronically digital logic computers structured using programs. For example, the claimed invention may use optical computers, quantum computers, analog computers, etc. Furthermore, where only one computer system or single machine is indicated, the use of the singular form of such a term may also indicate any structure of computer systems or machines that use processing individually or collectively. Due to the constantly changing nature of computers and networks, the description of computer system 500 shown in Figure 11 is merely an example. Many other structures of computer system 500 have more or fewer components than the computer system shown in Figure 11.

[0081] The network interface subsystem 516 provides an interface to an external network, including an interface to the communication network 518, and is coupled via the communication network 518 to corresponding interface devices in other computer systems or machines. The communication network 518 comprises many interconnected computer systems, machines, and physical communication connections (indicated by “links”). These communication links may be wireline links, optical links, wireless links (e.g., using WiFi or Bluetooth protocols), or any other physical devices for the communication of information. The communication network 518 may be any suitable computer network, such as a wide area network such as the Internet and / or a local-to-wide area network such as Ethernet. The communication network may be wired and / or wireless, and many communication networks use encryption and decryption processes, such as those available in conjunction with virtual private networks. The communication network uses one or more communication interfaces, which receive data from and transmit data to other systems. Embodiments of the communication interface commonly include Ethernet cards, modems (e.g., telephone, satellite, cable, or ISDN), (asynchronous) digital subscriber line (DSL) units, Firewire interfaces, USB interfaces, and the like. A communication algorithm ("protocol") may be specified using one or more communication languages, such as HTTP, TCP / IP, RTP / RTSP, IPX, and / or UDP.

[0082] The user interface input device 512 may include pointing devices such as alphanumeric keyboards, keypads, and mice, trackballs, toggle switches, touchpads, styluses, graphics tablets, optical scanners such as barcode readers, touchscreen electronic circuits for display devices, audio input devices such as speech recognition systems or microphones, gaze recognition, electroencephalogram pattern recognition, optical character recognition systems, and other types of input devices. Such devices may be connected to the computer system by wire or wirelessly. Generally, the term “input device” refers to all possible types of devices and processes for transferring data and information to the computer system 500 or over the communication network 518. The user interface input device generally allows the user to select some type of user interface output device, such as objects, icons, text, etc., that appear on the display subsystem.

[0083] The user interface output device 514 may include a display subsystem, a printer, a fax machine, or non-visual communication devices such as audio and haptic devices. The display subsystem may include flat panel devices such as cathode ray tubes (CRTs), liquid crystal displays (LCDs), image projection devices, or any other devices for creating visible stimuli, such as virtual reality systems. The display subsystem may also provide non-visual stimuli, such as through audio output, aroma generation, or haptic / tactile output (e.g., vibration and force) devices. Generally, the term “output device” refers to all possible types of devices and processes for transferring data and information from the computer system 500 to a user or another machine or computer system. Such devices may be connected to the computer system by wire or wirelessly. Note: Some devices transfer data and information both into and out of the computer; for example, a haptic device generates vibration and force on the user’s hand and also incorporates sensors for measuring the hand’s location and movement. Examples of technical applications of ergonomics and semiotics sciences can be used to improve the efficiency of any processing disclosed herein, such as any interaction with the design and manufacture of circuits, and user interaction with computers, using either of the input or output devices described above.

[0084] The memory subsystem 508 generally includes several memories, including a main random access memory ("RAM") 520 (or other volatile storage device) for storing program instructions and data during execution, and a read-only memory ("ROM") 522 for storing fixed instructions. The file storage subsystem 510 provides persistent storage for program and data files and may include a hard disk drive, a floppy disk drive with associated removable media, a CD-ROM drive, an optical drive, flash memory such as a USB drive, or a removable media cartridge. If the computer system 500 includes an input device that performs optical character recognition, printed text and symbols on paper may be used as devices for storing program and data files. Databases and modules used by some embodiments may be stored in the file storage subsystem 510.

[0085] The bus subsystem 504 provides devices for transmitting data and information between various components and subsystems of the computer system 500. Although the bus subsystem 504 is shown as a single bus, alternative embodiments of the bus subsystem may use multiple buses. For example, main memory using RAM may communicate directly with a file storage system using a direct memory access ("DMA") system. Morphological-semantic support for carrying out the invention The signifier “Commercial Solution” simply refers to an electronic structure, a process for a specified machine, a manufacturable circuit (and its Church-Turing equivalent), or a composition that is specific to a technical domain (and therefore non-preemptive—see Bilski) to which science and / or technology are applied for use in trade to solve an unsatisfied requirement of the technology, as described in the following paragraphs.

[0086] The signifier “abstract” is precisely defined by the inventor as follows (see MPEP 2111.01 (9th edition, Rev. 08.2017)): (when used in a patent claim for any effective embodiment disclosed herein for a new commercial solution that is a scientific use of one or more laws of nature (see Benson) and solves a technical problem for use in trade (see Diehr) or improves upon an existing solution for use in trade (see Diehr)).

[0087] a) A new commercial solution is “abstract” if it is not novel (for example, it is well known in the equivalent prior art (see Alice) and / or the use of an equivalent prior art solution has long been prevalent in science, engineering or trade (see Bilski)), and therefore, for example, it is “difficult to understand” how the commercial solution differs from an equivalent prior art solution, and is therefore not patentable under 35 U.S.C. § 102 (see Merriam-Webster’s definition of “abstract”) or b) If a new commercial solution includes the disclosure of at least two prior art documents in which the existing prior art includes at least one similar prior art solution (see KSR) or which the existing prior art can be combined by a person skilled in the art (often referred to as “PHOSITA,” see MPEP 2141-2144 (9th edition, Rev. 08.2017)) to be equivalent to the new commercial solution, then it is “abstract” and therefore, for example, “difficult to understand” how the new commercial solution differs from the existing prior art PHOSITA combination / application example, and is therefore not patentable under 35 U.S.C. § 103, or c) A new commercial solution is “abstract” if it is not disclosed with a description that would enable its practice, either because there is insufficient guidance in the description or because only a general implementation is described without specifying components, parameters or functions (see Mayo), and therefore PHOSITA cannot instantiate, for example, one embodiment of the new solution for use in a transaction without requiring special programming (see Kaz) (or, for example, circuit design) to be performed by PHOSITA, and therefore it is “difficult to understand,” for example, how any embodiment of the new commercial solution should be used in a transaction, and thus it is not patentable under 35 United States Code § 112. Forms for carrying out the invention - conclusion Modes for carrying out the invention, individually, describe any features, structures, functions, or characteristics, or any combination thereof, to the extent that such features, structures, functions, or characteristics, or any combination thereof, are made possible by modes for carrying out the invention as a whole in light of the knowledge and understanding of a person skilled in the art, regardless of whether such features, structures, functions, or characteristics, or any combination thereof, solve any problem disclosed herein, without limiting to the claims. When an ECIN has a particular feature, structure, function, or characteristic, it is in the knowledge and understanding of a person skilled in the art to use such feature, structure, function, or characteristic with respect to another ECIN, for example, as a substitute for another feature, structure, function, or characteristic, whether or not it is expressly described.

[0088] In view of modes for carrying out the invention, those skilled in the art will understand that many variations of any ECIN are possible, including the function and structure of the elements, that are as useful as the ECIN described herein. One or more elements of an ECIN can be substituted for one or more elements of another ECIN, as will be understood by those skilled in the art. Any description of any ECIN indicates its use in trade, thereby enabling those skilled in the art to use this ECIN in trade as well.

[0089] The modes for carrying out this invention are adequately described to provide knowledge and understanding. They should be neither exhaustive nor limiting of the exact structures described, but should be given the broadest scope that conforms to the disclosed principles and features. Without limitation, any and all equivalents described, shown, or incorporated by reference (or explicitly incorporated) in this patent application are incorporated in particular into the modes for carrying out the invention. Furthermore, any and all variations described, shown, or incorporated with respect to any one ECIN may also be included with any other ECIN. Any such variations include currently known variations as well as future variations, for example, any element used for implementability includes future equivalent elements that give the same function, regardless of the structure of the future equivalent element.

Claims

1. The compiler incorporates a balanced binary tree structure into the multiplication module of the matrix execution module (MXM) of the tensor processor, generates a set of instructions for executing a program on the matrix execution module incorporating the balanced binary tree structure, and more efficiently executes the sequence of instructions for stream reduction operations. A method that includes this.

2. The method according to claim 1, wherein the unit design for the partial aggregation logic in the balanced binary tree structure comprises a number of delay buffers corresponding to the initial data stream lag delay in order to cancel the delay of the partial aggregation logic.

3. The method according to claim 1, wherein the step of incorporating cancels delay dependencies in the aggregation logic by configuring the delay of each partial aggregation logic in the balanced binary tree structure to be offset between two operands input to the partial aggregation logic.

4. The method according to claim 3, wherein the step of canceling the delay dependency includes the step of configuring the output of the partial aggregation logic of each layer to be input to the partial aggregation logic of the next stage via a fixed delay buffer, thereby eliminating the need to shift the data stream according to the aggregation delay.

5. The method according to claim 3, wherein the step of canceling the delay dependency includes the step of unifying the stream reduction operations by processing the stream reduction operations in a common aggregation structure.

6. The method according to claim 1, wherein the step of incorporating the balanced binary tree structure enables the compiler to locally optimize the aggregation operation based on the removal of delay dependencies in the partial aggregation logic.

7. The method according to claim 1, wherein the step of incorporating the balanced binary tree structure includes the step of making the balanced binary tree structure having the same unit logic design available for a set of superlanes.

8. The method according to claim 7, wherein the step of making the balanced binary tree structure available includes the step of connecting two adjacent superlanes of the set of superlanes in an operational manner by routing the partial aggregation results with a delay of a number of cycles defined to match a predetermined latency.

9. The method according to claim 7, wherein the step of making the balanced binary tree structure available includes the step of operably connecting the partial aggregation results of the set of superlanes.

10. Steps to route the first partial result of the partial aggregation result for the first set of superlanes out of the set of superlanes to the second partial result of the partial aggregation result for the second set of superlanes out of the set of superlanes. The method according to claim 9, further comprising:

11. The step of delaying the second partial result by a defined number of cycles determined to match the latency of the first partial result. The method according to claim 10, further comprising:

12. The method according to claim 1, wherein the balanced binary tree structure is provided across at least four superlanes.

13. A tensor processor comprising a matrix execution module (MXM), wherein the MXM comprises a balanced binary tree, A compiler that generates a set of instructions for executing a program using the tensor processor incorporating the balanced binary tree, and is configured to execute the set of instructions more efficiently for stream reduction operations. A system equipped with these features.

14. The system according to claim 13, wherein the compiler generates configuration information for arranging a set of multiplexers at locations determined based on the location of the superlane.

15. The system according to claim 14, wherein the selection signals of the set of multiplexers are controlled by a group of configuration registers.

16. The system according to claim 13, wherein the compiler generates configuration information relating to unit design for implementing partial aggregation logic.

17. The system according to claim 16, wherein the unit design for the partial aggregation logic comprises a multiple entry delay buffer, a first set of routing channels, and a second set of routing channels.

18. The system according to claim 13, wherein the compiler makes available the balanced binary tree having the same unit logic design for a set of superlanes.

19. The system according to claim 18, wherein the compiler generates configuration information for operationally connecting two adjacent superlanes from the set of superlanes.

20. The system according to claim 13, wherein the balanced binary tree is provided across at least four superlanes.

Images