Neural network processing with link instructions

Abstract

The present disclosure relates to neural network processing with link instructions. Hardware and methods for neural network processing are provided. A method in a hardware node comprising a pipeline having a matrix vector unit (MVU), a first multi-function unit connected to receive input from the matrix vector unit, a second multi-function unit connected to receive output from the first multi-function unit, and a third multi-function unit connected to receive output from the second multi-function unit is provided. The method includes executing a first type of instruction using the MVU, the first type of instruction executable only by the MVU to generate a first result. The method also includes executing a second type of instruction, the second type of instruction executable only by one of the multi-function units and generating a second result, and passing the second result to the second and third multi-function units without storing either of the two results in a global register.

Description

[0001] This application is a divisional application of Chinese invention patent application filed on February 8, 2018, with application number 201880013993.2 and invention title "Neural Network Processing Using Linking Instructions". Background Technology

[0002] Neural network technology is used to perform complex tasks such as reading comprehension, language translation, or speech recognition. While neural networks can perform such tasks, deploying them using general-purpose CPUs or GPUs is expensive. Furthermore, while GPUs offer increased throughput compared to CPUs, they have lower latency. Summary of the Invention

[0003] In one example, this disclosure relates to a method for processing instructions in a hardware node including global registers and a pipeline, the pipeline comprising: a matrix vector unit including at least one accumulator register; a first multifunction unit connected to receive input from the matrix vector unit; a second multifunction unit connected to receive output from the first multifunction unit; and a third multifunction unit connected to receive output from the second multifunction unit. The method may include receiving a first type of instruction from an input queue via a global register, wherein the first type of instruction can only be executed by the matrix vector unit, and the matrix vector unit executing the first type of instruction and storing a first result of the first type of instruction in at least one accumulator register. The method may also include receiving a second type of instruction from an input queue via a global register, wherein the second type of instruction can only be executed by at least one of the first, second, or third multifunction units, and executing the second type of instruction using the first multifunction unit, wherein the execution includes performing an operation on the first result of at least the first type of instruction and generating a second result. The method may further include passing the second result to a second multifunction unit without storing the first or second result in a global register, and then passing the second result to a third multifunction unit.

[0004] In another example, this disclosure relates to a hardware node including a global register and a pipeline configured to process instructions, the pipeline including: a matrix vector unit including at least one accumulator register; a first multifunction unit connected to receive input from the matrix vector unit; a second multifunction unit connected to receive output from the first multifunction unit; and a third multifunction unit connected to receive output from the second multifunction unit. The hardware may also be configured to receive a first type of instruction from an input queue via a global register, wherein the first type of instruction can only be executed by the matrix vector unit, and the matrix vector unit executes the first type of instruction and stores a first result of the first type of instruction in at least one accumulator register; receive a second type of instruction from the input queue via a global register, wherein the second type of instruction can only be executed by at least one of the first, second, or third multifunction units; and execute the second type of instruction using the first multifunction unit, wherein the execution includes performing an operation on the first result of at least the first type of instruction and generating a second result, and passing the second result to the second multifunction unit without storing the first or second result in the global register, and then passing the second result to the third multifunction unit thereafter.

[0005] In yet another example, this disclosure relates to a hardware node including a global register and a pipeline configured to process instructions, the pipeline including: a matrix vector unit including at least one accumulator register; a first multifunction unit, wherein the first multifunction unit is connected to receive input from the matrix vector unit; a second multifunction unit, wherein the second multifunction unit is connected to receive output from the first multifunction unit; and a third multifunction unit, wherein the third multifunction unit is connected to receive output from the second multifunction unit, wherein each of the first, second, and third multifunction units further includes at least one of point-by-point addition and an operation-free block. The hardware can also be configured to receive a first type of instruction from the input queue via a global register, wherein the first type of instruction can only be executed by a matrix vector unit, and the matrix vector unit executes the first type of instruction and stores a first result of the first type of instruction in at least one accumulator register; receive a second type of instruction from the input queue via a global register, wherein the second type of instruction can only be executed by at least one of a first multifunction unit, a second multifunction unit, or a third multifunction unit; and execute the second type of instruction using the first multifunction unit, wherein the execution includes performing an operation on the first result of at least the first type of instruction and generating a second result, and passing the second result to the second multifunction unit without storing the first result or the second result in the global register, and then passing the second result to the third multifunction unit thereafter.

[0006] This summary is provided to introduce some concepts in a simplified form, which will be further described in the detailed embodiments below. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Attached Figure Description

[0007] This disclosure is illustrated by way of example and is not limited to the accompanying drawings, wherein the same reference numerals indicate similar elements. The elements in the drawings are illustrated for simplicity and clarity and are not necessarily drawn to scale.

[0008] Figure 1 It is a block diagram based on an example system that includes nodes interconnected via a data center network;

[0009] Figure 2 It is a block diagram based on an example system that includes distributed nodes;

[0010] Figure 3 It is a block diagram based on an example hardware node;

[0011] Figure 4 It is a block diagram of a sample neural functional unit;

[0012] Figure 5A and Figure 5B A block diagram of a neural functional unit based on another example is shown;

[0013] Figure 6 A block diagram of a hardware node (e.g., an FPGA) for performing neural network processing is shown according to an example.

[0014] Figure 7 A block diagram is shown illustrating a tile arrangement for performing neural network processing, based on an example.

[0015] Figure 8 A block diagram of a processing element according to an example is shown;

[0016] Figure 9 An example implementation of nodes including a partition matrix is ​​shown;

[0017] Figure 10 A block diagram of a multifunctional unit based on an example is shown;

[0018] Figure 11 A data flow graph for evaluating a neural network is shown based on an example;

[0019] Figure 12An example of instruction chain processing by a neural functional unit is shown, based on one example.

[0020] Figure 13 A data flow graph for evaluating a neural network is shown based on an example;

[0021] Figure 14 A schematic diagram illustrating how an instruction chain can be processed using a hardware node (e.g., an FPGA) is shown, based on an example; and

[0022] Figure 15 A flowchart illustrating a method for processing instructions, based on an example, is shown. Detailed Implementation

[0023] The examples disclosed in this disclosure relate to systems, methods, and components for implementing neural network-based processing. Some examples relate to deep neural networks (DNNs). A DNN can be any suitable neural network used for deep learning. Additional examples in this disclosure relate to functional units included as part of a node used to implement a DNN or similar neural network. Nodes can be implemented using portions or combinations of field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), erasable and / or complex programmable logic devices (PLDs), programmable array logic (PAL) devices, and general-purpose array logic (GAL) devices. Image files can be used to configure or reconfigure nodes such as FPGAs. Image files or similar files or programs can be delivered via a network link or a local link from a host CPU (e.g., PCIe). Information included in the image file can be used to program the hardware blocks of the node (e.g., logic blocks and reconfigurable interconnects of the FPGA) to achieve desired functionality. Desired functionality can be implemented to support any service that can be provided via a combination of computing, networking, and storage resources, such as via a data center or other infrastructure used for service delivery.

[0024] In one example, this disclosure relates to a DNN comprising multiple nodes (e.g., FPGAs) or groups of such nodes coupled to each other via a low-latency network. Aggregation platforms utilizing hundreds to thousands of such nodes (e.g., FPGAs) can advantageously provide: (1) significantly reduced training time by leveraging parallelism across hundreds of thousands of nodes; (2) the ability to enable novel training scenarios, such as in-situ online learning on real-time data; and (3) training models on an unprecedented scale while utilizing flexible and interchangeable homogeneous FPGA resources within hyperscale data centers spanning hundreds of thousands of servers. In one example, this advantage can be achieved by leveraging unconventional data representations that can utilize node architectures such as FPGAs.

[0025] The described aspects can also be implemented in a cloud computing environment. Cloud computing can refer to a model for enabling on-demand network access to a shared pool of configurable computing resources. For example, cloud computing can be used in the market to provide universal and convenient on-demand access to a shared pool of configurable computing resources. The shared pool of configurable computing resources can be rapidly provisioned via virtualization and published with low management effort or service provider interaction, and then scaled accordingly. Cloud computing models can consist of various characteristics, such as, for example, on-demand self-service, broad network access, resource pooling, rapid elasticity, quantifiable services, etc. Cloud computing models can be used to expose various service models, such as, for example, Hardware as a Service (“HaaS”), Software as a Service (“SaaS”), Platform as a Service (“PaaS”), and Infrastructure as a Service (“IaaS”). Different deployment models can also be used to deploy cloud computing models, such as private clouds, community clouds, public clouds, hybrid clouds, etc.

[0026] Machine learning services, such as those based on recurrent neural networks (RNNs), long short-term memory (LSTM) neural networks, or gated recurrent units (GRUs), can be implemented using the systems and nodes described in this disclosure. In one example, service-related content or other information (such as words, sentences, images, videos, or other such content / information) can be converted into a vector representation. This vector representation can correspond to techniques such as RNNs, LSTMs, or GRUs. Deep learning models can be trained offline before service initialization, and then deployed using the systems and nodes described in this disclosure. Nodes can be hardware programmable logic devices that can be specifically customized to perform the type of operations that occur in the context of a neural network such as a DNN. In one example, the state of the neural network model and the parameters used to control the model can be pinned to on-chip memory of a node comprising a distributed hardware platform. The neural network model can be pinned (e.g., preloaded) to on-chip memory at service startup, and the contents of the on-chip memory can remain unchanged unless the model needs to be changed or another event requires reloading the on-chip memory with the model. Therefore, in this example, contrary to other arrangements, the neural network model can be accessed directly into the on-chip memory (e.g., SRAM) of the hardware nodes instead of from the DRAM associated with the hardware platform. Attaching the model to a distributed set of programmable logic blocks (e.g., FPGA resources) allows the nodes (e.g., FPGAs) to operate at full capacity and can advantageously improve throughput and latency associated with the service. As an example, even a single request from the service can cause the distributed set of nodes to operate at full capacity, and thus deliver the results requested by the user of the service with very low latency.

[0027] In one example, a neural network model may include many layers, and each layer may be encoded as a matrix or vector of weights expressed as coefficients or constants obtained through offline training of the neural network. Programmable hardware logic blocks within the nodes can process the matrices or vectors to perform various operations, including multiplication, addition, and other operations on input vectors representing encoded information related to the service. In one example, the matrices or vectors of weights can be partitioned and fixed across multiple nodes using techniques such as graph partitioning. As part of this process, a large neural network can be converted into an intermediate representation (e.g., a graph), which can then be divided into smaller representations (e.g., subgraphs), and each weight matrix corresponding to each subgraph can be fixed to the node's on-chip memory. In one example, the model can be converted into fixed-size matrices and vectors. This allows the node's resources to operate on the fixed-size matrices and vectors in parallel.

[0028] Taking LSTM as an example, an LSTM network can include a series of repeating RNN layers or other types of layers. Each layer of an LSTM network can consume input at a given time step, such as the layer state from the previous time step, and can produce a new set of outputs or states. When using LSTM, single blocks of content can be encoded into a single vector or multiple vectors. As an example, words or combinations of words (e.g., phrases, sentences, or paragraphs) can be encoded into a single vector. Each block can be encoded into an individual layer of the LSTM network (e.g., at a specific time step). An LSTM layer can be described using a set of equations, such as the following:

[0029] i t =σ(W xi xt+W hi h t-1 +W ct c t-1 +b i

[0030] f t =σ(W xf x t +W hf h t-1 +W of c t-1 +b f )

[0031] c t =f t c t-1 i t tanh(W xc xt+W hc h t-1 +b c )

[0032] o t =σ(W xo x t +W ho h t-1 +W co c t +b o )

[0033] h t =o t tanh(c t )

[0034] In this example, within each LSTM layer, a combination of vector operations (e.g., dot product, inner product, or vector addition) and nonlinear functions (e.g., sigmoid, hyperbola, and tangent) can be used to process the input and hidden states. In some cases, most of the computationally intensive operations may come from the dot product, which can be implemented using dense matrix-vector and matrix-matrix multiplication routines. In one example, the processing of vector operations and nonlinear functions can be performed in parallel.

[0035] Figure 1 This is a block diagram of a system 100 comprising nodes interconnected via a data center network 110, as shown in the example. Figure 1As shown, multiple nodes 102, 104, and 106 can be coupled via a data center network. These nodes can be instantiated and used to parallelize multiple layers of a neural network such as an LSTM network. In one example, each node can be implemented as a server and may also include at least one hardware node (e.g., an FPGA). Thus, node 102 may include FPGA 122, node 104 may include FPGA 124, and node 106 may include FPGA 126. The FPGAs can be interconnected via a system based on an optical transport layer protocol. In one example, a first instance of FPGA 122 may be coupled to a first instance of FPGA 124 via transport link 132, and the first instance of FPGA 122 may also be coupled to a second instance of FPGA 124 via transport link 134. A first instance of FPGA 124 may be coupled to a first instance of FPGA 126 via transport link 136, and the first instance of FPGA 124 may also be coupled to a first instance of FPGA 126 via transport link 140. Similarly, a second instance of FPGA 124 can be coupled to a first instance of FPGA 126 via transport link 142, and a second instance of FPGA 124 can also be coupled to a second instance of FPGA 126 via transport link 138. Optical transport layer protocols can provide the FPGAs with the ability to transmit or receive packets or other such data to each other via data center network 110. FPGAs can also be interconnected in other configurations. For example, several instances of FPGA 122 can be coupled to several instances of FPGA 124 via multiple transport links 152. Similarly, several instances of FPGA 124 can be coupled to several instances of FPGA 126 via transport link 154. Although... Figure 1 The diagram shows a specific number and arrangement of nodes including FPGAs, but there may be more or fewer nodes arranged differently.

[0036] Figure 2This is a block diagram of a system 200 including distributed nodes, based on an example. In this example, multiple nodes can be implemented as server racks in a data center. Each server can be coupled to a top-of-rack (TOR) switch. Other racks, though not shown, can have similar configurations. Each server can include at least one or more nodes. Each node can include a server (e.g., server 204, server 206, or server 208), and each server can be coupled to a TOR switch (e.g., TOR switch 210). Server 204 can include host components, including CPUs such as CPU 214 and CPU 216, which can be coupled to hardware nodes, such as FPGA 218, via a local link (e.g., PCIe) 220. Each hardware node can also be coupled via (e.g., used to communicate across a network infrastructure for the data center) a network interface controller 222. Figure 2 The system illustrated allows nodes to process messages received from (and / or sent to) TOR switches or other switches. Using this example system, individual nodes can send packets directly to each other, thus allowing partitioning of even a single neural network spanning multiple FPGAs without inducing unacceptable latency. For communication, nodes can use lightweight protocols, including, for example, RDMA. Although Figure 2 A certain number of components of a system arranged in a certain way are shown, but there may be more or fewer components arranged differently.

[0037] Parallelization can also be performed within layers of a neural network by partitioning neural weights across multiple nodes. As an example, a single RNN model (e.g., including LSTM weight matrices) can be partitioned and fixed across multiple nodes. In this example implementation, the RNN model can be distributed across the memory (e.g., BRAM) of each FPGA in multiple FPGAs. In this example configuration, each individual FPGA in the multi-stage pipeline can store a portion of the LSTM weight matrix in fast on-chip memory (e.g., BRAM). This can advantageously lead to high throughput and low latency systems. At service startup, the LSTM weight matrix can be decomposed into matrices of a specific size (e.g., N×M matrices, where each of N and M is an integer equal to or greater than 8) and then loaded into the FPGA's on-chip memory. The runtime management layer can implement the allocation, scheduling, and management of the FPGAs. In one example, each node can be implemented as an LSTM-focused vector processor attached to one or more FPGAs using HaaS. Each node can be designed to run neural network evaluations—either as a PCIe-attached FPGA or as part of a HaaS pool of FPGAs.

[0038] Figure 3 This is a block diagram based on an example hardware node 300. Each hardware node 300 may include an Input Message Processor (IMP) 310 for receiving messages from other nodes and an Output Message Processor (OMP) 340 for processing outgoing messages to other nodes or components. Each node may also include a Control / Scalar Processor (CSP) 320 and a Neural Function Unit (NFU) 330. Although not shown, received messages received by a node may be stored in at least two different queues: (1) an IMP to CSP auxiliary queue and (2) an IMP to NFU data queue. Although not shown, outgoing messages may be stored in at least two different queues: (1) a CSP to IMP auxiliary queue and (2) an NFU to OMP data queue. In this example, a node may accept off-chip messages containing both auxiliary information such as control and scalar data and payload data (e.g., vector, matrix, or other tensor data structures). The auxiliary information may include requests to perform computationally intensive operations on the payload and then return the results as output messages. In this example, the incoming message is processed by a lightweight input message processor (IMP) 310, which sends auxiliary information to a control / scalar processor (CSP) 320 (which may be a NIOS-based control processor) and payload data (e.g., an input tensor) to a neural functional unit (NFU) 330, which may be implemented as a matrix-vector processor. As an example, the CSP 320 can then interpret the request and, based on its firmware, can send a series of instructions to the NFU 330. After a certain processing delay, the NFU can produce the result of the request, which can be combined with the auxiliary data produced by the CSP 320 in a lightweight output message processor (OMP) 340 and then sent off-chip. The CSP firmware can provide instructions to the NFU 330. Further details of the example instructions are discussed as part of the instruction set architecture (ISA). Runtime reloading of the firmware for the CSP 320 can also be performed. Therefore, in this example, the architecture is primarily event-driven. Input messages can arrive from many sources, including via a network. IMP can inspect the head of the message queue, and it can dequeue any instruction that needs to be executed and feed it through the system. Although Figure 3 A specific number of components are shown as example nodes arranged in a particular manner, but there may be more or fewer components arranged differently.

[0039] In one example, the NFU can be implemented as a matrix-vector processor designed to scale to a large portion of the FPGA's resources. In this example, the primary hardware acceleration goal of the NFU is to perform matrix-vector multiplication with high throughput and low latency by utilizing its matrix-vector units (MVUs) through the application of thousands of multiply-adders. The NFU can receive matrices of coefficients (e.g., constants) and can be used to multiply these coefficients with dynamically generated input vector data. Therefore, instead of storing the coefficients in the DRAM corresponding to the CPU / GPU, the coefficients can be preloaded into the on-chip memory corresponding to the NFU (e.g., the FPGA's block random access memory (BRAM)) at service startup. In one example, once the coefficients are loaded, they may never be loaded again unless the neural network model being used is modified or the service is restarted. In other words, as part of this example, the model can be partitioned and pinned to the on-chip memory of multiple nodes (e.g., the FPGA) in a distributed manner, with these nodes connected in a way that allows them to directly pass messages or groups to each other without relying on assistance from CPU resources.

[0040] In one example, the MDU can be fully pipelined and can be executed in O(n) time. 2 The NFU performs matrix-vector multiplication with a complexity of O(n), achieving a performance of 400-1800 billion fixed-point operations per second. While matrix-vector multiplication can represent the vast majority of fixed-point operations required to evaluate an LSTM layer, evaluation can also include various vector reductions, transcendentals, and bias vector additions. The NFU can also implement pipelined multifunction units (MFUs) to process these O(n) complexity vector functions in O(n) time. These MFUs can be organized into a chained architecture, where an MVU passes data to a first MFU, the first MFU passes data to a second MFU, and so on. In one example implementation of the NFU, one MVU and five MFUs can be used.

[0041] Based on the observation that vector functions may potentially dominate LSTM evaluation time in architectures where vector and matrix-vector operations take approximately the same amount of time to compute, the chained architecture allows NFU to leverage large-scale pipelined parallelism between a single matrix-vector multiplication and several vector functions. The NFU's memory subsystem can also be configured to support high throughput. As an example, the memory subsystem can support matrix value throughput up to 1.8 TB / s, as well as the simultaneous loading and storage of six vectors.

[0042] Figure 4An example implementation of NFU 400 is shown. NFU 400 may include an input queue (IQ) 410 for receiving input data and an output queue (OQ) 420 for outputting output data. Therefore, NFU 400 can introduce external vector and matrix data through its input queue (IQ) 410 and emit vector data through its output queue (OQ) 420. NFU 400 may include a Global Vector Register File (GVRF) 430 for providing a centralized location that can be used to store vector data. NFU 400 may include a Matrix-Vector Unit (MVU) 440 and five Multifunction Units (MFUs) (e.g., MFU#0 450, MFU#1 460, MFU#2 470, MFU#3 480, and MFU#4 490, as shown). Figure 4 (As shown in the diagram). MFU 440 may include a matrix-register file 442 for storing matrices that can be preloaded when a node-supported service begins. Each MFU may also include a Local Vector Register File (LVRF) 453 for storing local vector data for the corresponding MFU (e.g., LVRF 452, LVRF 462, LVRF 472, LVRF 482, and LVRF 492). Instructions can be executed sequentially by NFU 400. A series of instructions can be configured such that a pipeline (including, for example, one MFU 440 and five MFUs) executes the instruction chain in parallel. As an example, matrix operations (which use the MFU 440) can be used to multiply the input vector data, which can then be pipelined to the output queue 420. Various data flow possibilities in NFU 400 are via... Figure 4 The two dashed paths and one solid path are shown in the diagram. Further details and examples of the instruction chain will be provided later.

[0043] The example NFU 400 can operate on vectors and matrices. Vectors are sets of 1D scalar elements, and matrices are sets of 2D scalar elements. The size of the elements, vectors, and matrices can be set using the parameters in Table 1 below.

[0044]

[0045] Table 1

[0046] Certain parameters (e.g., as shown in Table 1) can be used to configure the NFU 400 at design time or afterwards. In one example, four parameters can be used to configure the NFU 400. The first parameter can be the data type of the matrix and vector elements, specifically the width of a single element (ELEM_WIDTH). As an example, 8-bit fixed-point data types, 16-bit fixed-point data types, 27-bit fixed-point data types, and 32-bit floating-point data types can be a set of data types in one example. The width of each data bus in the NFU 400 can be configured to ELEM_WIDTH*LANES bits; each vector can use ELEM_WIDTH*HWVEC_ELEMS bits in memory; and each matrix can use ELEM_WIDTH*HWVEC_ELEMS*HWVEC_ELEMS bits in memory.

[0047] The second parameter can be the hardware vector size (HWVEC_ELEMS). In one example, all vectors stored within the NFU 400 can have a fixed number of elements equal to HWVEC_ELEMS, and all vector instructions can accept HWVEC_ELEMS elements as input and / or produce HWVEC_ELEMS elements as output. Furthermore, all matrices can have a fixed number of elements equal to HWVEC_ELEMS.

[0048] Many applications can have their own algorithmic dimensions, which may differ from the hardware vector size. When this is the case, programmers (or compilers) can use techniques such as matrix blocking to map high-level operations to the hardware vector size. For example, Table 2 below shows an application with a 500x500 matrix size and a vector size of 500 elements; however, the NFU400's hardware vector size is 250. The functions shown in Table 2 can be used to resolve this difference.

[0049]

[0050] Table 2

[0051] The third parameter can be the number of vector channels (LANES), which describes how many elements should be operated in parallel within each MFU. As an example, the number of parallel operations within a matrix-vector unit (MVU) 440 can be defined as LANES * HWVEC_ELEMS, assuming there are HWVEC_ELEMS shards, each with LANES multipliers and adders. Shards are further described as part of the description corresponding to the example matrix-vector unit (MVU) 440. Furthermore, each NFU data bus, including the top-level port, can carry LANES vector elements per cycle for a total width of LANES * ELEM_WIDTH bits. In one example, LANES is an integer factor of HWVEC_ELEMS to avoid bit padding, since vectors operate in blocks of LANES size and require HWVEC_ELEMS / LANES cycles to process the vectors.

[0052] The fourth parameter can be the size of the matrix register file (NRF_SIZE), which stores a given number of HWVEC_ELEMS x HWVEC_ELEMS matrices in the on-chip memory corresponding to the NFU (e.g., fast on-chip BRAM (see description later)). In one example, the memory resource requirements on the node (e.g., the number of BRAM resources on the FPGA) can be derived using the following set of formulas (note that ceil(x,y) rounds x to the nearest multiple of y):

[0053] BRAM width =ceiL(LANES*DATA_WIDTH, 40)

[0054]

[0055] Regarding the memory subsystem, the NFU 400 can distribute its internal storage across three main types of memory. First, the matrix register file can be used to store MRF_SIZE HWVEC_ELEMS x HWVECS_ELEMS matrices in a series of fast on-chip random access memories (e.g., BRAM in the FPGA). These BRAMs can be distributed across the entire matrix vector unit, and each cycle can provide LANES matrix elements to achieve a total on-chip matrix throughput of HWVEC_ELEMS * LANES * ELEM_WIDTH bits / cycle. In this example, O(HWVEC_ELEMS) might be required. 2 The matrix is ​​stored in the matrix register file in 10 cycles; because such matrix storage can be performed in the preload step and then amortized over many matrix-vector multiplications.

[0056] Next, as Figure 4 As shown, the Global Vector Register File (GVRF) 430 can be used as a centralized location for storing vector data, which a programmer can use. An example configuration for the GVRF can store 32 vectors and can read LANES vector elements per cycle while also writing LANES vector elements per cycle. Furthermore, each multifunction unit in the NFU can have its own Local Vector Register File (LVRF), which can also read and write LANES vector elements per cycle. Therefore, in an example NFU with 5 MFUs, the total vector memory bandwidth is 6*LANES reads and 6*LANES writes per cycle for 12 separate vectors. This vector memory architecture is configured to support linking several vector functions together, one of which can be read from the GVRF, each of which can be read and written to an LVRF, and one of which can be written back to the GVRF.

[0057] The NFU's Matrix-Vector Unit (MVU) 440 can perform pipelined, high-throughput, low-latency matrix-vector multiplication. In one example, the MVU 440 achieves this using LANES*HWVEC_ELEMS multipliers and adders, and its throughput can be measured as 2*LANES*HWVEC_ELEMS*fmax operations per second. In one example, unlike typical high-throughput matrix-vector multipliers that rely on vector batching, the MVU 440 accepts one vector at a time into its pipeline.

[0058] Figure 5A and Figure 5BA block diagram of a neural functional unit (NFU) 500 according to another example is shown. The NFU 500 may include an input queue unit 510, a global vector register file (VRF) 520, a matrix-vector multiplication unit 530, and N multifunction units (MFUs). As an example, the NFU 500 may include MFU 0 540, MFU 1 560, and MFU N-1 580. In this example, each of these components may be connected in a pipelined manner. The NFU 500 may also include an arrangement of multiplexers and demultiplexers that can be used to control the routing of vectors, instructions, or other data via the NFU 500. For example, the NFU 500 may include multiplexers MUX 0 504, MUX 1 506, MUX 2 510, and MUX 544, and it may also include demultiplexers DEMUX 502, DEMUX 508, DEMUX 512, DEMUX 514, and DEMUX 516. As an example, demultiplexer DEMUX 502 can receive instructions or data from input queue unit 510 and couple it to multiplexer MUX 0 504 or multiplexer MUX 1 506. These alternative couplings can be implemented by providing control signals to the demultiplexer. The control signals can be generated by a decoder associated with NFU 500. Each MFU may include a Local Vector Register File (LOCAL VRF) 542, an adder block (+) 546, a multiplier block (X) 548, a hyperbolic tangent function block (TANH) 550, an S-shaped block (SIGM) 552, and an operation-free (NOP) block 554, as well as memory management functionality. Although Figure 5A and Figure 5B A certain number of components of the NFU500 are shown arranged in a certain way, but there may be more or fewer components arranged differently.

[0059] Figure 6 A block diagram of a hardware node (e.g., an FPGA) 600 for performing neural network processing is shown according to an example. Hardware node 600 may include columns of processing logic blocks (e.g., digital signal processors (DSPs)) 622 and memory elements (e.g., block RAMs) 610. A small set of BRAMs and DSPs can be configured to create processing slices, such as processing slice 630. Figure 6 In the example, each slice (e.g., processing slice 630) may include BRAMs 632, 634, 636, 638, and 640, which may be arranged as shown. Each processing slice 630 may also include processing logic blocks (e.g., digital signal processors (DSPs)) 642, 644, 646, 648, 650, and 652, which may be arranged as shown. Figure 6The arrangement is shown in the diagram. The processing logic block can be used to multiply the input vector by a row of weights. Adder 660 can be used to sum the outputs of the processing logic block. Therefore, in this example, each slice can perform a pointwise dot product operation. Although Figure 6 A certain number of components of hardware node 600 are shown arranged in a certain manner, but there may be more or fewer components arranged differently.

[0060] Figure 7 A block diagram of an arrangement 700 of slices for performing neural network processing, according to an example, is shown. Figure 7 As shown in the example, these vectors are fed to HWVEC_ELEMS processing elements via tree broadcast (e.g., in...). Figure 7 An array (referred to as a slice in the text). Therefore, as an example, vectors received via vector inlet 702 can be stored in... Figure 7 The entry tree register (REG) 710 is shown. Next, the vector can be fan-out to two or more registers (e.g., entry tree register 712 and entry tree register 714). An additional parameter (fan-out) can be used to describe the number of connections fanned out from each node in the entry tree. Processing elements (e.g., slices 722, 724, 726, and 728) can process the vector, and each slice can provide an output to an exit shift register (REG) (e.g., exit shift register 742 and exit shift register 744). The output from the MVU can be shifted out through vector exit block 750. The processing elements (e.g., slices) can perform LANES multiplication and LANES addition in each cycle.

[0061] An MVU can accept LANES input vector elements per cycle and, after a pipeline delay, output LANES elements for the corresponding output vector in each cycle. This delay for a single input vector can be defined as log0. FANOUT (HWVEC_ELEMS) + 2 * HWVEC_ELEMS / LANES, because logs are needed. FANOUT (HWVEC_ELEMS) cycles are needed to allow data to permeate through the ingress tree, HWVEC_ELEMS / LANES cycles are needed to compute the output, and HWVEC_ELEMS / LANES cycles are needed to send the vector through the egress pipeline.

[0062] Each processing element (e.g., a slice) can store a row of MRF_SIZE matrix and use its processing element to compute the dot product of vectors over a single row of the selected matrix. Figure 8A block diagram of a processing element 800 according to an example is shown. In this example, the processing element can be implemented as a generic matrix multiplication for a vector (GEMV) slice 810. Each processing element 800 can receive input data from an entry tree (e.g., such as...). Figure 5A and Figure 5B (as shown in the image) and output the data to the output pipeline (e.g., as shown in the image). Figure 7 (As shown in the diagram). Each processing element may include a block RAM (e.g., BRAM 820 for matrix row 1). In this example, each processing element (e.g., a GEMV slice) may include a BRAM that can store row 1 of the matrix and process the vector data in that row. In one example, the processing element may use a processing block (e.g., PE 830 for matrix row 1) to compute the dot product between the vector data and any input data (e.g., received via the path "Input Data from Ingress Tree"). Output data may be provided for further processing via the path "Output Data to Outgress Pipeline". Although Figure 8 A certain number of components of GEMV patch 810 arranged in a certain manner are shown, but there may be more or fewer components arranged differently.

[0063] Figure 9 An example implementation of node 900 in a slice matrix is ​​shown. In this example implementation, node 900 with 252 slices processes data from a 252x252 matrix 910. Example node 900 can receive data via input queue 920, which can be coupled to broadcast block 930 via bus 922. In this example, the bus width of bus 922 can be 192 bits. Broadcast block 930 can be coupled to SRAM blocks via individual buses to allow for the rapid loading of pre-trained neural network models into node 900. As an example, Figure 9 Segment 0 940, segment 1 950, and segment 251 960 are shown. Segment 0 940 can be coupled to broadcast block 930 via a 192-bit bus 932. Segment 1 950 can be coupled to broadcast block 930 via a 192-bit bus 934. Segment 251 960 can be coupled to broadcast block 930 via a 192-bit bus 936. Each segment may include a corresponding SRAM and dot product unit. As an example, segment 0 940 may include SRAM 942 and dot product unit 944, which can be coupled to each other using a 192-bit bus 946. Segment 1 950 may include SRAM 952 and dot product unit 954, which can be coupled to each other using a 192-bit bus 956. Segment 251 960 may include SRAM 962 and dot product unit 964, which can be coupled to each other using a 192-bit bus 966. Each SRAM can store one row of the weight matrix, which can be based on... Figure 9 The addressing scheme shown is stored. Each dot product unit can be implemented as a 12-input (12-INPUT) dot product unit. Upon service startup, the pre-trained neural network model weights can be fed into the FPGA's on-chip memory (e.g., ...). Figure 9 The coefficients are stored in SRAM (e.g., representing neurons) and pinned to a specific arrangement, allowing computational units to generate output with very high throughput. In one example, each row of the matrix can represent a neuron, and each column entry in that row can represent the synaptic weights attached to that neuron. The coefficients stored in matrix form (e.g., representing neurons) can be preloaded into SRAM or other memory associated with the hardware node. If a neuron does not fit into a single hardware node (e.g., an FPGA), a subset of neurons can be loaded into a second hardware node, and so on. In one example, a graph partitioning technique can be used to distribute the neuron weight matrix. This allows the graph representing the neuron weight matrix to be partitioned into subgraphs, which can then be pinned to the memory of different hardware nodes that can communicate with each other using lightweight transport protocols or other types of protocols. For example, using via... Figure 1 and Figure 2 The described architecture and system allow hardware nodes to communicate directly with each other.

[0064] Broadcast block 930 may include, as previously stated, Figure 5A and Figure 5BThe entry tree discussed. This example illustrates an architecture built using an FPGA that operates on fixed-size matrices and vectors. In this example, the natively supported size is a 252x252 square matrix. In other parameterized instances of these designs, matrices of other shapes and sizes can be used. In this example, there is a 24 square matrix that can be stored in on-chip memory (e.g., SRAM). Therefore, in this example, there is an array of 252 slices, and each slice is a computational unit. Each SRAM can receive 192 bits from the broadcast block and output 192 bits to 12 input dot product units per clock cycle. This translates to feeding 12 elements per cycle at 16 bits per element. As an example, the row of SRAM corresponding to address 0 (ADDR 0) is configured to store elements 0-0 to 0-11, which are the first 12 elements. In ADDR 1, another 12 elements are stored, and so on. This example illustrates a packing strategy of feeding multiple elements per row per clock cycle. In this example, only 21 rows of SRAM are used, from address 0 to address 20, and this is sufficient to store the entire square matrix. In this example, up to twenty-four 252x252 square matrices can be packed into the on-chip memory corresponding to node 900. Depending on the numerical precision of the weights, fewer or more matrices can be packed. As an example, 8-bit mode can be used to pack up to 48 matrices of the same size as 16-bit mode. In practice, other permutations can also be supported.

[0065] The 12 input dot product units perform the computation, and therefore, in this example, node 900 comprises a vector multiplication-addition tree. As an example, to perform a dot product operation, element-wise multiplication between each element of the row and each element of the vector can be performed, and then summed into an accumulated variable. Figure 9 In the example shown, each slice is responsible for computing one row of the matrix. In each clock cycle, 12 elements from the matrix can be processed at a time. This example corresponds to 16-bit integer operations (e.g., each weight is represented by 16 bits), and therefore each SRAM entry is 512 × 192 bits. Other bit sizes, including, for example, 1 bit, 2 bits, 4 bits, or 8 bits, can be used to represent weights corresponding to neural network patterns.

[0066] Figure 10 A block diagram of a multifunction unit (MFU) 1000 according to an example is shown, which corresponds, for example, to... Figure 4The previously described NFUs. Regarding the Multifunctional Units (MFUs) of NFUs, they can perform several vector functions based on local and / or external data. Example MFU implementations can support pointwise addition, multiplication, sigmoid and hyperbolic tangent functions, as well as pass and memory management functionality. In one example, each MFU can be configured such that each operator required for each instruction type that the MFU can process (e.g., a hardware block for performing the operation) is provided in each MFU, and each operator is copied along a chained path of multiple MFUs (e.g., as...). Figure 4 (As shown in the diagram). In this way, chained instructions (explained later) can be flexibly formed without having to specify or provide the position of the instructions in the chain. Lookup tables can be used to implement sigmoid and hyperbolic tangent. For the selected function, in one example, a single MFU can perform a certain number of operations per cycle (e.g., operations specified by the LANES parameter described earlier), while adding 1 to 2 delay cycles for addition / multiplication and sigmoid / tangent respectively. In this example, MFU 1000 may include hardware blocks configured to perform different operations. Each hardware block can perform an operation on values ​​or operands received from several sources, including, for example, previous MFUs, MVUs, or GVRFs. Intermediate values ​​can be stored in a local vector register file (LVRF) 1010. As an example, one hardware block 1020 can perform an addition operation. Another hardware block 1030 can perform a multiplication operation. Another block 1040 can perform a tangent (e.g., TANH) operation. Another block 1050 can perform a sigmoid (SIGM) operation. Another 1060 can perform no-op (NOP). The outputs of various hardware blocks can be supplied to the multiplexer 1060. Based on one or more control signals from various queue controllers or other control logic, the multiplexer (MUX) can provide an output that can be coupled to the next MFU in the pipeline or coupled to an output queue. In one example, the control signals can be signals received from the instruction decoder corresponding to the neural functional unit (NFU). The output of the MUX can also be coupled to the LVRF 1010 for storing intermediate values ​​corresponding to the various operations performed by the example MFU. In one example, a control processor that may be adjacent to the pipeline can have an instruction queue, and it can dynamically write instructions to the relevant queues for the various components of the NFU. When instructions are executed, they can fan out to a separate control queue, and it can be determined which part of the NFU is activated. Each component of the NFU can access the head of the control queue and perform an operation accordingly, such as addition, multiplication, tangency, sigmoid, or NOP. The control queue may not be dequeued until all vectors have been passed through, and then the control queue can be passed to the next component. Although Figure 10A certain number of components of an MFU1000 arranged in a particular manner are shown, but it may have more or fewer components arranged differently. As an example, the MFU 1000 may include additional hardware blocks for performing other operations, such as softmax operation, rectified linear unit (ReLU) operation, activation block operation, etc.

[0067] For a given example instruction set architecture (ISA) associated with a node including an NFU, instructions can always be executed in program order. Furthermore, all instructions can operate on vectors and / or matrices of native size HWVEC_ELEMS. The ISA can be used to expose both the distributed memory system and the large-scale pipelined parallelism available in the NFU pipeline. In one example, this parallelism can be leveraged by explicitly identifying two or more instructions as members of a chain. An example chain can be configured to execute all three instructions in parallel across an MVU and two MFUs using instructions for matrix-vector multiplication, vector-vector addition, and vector sigmoid functions. In one example, an instruction qualifies as a chain if: a series of related instructions with at most one input and at most one output requires global memory; all other arguments depend only on intermediate values ​​and the contents of local memory; and each functional unit uses an NFU at most once in pipelined order.

[0068] In one example, a program can be compiled or constructed to attempt to chain as many instructions together as possible to maximize performance. A pair of instructions can be chained together by declaring a forward flag in the destination argument of the first instruction and replacing the source address argument with a receive flag in the second instruction. The chain continues until an instruction no longer declares forwarding, or all functional units in the NFU have been used in the chain. Therefore, in this example, declaring forwarding in one instruction and not declaring receive in the next is illegal (and vice versa).

[0069] Instructions can be mapped to functional units of an NFU based on their resource requirements. For example, matrix-vector multiplication can only occur in a matrix-vector unit (MVU), while vector-vector addition can only occur in a multifunction unit (MFU). In this example, the instruction chain must be... Figure 4The pipeline shown uses the functional units of NFUs sequentially from left to right: Input Queue (IQ), MVU, then each MFU, and then Output Queue (OQ). However, in one example, the chain can start from IQ, MVU, or the first MFU. Finally, in this example, the first instruction in the chain that requires an MFU will be mapped to MFU 0, the next instruction will use MFU 1, and so on. This is important in this example because the local memory used by the MFU instruction is implicitly set by the depth of that instruction in the chain to be used for both storage and loading.

[0070] Most instructions can take the same general set of arguments: source, destination, and an optional second source. There may also be some memory management instructions that can specifically identify the target MFU, but the MFU to be executed is usually inferred from the instruction's position in the chain. Each source and destination argument can use one of the strong types in Table 3 below. For example, a DST argument means that the instruction's output can be forwarded, stored globally, or stored locally, while a GDST argument can only be stored globally. In this example, only instructions that take GSRC can be appended to the chain, and only instructions that supply DST can pass data into the chain.

[0071] Normally, most instructions can be used in a chain or in standalone mode (global source and global destination). However, in this example, memory management instructions (those starting with VRF_) cannot be used in a chain, and some instructions (V_PASS, V_STORE) can only be used within a chain.

[0072]

[0073]

[0074] Table 3

[0075] Node services can use an Application Programming Interface (API) to communicate with each node, including the Control / Scalar Processor (CSP) and the NFU. The API can be used to send instructions to the NFU and accept / formulate requests / responses to the network. A node service can receive requests from the network, initiate subroutines using arguments from the request, and then send a response back over the network. In this example, the request may include a node header interpreted by the CSP and a payload forwarded to the NFU. Similarly, outgoing responses may have a node header and (optionally) a payload.

[0076] The programming model for nodes allows subroutines to take up to 30 runtime arguments. These arguments can be passed to the subroutines as “auxiliary data” via the node header. In one example, Aux[0] can be reserved for selecting a subroutine. In one example, a common use of runtime arguments is to set the number of iterations for a given LSTM evaluation. Each subroutine can be a series of API calls that perform matrix and vector operations. Each of these API calls can correspond to an NFU instruction, and when the CSP encounters one of these API calls, it can send that instruction to the NFU. In one example, all instructions can operate on a vector of dimension HWVEC_ELEMS and a matrix of dimension HWVEC_ELEMS x HWVECV_ELEMS. A set of auxiliary functions (as shown in Table 4 below) can be used to set the appropriate bit fields in the arguments:

[0077]

[0078] Table 4

[0079] Additionally, three auxiliary constants can be used, as shown in Table 5 below:

[0080] NEXT Set only the forward flag of DST PREV Set only the receive flag for GSRC ACC Sets only the store flag for DST (only used with the mv_mul API)

[0081] Table 5

[0082] One possible subroutine is loopback, which takes a vector from the input queue into the NFU and stores it in global memory, then reads it from global memory and sends it out of the NFU through the output queue. Example loopback subroutines are shown in Table 6 below:

[0083]

[0084]

[0085] Table 6

[0086] The loopback example can be extended by performing an element-wise sigmoid on the input vector before sending it back. One way to achieve this is to call the input vector, sigmoid, and output vector APIs in standalone mode, as shown in Table 7 below:

[0087]

[0088] Table 7

[0089] The example in Table 7 above does not use any instruction chaining; however, higher performance can be achieved by chaining all three operations together. To implement chaining, the data flow between the input queue, the S-shaped MFU execution, and the output queue is analyzed. Regarding... Figure 4 In the example NFU, the chain needs to pass data through the MVU and the other four MFUs in the pipeline. The chain uses fewer cycles because, as an example, it avoids multiple round trips through global memory, which can be a time-consuming operation. Table 8 below shows an example of a subroutine that has been optimized using the chain.

[0090]

[0091] Table 8

[0092] The second method uses more instructions but executes in significantly fewer cycles. Note that the second method discards both the input and output vectors after its initial use. If it is necessary to store these values ​​for later use, then subroutine 4 in Table 9 can be used.

[0093]

[0094] Table 9

[0095] The procedure for adding two vectors is built upon the procedure for obtaining the sigma of the vectors. The key difference is that addition requires a second input vector, which, in one example, must always originate from the local memory of the MFU performing the addition. In this example, if the second operand is not already in local memory, then another instruction (e.g., vrf_g_I_copy) is needed to place it there. Assuming that subroutine 4 of Table 9 has been executed and the vector has been stored in global address 0, and the sigma of that vector is 0 in the local address of MFU 0, the following API call adds the two vectors together and stores the result back in global memory.

[0096]

[0097] Table 10

[0098] As discussed earlier, any matrices used by programs employing the processing components are preloaded at startup because, in this example, the cost is O(HWVEC_ELEMS). 2It takes O(HWVEC_ELEMS) cycles to store the matrix from outside the node, while each other operation takes O(HWVEC_ELEMS) cycles. It may also be expected to preload a set of bias vectors that remains constant throughout execution. The following example subroutine stores two weight matrices in matrix memory (the matrix register file within the MVU). It then loads two bias vectors, one into the local memory of MFU 1 and one into MFU 2.

[0099]

[0100] Table 11

[0101] The performance of vector loading can be improved by using example subroutines with instruction links, as shown in Table 12 below.

[0102]

[0103]

[0104] Table 12

[0105] Once the matrices and vectors have been preloaded, Table 13 is an example illustrating the execution of the following subroutine: v_out = (m[0]*v_in[2] + m[1]*v_in[3] + v_in[0])*v_in[1]. Therefore, in this example, it is assumed that subroutine 6 has been called to preload m[0], m[1], v_in[0], and v_in[1].

[0106]

[0107] Table 13

[0108] In one example, the MVU feature can be used to optimize the resource utilization of subroutine 8 in Table 13, which uses MFU 0 to sum the results of two mv_mul() operations. MVU allows the result of matrix-vector multiplication to be kept in its accumulator register instead of the output vector. If the feature is enabled for mv_mul (by declaring the store flag), the result of the next mv_mul will be summed with the result of the first mv_mul. In this example, to perform this optimization, subroutine 6 can be modified to store v_in[0] and v_in[1] in MFU 0 and MFU 1 instead, as shown in Table 14 below. Table 15 below shows the optimized matrix-vector subroutine using links.

[0109]

[0110] Table 14

[0111]

[0112]

[0113] Table 15

[0114] Additional improvements can be made to subroutines by using coding and compiler techniques that avoid hard-coding addresses and instead expose the address intent using well-named constants. Furthermore, typedefs can be provided to identify vector, matrix, and MFU addresses respectively using VectorNum, Matrix, and MFU. To ensure consistency in coding, the styles shown in Table 16 below can be used:

[0115]

[0116] Table 16

[0117] In this example, some of the addresses shown in Table 17 below must be set to physical addresses in the node's distributed memory:

[0118]

[0119]

[0120] Table 17

[0121] As an example, an LSTM evaluator for a node with CSP and NFU may include a preloading step, followed by several iterations of evaluation until an output is produced. Thus, in this example, the preloading step first stores the weight matrix and bias vector required for evaluation across a distributed memory spanning the NFU. Next, for each iteration of the evaluation, the input vector arrives and is processed through an increasingly longer chain of instructions until an output is produced.

[0122] Figure 11 A dataflow graph 1100 for neural network evaluation is shown according to an example. At a high level, in this example, an input vector (IQ_V) is loaded and multiplied by two weight matrices (MV_MUL 1 and 4). The history vector is also multiplied by two other weight matrices (MV_MUL 2 and 3). Next, the results of one input / history pair are summed together, and so are another input / history pair (VV_ADD 1 and 3). After that, each summed vector is added against the bias vectors (VV_ADD 2 and 4), and then each summed vector is applied against the sigmoid (V_SIGM 1 and 2). The sigmoid results are then multiplied together, and the result is the final output (VV_MUL and OQ_V).

[0123] The dependency analysis of this dataflow graph shows opportunities for four instruction chains. Dependency analysis can be performed using a compiler or similar tools. For example, chain A (via...) Figure 11 (The dashed line in the diagram shows) obtains its vector input from the input queue, stores the vector in the GVRF, and then uses it to perform MV_MUL1 on a matrix in local memory. The result of MV_MUL is also stored in local memory. In this example, chain B (via...) Figure 11 (shown by the dashed line in the diagram) can immediately follow chain A through the pipeline because it takes an input (history vector) from global memory, retrieves the matrix for MV_MUL 2 from local memory, adds the preloaded bias vector for VV_ADD 1 in local memory to the locally stored result of MV_MUL 1 for VV_ADD 2, and performs V_SIGM 1 on the result of VV_ADD 2. The result of V_SIGM is also stored in local memory so that chain D (via... .... Figure 11 (The dashed line in the image shows this).

[0124] From Figure 11 The example data flow graph can be implemented in an optimized manner in the following example subroutines, which include subroutine 11 for LSTM initialization and subroutine 12 for LSTM evaluation.

[0125]

[0126] Table 18

[0127]

[0128]

[0129] Table 19

[0130] Figure 12 An example processing 1200 is shown, illustrating an instruction chain based on an example neural functional unit. Additionally, see reference... Figure 12 The diagram shows an approximate graph of the total execution time for subroutine 12. As previously explained, once the instruction chain (chain AD) has been controlled by the processor (e.g., Figure 3 The control / scalar processor) mapping processes instructions in a pipelined manner from left to right (e.g., the pipeline includes...). Figure 4The diagram shows the MVU and five MFUs. Therefore, chain A includes the processing of the MV_MUL1 instruction after loading the input vector into the matrix register file corresponding to the MVU. Chain B includes the parallel processing of the instructions MV_MUL2 and W_ADD1 by the MVU and MFU0. Simultaneously, W_ADD2 and V_SIGM1 are processed using, for example, MFU1 and MFU2. Chain C (which consists of only one instruction—MV_MUL3) is processed by the MVU. Finally, chain D is processed by the entire pipeline. The MVU processes both MV_MUL4 and W_ADD3 simultaneously, and the MFUs process the remaining instructions in chain D in parallel, including two instructions, V_PASS, which simply pass the input received from the previous MFU to the next MFU or output queue (OQ). In this example, the total execution time is primarily driven by the MV_MUL instruction, while all other vector operations are pipelined to MV_MUL. Furthermore, W_ADD1 and W_ADD3 occur within the MVU's accumulator register.

[0131] In another LSTM implementation, two additional steps are performed before the vector and scalar data are processed by the NFU. First, a loop is used to unroll the LSTM for the multiple iterations provided for each request. Second, matrix blocking is used to map the algorithm matrix and vector sizes to processors that can have different native sizes. In this example, we have 250 HWVEC_ELEMS, while the algorithm input vector is of size 200, the remaining algorithm vectors are of size 500, the algorithm input matrix is ​​500x200, and the algorithm history matrix is ​​500x500. Therefore, the input matrix-vector multiplication is blocked as two MV_MUL() calls (with some zero padding), while the history matrix-vector multiplication is blocked as four MV_MUL() calls and two VV_ADD() calls.

[0132] As part of this example, there are three subroutines: one to initialize the set of matrices, one to initialize the bias vectors, and one to perform an LSTM evaluation on the set of vectors. As an example, at startup, a use case would be to call matrix initialization (e.g., subroutine 13 in Table 20) three times on each of the eight matrices to load 24 matrices, and then call bias vector initialization (e.g., subroutine 14 in Table 21) once. Then, at runtime, in this example, an LSTM evaluation would be called for each query (e.g., subroutine 15 in Table 22).

[0133]

[0134] Table 20

[0135]

[0136] Table 21

[0137]

[0138]

[0139]

[0140]

[0141]

[0142]

[0143]

[0144] Table 22

[0145] While the above subroutine examples provide corresponding instruction sets to illustrate various embodiments, such instruction sets may include more or fewer instructions. Furthermore, these instructions may be executed in different orders. Additionally, although several parameters for configuring or controlling various aspects of nodes and node processing are described, these parameters are merely examples. Additional or fewer parameters may be used with different versions of the examples discussed in this disclosure.

[0146] Figure 13A dataflow graph 1300 for neural network evaluation is shown according to an example. The dataflow graph 1300 includes four instruction chains: chain 1, chain 2, chain 3, and chain 4. Chain 1 includes performing a matrix multiplication operation between weights corresponding to the neural network model (e.g., W_xi, which may correspond to the neural network model) and a runtime input vector (e.g., X). Chain 1 also includes performing a matrix multiplication operation between past values ​​of the input vector (e.g., H_PREV) and historical weights (e.g., W_hi). The multiplication output can be added using pointwise addition of the vectors. Next, as part of chain 1, a sigmoid operation can be performed. As part of chain 2, a matrix multiplication operation can be performed between the forget gate weight matrix (e.g., W_xf) and the runtime input vector (e.g., X). Chain 2 also includes performing a matrix multiplication operation between past values ​​of the input vector (e.g., H_PREV) and historical forget gate weight matrices (e.g., W_hf). The multiplication output can be added using pointwise addition of the vectors. Next, as part of chain 2, the bias input vector (e.g., b_i) can be added to the output of the addition operation. Next, as part of chain 2, a sigmoid operation can be performed. Now turning to chain 3, as part of chain 3, a matrix multiplication operation can be performed between the gate weight matrix (e.g., W_xc) and the runtime input vector (e.g., X). Chain 3 also includes performing a matrix multiplication operation between past values ​​of the input vector (e.g., H_PREV) and the historical gate weight matrix (e.g., W_hc). The multiplication output can be added using pointwise addition of the vectors. Next, as part of chain 3, the bias gate vector (e.g., b_c) can be added to the output of the addition operation. Next, as part of chain 3, a hyperbolic function operation can be performed. Regarding chain 4, a matrix multiplication operation can be performed between the output gate weight matrix (e.g., W_xo) and the runtime input vector (e.g., X). Chain 4 also includes performing a matrix multiplication operation between past values ​​of the input vector (e.g., H_PREV) and the historical output gate weight matrix (e.g., W_ho). The multiplication outputs can be added using pointwise vector addition. Next, as part of chain 4, the bias output gate vector (e.g., b_o) can be added to the output of the addition operation. Next, as part of chain 4, a sigmoid operation can be performed. Additional operations can be performed on the outputs of all four chains. Although Figure 13 The diagram illustrates a certain number of operations performed using a certain number of chains, but different numbers of chains can be used to perform additional or fewer operations.

[0147] Figure 14 A diagram illustrating how an instruction chain is processed using a hardware node (e.g., an FPGA) according to an example is shown. Example processing 1400 can be implemented using an architecture corresponding to the hardware node previously described. As shown via processing aspect 1410, a control processor (e.g., Figure 3 The control / scalar processor 320 can be configured to receive and decode instructions received from the queue. As part of the processing aspect 1410, the received instructions are processed such that they are fanned out into separate control queues, such as... Figure 14 The various queues shown are part of block 1410. In one example, an independent control queue can determine which part of a neural functional unit (e.g., any NFU described earlier) is activated. Each processing unit, including MVUs and MFUs, will see the head of the control queue associated with that unit. In this example, having independent control queues for various processing elements allows instruction chains to be processed in parallel to the extent that the input to the chain does not depend on the output from another chain. Additionally, with respect to MFUs, various multiplexers, control gates, or other similar structures will be activated based on corresponding control signals issued by the respective control queues. As an example, the addition block of an MFU can receive input via a bitstream input to the hardware node when the multiplexer that controls the input to the addition block of the MFU passes through the input. In one example, the bitstream can represent the contents of a lookup table and its connections. The queue for inputting data to the addition block may not be dequeued until all vectors have passed through. Intermediate outputs generated during the execution of the corresponding chain can be stored in a local vector register file corresponding to the associated MVU or MFU. Subsequent instructions from the corresponding chain can then retrieve this value for further consumption. Most instructions may include a set of arguments: source, destination, and an optional second source. Each source and destination argument can use one of the argument types from Table 4, as previously described. As an example, a DST argument means that the instruction's output can be forwarded, stored globally, or stored locally, while a GDST argument means that the instruction's output can only be stored globally (e.g., as part of the associated GVRF register). In one example, only instructions that can obtain input from the GSRC can be appended to the chain, and only instructions that can supply DST arguments can pass data down to another chain.

[0148] refer to Figure 14In the example shown, once the opcode associated with the instruction at decision block 1420 is decoded, it can be determined which target functional unit the instruction targets. If the instruction targets the input queue (IQ_V (block 1412)) and the argument is DST (block 1422), the hardware node logic can move the next vector into the DST via the input queue into the NFU (block 1424). If the argument associated with the instruction is GDST (Global DST), the next vector can be moved into the global register at the address associated with GDST. If the argument associated with the instruction is LDST (Local DST), the next vector can be moved into the local register at the address associated with LDST. If the argument associated with the instruction is FGDST, the next vector can be moved into the global register at the address associated with FGDST.

[0149] Continue to refer to Figure 14 If the instruction targets a matrix-vector multiplication unit (MVU) (e.g., MV_MUL (block 1414)), the logic associated with the hardware node can take steps related to matrix multiplication obtained from a register or a specified address. The MV_MUL instruction may include multiplying a matrix at the MSRC by a vector from the GSRC and delivering it to the DST. Local storage can occur in the accumulator register and is mutually exclusive with forwarding. Regarding the source of the vector data, as part of block 1432, the logic associated with the hardware node can determine whether the data source is the GSRC or the output of another matrix multiplication operation (block 1438). Depending on the source, the vector data can be provided to the MVU. Alternatively, if the source of the vector data is a specific address (e.g., OP#2X21 (block 1436)), then the vector data can be loaded from that source and provided to the MVU. Additionally, regarding the destination of the vector data, if the argument associated with the instruction is the GDST (Global DST), then the next vector can be moved into the global register according to the address associated with the GDST. Intermediate results can be locally stored in the accumulator (ACC). Advantageously, in one example, instead of storing the results as intermediate results, they can be forwarded directly to the next MVU (as shown in block 1440). If the argument associated with the instruction is FGDST, then the next vector can be moved into the global register at the address associated with FGDST.

[0150] Still referencing Figure 14If the instruction is an S-shaped operation (V_SIGM (block 1416)) and therefore targets one of the MFUs, then the logic associated with the hardware node can determine which of the MFUs (e.g., MFU#0, MFU#1, MFU#2, MFU#3, or MFU#4) to target (block 1452). In this example, as... Figure 14 As shown, this instruction targets MFU#0 for an S-shaped operation. As part of block 1460, additional processing can be performed by logic associated with the hardware node. For example, in block 1462, the logic associated with the hardware node can determine whether the data source is GSRC or output from another source (block 1438). Depending on the source, vector data can be provided from GSRC or other sources (block 1466). A point-by-point S-shaped operation can be performed on the vector data (e.g., obtained from GSRC) and delivered to the DST. At block 1464, the logic associated with the hardware node can determine the destination of the output. Data can be provided to any destination listed in block 1468. Although... Figure 14 It illustrates a series of actions linked in some way to an instruction, but other actions can be performed. Additionally, data can be obtained from... Figure 14 Obtained from sources other than those described in the text, and can be provided to Figure 14 Other destinations besides those described in the text.

[0151] Figure 15 A flowchart 1500 of a method for processing instructions according to an example is shown. In one example, the method can be executed using a hardware node that includes a pipeline for processing instructions, wherein the pipeline may include: a matrix vector unit; a first multifunction unit, wherein the first multifunction unit may be connected to receive input from the matrix vector unit; a second multifunction unit, wherein the second multifunction unit may be connected to receive output from the first multifunction unit; and a third multifunction unit, wherein the third multifunction unit may be connected to receive output from the second multifunction unit. The method may include steps (e.g., step 1520) including: receiving a first type of instruction from an input queue via a global register, wherein the first type of instruction can only be executed by the matrix vector unit, and executing the first type of instruction by the matrix vector unit and storing a first result of the first type of instruction in at least one accumulator register. The first type of instruction may be a matrix operation, and the second type of operation may be any of the auxiliary operations, such as addition, multiplication, tangent, or sigmoid. The auxiliary operations may also include NOPs. The hardware node may include any of the neural functional units described above.

[0152] The method may also include steps (e.g., step 1530) including: receiving a second type of instruction from an input queue via a global register, wherein the second type of instruction can only be executed by at least one of a first multifunction unit, a second multifunction unit, or a third multifunction unit; and executing the second type of instruction using the first multifunction unit, wherein the execution includes performing an operation on a first result of at least a first type of instruction and generating a second result. In one example, this step may be performed by any of the NFUs previously described. During instruction execution, they may fan out into separate control queues, and it may be determined which part of the NFU is active. Each component of the NFU can access the head of the control queue and perform operations accordingly, such as addition, multiplication, tangency, sigmoid, or NOP. The control queue may not dequeue until all vectors have passed through, and then the control queue can be passed to the next component.

[0153] The next step (e.g., step 1540) may include: passing the second result to the second multifunction unit without storing the first or second result in a global register, and then passing the second result to the third multifunction unit. Passing the result without storage may include targeting the MFU but performing only no operation. This can be determined by the logic associated with the hardware node. Because the result is not stored in a global register, expensive operations such as writes to global registers can be eliminated during the execution of certain linking instructions. Figure 12 Chain D is an example of different types of instructions chained together, including passing the result to the output queue via two MFUs. The MVU processes MV_MUL4 and W_ADD3 simultaneously, and the MFUs process the remaining instructions in chain D in parallel, including two instructions, V_PASS, which simply pass the input received from the previous MFU to the next MFU or the output queue (OQ). In this example, the total execution time is primarily driven by the MV_MUL instructions, while other vector operations are pipelined to MV_MUL. Furthermore, W_ADD1 and W_ADD3 occur within the MVU's accumulator register.

[0154] In summary, this disclosure relates to a method for processing instructions in a hardware node including global registers and a pipeline, the pipeline comprising: a matrix vector unit including at least one accumulator register; a first multifunction unit connected to receive input from the matrix vector unit; a second multifunction unit connected to receive output from the first multifunction unit; and a third multifunction unit connected to receive output from the second multifunction unit. The method may include receiving a first type of instruction from an input queue via a global register, wherein the first type of instruction can only be executed by the matrix vector unit, and the matrix vector unit executing the first type of instruction and storing a first result of the first type of instruction in at least one accumulator register. The method may also include receiving a second type of instruction from an input queue via a global register, wherein the second type of instruction can only be executed by at least one of the first, second, or third multifunction units, and executing the second type of instruction using the first multifunction unit, wherein the execution includes performing an operation on the first result of at least the first type of instruction and generating a second result. The method may further include passing the second result to a second multifunction unit without storing the first or second result in a global register, and then passing the second result to a third multifunction unit.

[0155] In one example, each of the first, second, and third multifunction units may further include a pointwise addition block, a pointwise multiplication block, an S-shaped block, a hyperbolic tangent block, and an operation-free block. The method may also include passing the second result to the output queue coupled to the pipeline after passing it to the third multifunction unit. The first type of instruction may be a vector-type instruction, and the second type of instruction may be a scalar-type instruction. In this example, the first type of instruction may include a store indicator that causes the first result to be stored in at least one accumulator register, and the second type of instruction may include a forward indicator that causes the second result to be passed to the next unit in the pipeline. The first type of instruction and the second type of instruction may include an instruction chain.

[0156] In another example, this disclosure relates to a hardware node including a global register and a pipeline configured to process instructions, the pipeline including: a matrix vector unit including at least one accumulator register; a first multifunction unit connected to receive input from the matrix vector unit; a second multifunction unit connected to receive output from the first multifunction unit; and a third multifunction unit connected to receive output from the second multifunction unit. The hardware may also be configured to receive a first type of instruction from an input queue via a global register, wherein the first type of instruction can only be executed by the matrix vector unit, and the matrix vector unit executes the first type of instruction and stores a first result of the first type of instruction in at least one accumulator register; receive a second type of instruction from the input queue via a global register, wherein the second type of instruction can only be executed by at least one of the first, second, or third multifunction units; and execute the second type of instruction using the first multifunction unit, wherein the execution includes performing an operation on the first result of at least the first type of instruction and generating a second result, and passing the second result to the second multifunction unit without storing the first or second result in the global register, and then passing the second result to the third multifunction unit thereafter.

[0157] In this example, each of the first, second, and third multifunction units may further include a pointwise addition block, a pointwise multiplication block, an S-shaped block, a hyperbolic tangent block, and an operation-free block. The hardware node may also be configured to pass the second result to an output queue coupled to the pipeline after passing it to the third multifunction unit. The first type of instruction may be a vector-type instruction, and the second type of instruction may be a scalar-type instruction. In this example, the first type of instruction may include a store indicator that causes the first result to be stored in at least one accumulator register, and the second type of instruction may include a forward indicator that causes the second result to be passed to the next unit in the pipeline. The first type of instruction and the second type of instruction may include an instruction chain.

[0158] In yet another example, this disclosure relates to a hardware node including a global register and a pipeline configured to process instructions, the pipeline including: a matrix vector unit including at least one accumulator register; a first multifunction unit, wherein the first multifunction unit is connected to receive input from the matrix vector unit; a second multifunction unit, wherein the second multifunction unit is connected to receive output from the first multifunction unit; and a third multifunction unit, wherein the third multifunction unit is connected to receive output from the second multifunction unit, wherein each of the first, second, and third multifunction units further includes at least one of point-by-point addition and an operation-free block. The hardware can also be configured to receive a first type of instruction from the input queue via a global register, wherein the first type of instruction can only be executed by a matrix vector unit, and the matrix vector unit executes the first type of instruction and stores a first result of the first type of instruction in at least one accumulator register; receive a second type of instruction from the input queue via a global register, wherein the second type of instruction can only be executed by at least one of a first multifunction unit, a second multifunction unit, or a third multifunction unit; and execute the second type of instruction using the first multifunction unit, wherein the execution includes performing an operation on the first result of at least the first type of instruction and generating a second result, and passing the second result to the second multifunction unit without storing the first result or the second result in the global register, and then passing the second result to the third multifunction unit thereafter.

[0159] In this example, the hardware node can also be configured to pass the second result to the output queue coupled to the pipeline after passing it to the third multifunction unit. The first type of instruction can be a vector-type instruction, and the second type of instruction can be a scalar-type instruction. In this example, the first type of instruction may include a store indicator that causes the first result to be stored in at least one accumulator register, and the second type of instruction may include a forward indicator that causes the second result to be passed to the next unit in the pipeline. The first type of instruction and the second type of instruction may include an instruction chain.

[0160] It should be understood that the methods, modules, and components described herein are merely exemplary. Alternatively or additionally, the functional descriptions herein may be performed at least in part by one or more hardware logic components. Examples, but not limited to, illustrative types of hardware logic components that may be used include Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), Systems-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), etc. In an abstract but still explicit sense, any arrangement of components that implements the same functionality is effectively “associated” to achieve the desired functionality. Therefore, any two components combined herein to achieve a particular functionality can be considered “associated” with each other to achieve the desired functionality, regardless of the architecture or intermediate components. Similarly, any two such associated components can also be considered “operably connected” or “coupled” with each other to achieve the desired functionality.

[0161] Functionality associated with some of the examples described in this disclosure may also include instructions stored in non-transient media. As used herein, the term "non-transient media" refers to any medium that stores data and / or instructions that cause a machine to operate in a particular manner. Exemplary non-transient media include non-volatile media and / or volatile media. Non-volatile media include, for example, hard disks, solid-state drives, magnetic disks or magnetic tapes, optical disks or optical tapes, flash memory, EPROM, NVRAM, PRAM, or other such media, or networking versions of such media. Volatile media include, for example, dynamic memory such as DRAM, SRAM, cache, or other such media. Non-transient media are distinct from transmission media but can be used in conjunction with transmission media. Transmission media are used to transmit data to and / or to the machine and instructions from the machine. Exemplary transmission media include coaxial cables, fiber optic cables, copper wires, and wireless media such as radio waves.

[0162] Furthermore, those skilled in the art will recognize that the boundaries between the functionalities of the above-described operations are merely illustrative. The functionality of multiple operations may be combined into a single operation, and / or the functionality of a single operation may be distributed among additional operations. Moreover, alternative embodiments may include multiple instances of a particular operation, and the order of operations may be changed in various other embodiments.

[0163] Although specific examples are provided in this disclosure, various modifications and alterations may be made without departing from the scope of this disclosure as set forth in the following claims. Accordingly, the specification and drawings should be regarded as illustrative rather than restrictive, and all such modifications are intended to be included within the scope of this disclosure. Any benefit, advantage, or solution to the problem described herein with respect to specific examples is not intended to be construed as a key, essential, or necessary feature or element of any or all claims.

[0164] Furthermore, as used herein, the term "a" or "an" is defined as one or more. Moreover, the use of introductory phrases such as "at least one" and "one or more" in claims should not be construed as implying that the introduction of another claim element by the indefinite article "a" or "an" limits any particular claim, including the claim element containing such an introduction, to an invention containing only one such element, even if the same claim includes the introductory phrase "one or more" or "at least one" and indefinite articles such as "a" or "an." The same applies to the use of definite articles.

[0165] Unless otherwise stated, terms such as “first” and “second” are used to arbitrarily distinguish between the elements described by these terms. Therefore, these terms are not necessarily intended to indicate the time or other priority of these elements.

Claims

1. A method for processing instructions in a hardware node including a global register and a pipeline, the pipeline comprising: A matrix-vector unit, the matrix-vector unit including at least one accumulator register; A first multifunctional unit, wherein the first multifunctional unit is connected to receive input from the matrix vector unit; The method includes: a second multifunction unit connected to receive an output from the first multifunction unit; and a second multifunction unit connected to receive an output from the first multifunction unit. Receive a first type of instruction from the input queue, wherein the first type of instruction can only be executed by the matrix vector unit, and the matrix vector unit executes the first type of instruction and stores the first result of the first type of instruction in the at least one accumulator register; Receive a second type of instruction from the input queue, wherein the second type of instruction can only be executed by at least one of the first multifunction unit or the second multifunction unit, and execute the second type of instruction using the first multifunction unit, wherein the execution includes performing an operation on the first result of at least the first type of instruction and generating a second result; and The second result is passed to the second multifunction unit or coupled to the output queue of the pipeline.

2. The method according to claim 1, wherein each of the first multifunctional unit and the second multifunctional unit comprises: Pointwise addition block or no operation block.

3. The method according to claim 1, further comprising: After passing the second result to the second multifunction unit, the second result is passed to the output queue coupled to the pipeline.

4. The method of claim 1, wherein the first type of instruction includes vector type instructions, and the second type of instruction includes scalar type instructions.

5. The method of claim 1, wherein the first type instruction includes a storage indicator that causes the first result to be stored in the at least one accumulator register.

6. The method of claim 1, wherein the second type of instruction includes a forward indicator that causes the second result to be passed to the next unit in the pipeline.

7. The method of claim 1, wherein each of the first type of instruction and the second type of instruction comprises an instruction chain.

8. A hardware node, comprising: Global registers; A pipeline configured to process instructions, the pipeline comprising: a matrix vector unit including at least one accumulator register; a first multifunction unit connected to receive input from the matrix vector unit; and a second multifunction unit connected to receive output from the first multifunction unit; and The hardware node is also configured to: A first type of instruction is received from the input queue via the global register, wherein the first type of instruction can only be executed by the matrix vector unit, and the matrix vector unit executes the first type of instruction and stores the first result of the first type of instruction in the at least one accumulator register. A second type of instruction is received from the input queue via the global register, wherein the second type of instruction can only be executed by at least one of the first multifunction unit or the second multifunction unit, and the second type of instruction is executed using the first multifunction unit, wherein the execution includes performing an operation on the first result of at least the first type of instruction and generating a second result, and Without storing the first or second result in the global register, the second result is passed to the second multifunction unit.

9. The hardware node of claim 8, wherein the hardware node is further configured to: after passing the second result to the second multifunction unit, pass the second result to the output queue coupled to the pipeline.

10. The hardware node of claim 8, wherein the first type of instruction includes vector type instructions, and the second type of instruction includes scalar type instructions.

11. The hardware node of claim 8, wherein the first type of instruction includes a storage indicator that causes the first result to be stored in the at least one accumulator register.

12. The hardware node of claim 8, wherein the second type of instruction includes a forward indicator that causes the second result to be passed to the next unit in the pipeline.

13. The hardware node of claim 8, wherein each of the first type of instruction and the second type of instruction comprises an instruction chain.

14. A hardware node, comprising: Global registers; A pipeline configured to process instructions, the pipeline comprising: a matrix vector unit including at least one accumulator register; a first multifunction unit connected to receive input from the matrix vector unit; and a second multifunction unit connected to receive output from the first multifunction unit, wherein each of the first and second multifunction units includes at least one of point-by-point addition or no-operation block; and The hardware node is also configured to: A first type of instruction is received from the input queue via the global register, wherein the first type of instruction can only be executed by the matrix vector unit, and the matrix vector unit executes the first type of instruction and stores the first result of the first type of instruction in the at least one accumulator register. A second type of instruction is received from the input queue via the global register, wherein the second type of instruction can only be executed by at least one of the first multifunction unit or the second multifunction unit, and the second type of instruction is executed using the first multifunction unit, wherein the execution includes performing an operation on the first result of at least the first type of instruction and generating a second result, and Without storing the first or second result in the global register, the second result is passed to the second multifunction unit.

15. The hardware node of claim 14, wherein the hardware node is further configured to: after passing the second result to the second multifunction unit, pass the second result to the output queue coupled to the pipeline.

16. The hardware node of claim 14, wherein the first type of instruction includes vector type instructions, and the second type of instruction includes scalar type instructions.

17. The hardware node of claim 14, wherein the first type of instruction includes a storage indicator that causes the first result to be stored in the at least one accumulator register.

18. The hardware node of claim 14, wherein the second type of instruction includes a forward indicator that causes the second result to be passed to the next unit in the pipeline.

19. The hardware node of claim 14, wherein each of the first type of instruction and the second type of instruction comprises an instruction chain.

Images