Deep neural network accelerator with independent data paths for simultaneous processing of different classes of operations

Abstract

Deep neural network accelerators (DNNs) having independent data paths for concurrently processing different classes of operations and related methods are described. An example DNN accelerator includes an instruction scheduler to receive instruction chains having instructions to perform a first class of operations and a second class of operations corresponding to a neural network model. The DNN accelerator also includes a first data path and a second data path, where each data path is configured to locally execute at least one instruction chain before outputting any results. The instruction scheduler is configured to forward instructions to perform the first class of operations to the first data path and to forward instructions to perform the second class of operations to the second data path to overlap, in time, execution of at least a subset of the first class of operations with execution of at least a subset of the second class of operations.

Description

Background Technology

[0001] Neural network technology is used to perform complex tasks such as reading comprehension, language translation, or speech recognition. While neural networks can perform these tasks, deploying them using general-purpose CPUs or GPUs is costly. Furthermore, although GPUs offer higher throughput compared to CPUs, their latency is poor. Hardware accelerators used to accelerate such tasks are also limited in their ability to improve latency. Summary of the Invention

[0002] In one example, this disclosure relates to a deep neural network (DNN) accelerator including an instruction scheduler configured to receive one or more instruction chains, wherein at least one of the instruction chains includes both instructions for performing a first type of operation corresponding to a neural network model and instructions for performing a second type of operation corresponding to a neural network model. The DNN accelerator may further include a first data path including at least one matrix register file and at least one matrix-vector multiplier. The DNN accelerator may also include a second data path different from the first data path, the second data path including at least one vector register file and at least one functional unit, wherein each of the first and second data paths is configured to execute at least one instruction chain locally before outputting any result, and wherein the instruction scheduler is configured to forward at least a first set of instructions for performing the first type of operation to the first data path and forward at least a second set of instructions for performing the second type of operation to the second data path, and wherein the first and second data paths are configured to overlap in time the execution of a subset of at least the first type of operation corresponding to the first instruction set and the execution of a subset of at least the second type of operation corresponding to the second instruction set.

[0003] In another example, this disclosure relates to a method in a deep neural network (DNN) accelerator, the DNN accelerator including an instruction scheduler configured to receive one or more instruction chains, wherein at least one of the instruction chains includes both instructions for performing a first type of operation corresponding to a neural network model and instructions for performing a second type of operation corresponding to a neural network model, a first data path including at least one matrix register file and at least one matrix-vector multiplier, and a second data path, distinct from the first data path, including at least one vector register file and at least one functional unit, wherein each of the first and second data paths is configured to locally execute at least one instruction chain before outputting any result. The method may include the instruction scheduler forwarding at least a first set of instructions for performing the first type of operation to the first data path. The method may also include the instruction scheduler forwarding at least a second set of instructions for performing the second type of operation to the second data path. The method may further include using the first and second data paths to temporally overlap the execution of a subset of the first type of operation corresponding to the first instruction set and the execution of a subset of the second type of operation corresponding to the second instruction set.

[0004] In yet another example, this disclosure relates to a deep neural network (DNN) accelerator including a first data path, the first data path including at least one matrix register file and at least one matrix-vector multiplier. The DNN accelerator may also include a second data path different from the first data path, the second data path including at least one vector register file and at least one functional unit. The DNN accelerator may also include a switch coupled to a first local memory associated with the first data path and a second local memory associated with the second data path. The DNN accelerator may also include an instruction scheduler configured to: (1) access one or more instruction chains from at least one instruction queue, wherein at least one of the one or more instruction chains includes both instructions for performing a first type of operation corresponding to a neural network model and instructions for performing a second type of operation corresponding to a neural network model; and (2) divide the at least one or more instruction chains into a first chain including instructions solely for performing the first type of operation and a second chain including instructions solely for performing the second type of operation. The instruction scheduler can also be configured to forward a first chain to a first data path and a second chain to a second data path, wherein the first data path is configured to execute the first chain using a first local memory before outputting any results, and the second data path is configured to execute the second chain using a second local memory before outputting any results, and wherein the first data path and the second data path are further configured to overlap in time the execution of at least a subset of a first type of operation corresponding to the first chain and the execution of at least a subset of a second type of operation corresponding to the second chain.

[0005] This summary is provided to introduce a selection of concepts in a simplified form, which will be further described in the detailed description below. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Attached Figure Description

[0006] This disclosure is illustrated by way of example and is not limited to the accompanying drawings, wherein the same reference numerals indicate similar elements. The elements in the drawings are shown for simplicity and clarity and are not necessarily drawn to scale.

[0007] Figure 1 It is an architecture diagram of a neural functional unit (NFU) associated with a deep neural network (DNN) accelerator, based on an example.

[0008] Figure 2 This is an architecture diagram of another NFU associated with a DNN accelerator, based on an example.

[0009] Figure 3 This is a block diagram of an example system that includes DNN accelerators interconnected via a data center network;

[0010] Figure 4 It is a block diagram based on an example system that includes a DNN accelerator;

[0011] Figure 5 This is a block diagram based on an example DNN accelerator;

[0012] Figure 6 It is a block diagram of a neural functional unit based on an example;

[0013] Figure 7 It is a block diagram of another neural functional unit based on an example;

[0014] Figure 8 It is based on an example timing diagram, which shows a chain of instructions that are time-overlapping and executed by neural functional units;

[0015] Figure 9 This is another timing diagram based on an example, which shows a chain of instructions that are time-overlapping and executed by neural functional units;

[0016] Figure 10 This is another timing diagram based on an example, which illustrates a chain of instructions executed by neural functional units that are time-overlapping; and

[0017] Figure 11 A flowchart illustrating a method for processing instructions, based on an example, is shown. Detailed Implementation

[0018] The examples disclosed in this disclosure relate to systems, methods, and components for implementing neural network-based processing. Some examples relate to deep neural networks (DNNs). A DNN can be any suitable neural network for performing deep learning. As used herein, the term DNN includes, but is not limited to, any neural network comprising at least two layers between an input layer and an output layer. Some examples in this disclosure relate to DNN accelerators having independent data paths for simultaneously processing different classes of operations. DNN accelerators can be implemented using portions or combinations of field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), erasable and / or complex programmable logic devices (PLDs), programmable array logic (PAL) devices, and general-purpose array logic (GAL) devices. Image files can be used to configure or reconfigure DNN accelerators such as FPGAs. Image files or similar files or programs can be delivered from a host CPU via a network link or local link (e.g., PCIe). Information included in the image file can be used to program the hardware blocks of the DNN accelerator (e.g., logic blocks and reconfigurable interconnects of the FPGA) to achieve the desired functionality. The desired functionality can be implemented to support any service that can be provided via a combination of computing, networking, and storage resources, such as via data centers or other infrastructure used for service delivery.

[0019] Some examples may also involve DNN accelerators that can be configured to handle models that include not only tensor operations (e.g., matrix-vector multiplication) but also other heterogeneous nonlinear operations (e.g., softmax). As examples, transformer-based models (such as bidirectional encoder representation (BERT) models from transformers, GPT-2 models, and RoBERTa) include multiple classes of operations. When implemented using functional units arranged in sequential data paths, such models may encounter stalling and other execution problems. Some examples in this disclosure involve splitting sequentially pipelined data paths in a DNN accelerator into multiple independent data paths based on the types of operations they handle, allowing multiple classes of DNN operations to be executed concurrently using a single-threaded SIMD programming model with local communication only.

[0020] Figure 1This is an example architecture diagram of a Neural Functional Unit (NFU) 100 associated with a Deep Neural Network (DNN) accelerator. NFU 100 may include two separate data paths, data path 110 and data path 120. In this example, data path 110 may be configured as a Matrix Processing Unit (MPU) capable of performing tensor operations, such as matrix-vector multiplication or matrix inversion. Data path 110 may also include a Multifunctional Unit (MFU) for performing activation operations. Data path 110 may be configured to primarily handle first-class operations (e.g., matrix-based operations). In this example, data path 120 may be configured as a Vector Processing Unit (VPU) capable of performing tensor operations, such as vector-vector addition, vector-vector multiplication, vector-vector subtraction, or other nonlinear operations. Data path 110 may also be coupled to local memory 130 and local switch 150. Data path 120 may be coupled to local memory 140 and local switch 160. Figure 1 The example NFU 100 shown advantageously increases the efficient utilization of the data path and reduces inference latency. This is because... Figure 1 The NFU 100 in the code removes any pauses caused by structural hazards by splitting the data pipeline into multiple independent data paths and allowing instructions to be executed simultaneously locally using feedback. Therefore, as an example, transformer-based models such as the BERT model can adopt... Figure 1 The partitioning scheme shown. Although... Figure 1 The NFU 100 is shown as having a certain number of components arranged in a certain way, but the NFU 100 may include additional components or fewer components arranged in a different way.

[0021] Figure 2 This is an architectural diagram of a neural functional unit (NFU)200 associated with a deep neural network (DNN) accelerator, based on an example. However, as... Figure 2 As shown, the partitioning scheme is not limited to just two data paths; instead, it can be extended to any number of data paths, allowing layers requiring matrix processing and layers requiring vector processing to run simultaneously. As an example, each data path executes multiple instruction chains in its local loop, finally exchanging result data via a local switch and repeating these steps. A series of steps can be associated with an instruction chain that can include a set of SIMD instructions in a single-threaded program, starting with a read instruction and ending with a write instruction, including all intermediate data operation instructions (e.g., matrix multiplication or activation operations). This arrangement maximizes the use of two data paths whenever back-to-back instruction chain flows using different data paths exist. This is because the latency of one chain hides the latency of the other, thus reducing the overall inference latency.

[0022] Continue to refer to Figure 2 The NFU 200 may include four independent data paths: data path A 202, data path B 204, data path C 206, and data path D 208. In this example, each data path can be configured as a matrix processing unit (MPU) capable of performing tensor operations such as matrix-vector multiplication or matrix inversion, or as a vector processing unit (VPU) capable of performing tensor operations such as vector-vector addition, vector-vector multiplication, vector-vector subtraction, or other nonlinear operations. Data path A 202 may also be coupled to local memory 212 and local switch 222. Data path B 204 may also be coupled to local memory 214 and local switch 224. Data path C 206 may also be coupled to local memory 216 and local switch 226. Data path D 208 may also be coupled to local memory 218 and local switch 228. Although... Figure 2 The NFU 200 is shown as having a certain number of components arranged in a certain way, but the NFU 200 may include additional components or fewer components arranged in a different way. As an example, the number of data paths can be extended to any number of data paths required by the neural network model.

[0023] Figure 1 and Figure 2 One benefit of the architecture shown is its adaptability to heterogeneous computations and its agility / flexibility in adapting to new operators in rapidly evolving DNN models. Because each partitioned data path is assigned a specific set of operations in this architecture, not only can each data path be better optimized based on the assigned operation type, but new types of operators can also be added incrementally. As an example, regarding... Figure 1 The described bidirectional partitioning can optimize one data path for heavy tensor operations and another for nonlinear normalization operations such as softmax and layer normalization. These advantages can be achieved using configurable logic, such as FPGAs, which leverage the greater flexibility of a given silicon region. Thus, as an example, each data path can be configured accordingly, depending on the available FPGA resources, and new data paths can be easily added to existing ones when new operators need to be introduced. Besides FPGAs, other types of configurable hardware logic can also be used. As examples, application-specific integrated circuits (ASICs), erasable and / or complex programmable logic devices (PLDs), programmable array logic (PAL) devices, and general-purpose array logic (GAL) devices can also be used to implement data paths.

[0024] Figure 1 and Figure 2Another advantage of the architectures shown is the localized movement of data within the distributed memory. Therefore, these architectures do not have complex memory hierarchies; instead, they restrict data movement to its local memory during computation. Thus, each data path only receives / provides input / output from / to its neighbors, which maps well to acyclic data flows in DNN models. In contrast, traditional accelerator architectures include memory hierarchies with caches, which increases the cost of such architectures. Furthermore, the use of global shared memory requires multiplexing large amounts of traffic from multiple cores, increasing the complexity associated with shared memory architectures, such as synchronization and consistency. Moreover, advantageously, the localized data movement across distributed memory in the proposed architecture contributes to scalability, as its complexity increases only linearly with the addition of partitions or data paths.

[0025] Furthermore, the architecture of the DNN accelerator disclosed herein improves energy efficiency by using less energy through minimizing transactions between computational units and external memory. Finally, the architecture described herein preserves the use of a single-threaded programming model. This allows for easier code generation and additional optimizations from the compiler. Moreover, the code does not require synchronization locks, thus mitigating the risk of deadlock. In contrast, accelerator architectures relying on multi-threaded or multi-process programming models may require additional effort in programming, optimization, and validation.

[0026] In one example, this disclosure also relates to a DNN comprising multiple nodes, including a DNN accelerator, or such groups of nodes coupled to each other via a low-latency network. A converged platform utilizing hundreds or thousands of such nodes (e.g., FPGAs) can advantageously provide: (1) significantly reduced training / inference time by leveraging the parallelism of hundreds of thousands of nodes; (2) enabling new training / inference scenarios, such as online learning on real-time data; and (3) training unprecedentedly large-scale models in hyperscale data centers spanning hundreds of thousands of servers, utilizing flexible and alternative homogeneous FPGA resources. In one example, these advantages can be achieved by leveraging unconventional data representations that can utilize node architectures (such as FPGAs).

[0027] The aspects described can also be implemented in a cloud computing environment. Cloud computing can refer to a model used to provide on-demand network access to a shared pool of configurable computing resources. For example, cloud computing can be available on the market to provide ubiquitous and convenient on-demand access to a shared pool of configurable computing resources. A shared pool of configurable computing resources can be rapidly provisioned via virtualization and released with minimal management effort or service provider interaction, and then scaled accordingly. The cloud computing model can consist of various features, such as on-demand self-service, broad network access, resource pooling, rapid elasticity, and measurable services. The cloud computing model can be used to expose various service models, such as Hardware as a Service (“HaaS”), Software as a Service (“SaaS”), Platform as a Service (“PaaS”), and Infrastructure as a Service (“IaaS”). The cloud computing model can also be deployed using different deployment models, such as private cloud, community cloud, public cloud, and hybrid cloud.

[0028] Machine learning services, such as those based on recurrent neural networks (RNNs), convolutional neural networks (CNNs), long short-term memory (LSTM) neural networks, or gated recurrent units (GRUs), can be implemented using the systems and nodes described in this disclosure. In one example, service-related content or other information, such as words, sentences, images, videos, or other such content / information, can be converted into vector representations. Deep learning models can be trained offline before service initialization, and then deployed using the systems and nodes described in this disclosure. Nodes can be hardware-programmable logic devices that can be specifically customized to perform the types of operations that occur in the context of neural networks such as DNNs.

[0029] In one example, the state of the neural network model and the parameters used to control the model can be stored in the on-chip memory (e.g., BRAM) of the DNN accelerator (e.g., FPGA) after being transferred from external memory (e.g., DRAM). The neural network model can be pinned (e.g., preloaded) to the on-chip memory at service startup, and the contents of the on-chip memory can remain unchanged unless the model needs to be changed or other events require the model to be reloaded into the on-chip memory. Therefore, in this example, the neural network model can first be accessed from the DRAM associated with the hardware platform and then directly loaded into the on-chip memory (e.g., BRAM) of the DNN accelerator.

[0030] In one example, a neural network model can consist of many layers, each encoded as a matrix or vector of weights, represented as coefficients or constants obtained through offline training of the neural network. Programmable hardware logic blocks within nodes can process the matrices or vectors to perform various operations, including multiplication, addition, and other operations on input vectors representing encoded information related to the service. In one example, the matrix or vector of weights can be partitioned and fixed across multiple DNN accelerators using techniques such as graph partitioning. As part of this process, a large neural network can be transformed into an intermediate representation (e.g., a graph), which can then be sculpted into smaller representations (e.g., subgraphs), and each weight matrix in the weight matrix corresponding to each subgraph can be fixed in the local memory of the DNN accelerator. In one example, the model can be transformed into fixed-size matrices and vectors. This allows the resources of the DNN accelerator to operate in parallel on fixed-size matrices and vectors.

[0031] Figure 3 This is a block diagram of an example system 300 including nodes, where the nodes include DNN accelerators interconnected via a data center network 310. For example, as... Figure 3 As shown, multiple nodes 302, 304, and 306 can be coupled via a data center network 310. These nodes can be instantiated and used to parallelize multiple layers of a neural network model, such as a bidirectional encoder representation (BERT) model from a transformer. In one example, each node can be implemented as a server and may also include at least one DNN accelerator. Thus, node 302 may include DNN accelerator 322, node 304 may include DNN accelerator 324, and node 306 may include DNN accelerator 326. The DNN accelerators can be interconnected via a system based on a lightweight transport layer protocol. In one example, a first instance of DNN accelerator 322 may be coupled to a first instance of DNN accelerator 324 via transport link 332, and the first instance of DNN accelerator 322 may be further coupled to a second instance of DNN accelerator 324 via transport link 334. A first instance of DNN accelerator 324 can be coupled to a first instance of DNN accelerator 326 via transmission link 336, and the first instance of DNN accelerator 324 can be further coupled to the first instance of DNN accelerator 326 via transmission link 340. Similarly, a second instance of DNN accelerator 324 can be coupled to the first instance of DNN accelerator 326 via transmission link 342, and the second instance of DNN accelerator 324 can be further coupled to the second instance of DNN accelerator 326 via transmission link 338. A lightweight transport layer protocol can provide DNN accelerators with the ability to transmit or receive data packets or other such data to each other via data center network 310.

[0032] Continue to refer to Figure 3 DNN accelerators can also be interconnected in other configurations. For example, several instances of DNN accelerator 322 can be coupled to several instances of DNN accelerator 324 via multiple transmission links 352. Similarly, several instances of DNN accelerator 324 can be coupled to several instances of DNN accelerator 326 via transmission link 354. Although Figure 3 A certain number and arrangement of nodes including the DNN accelerator are shown, but there may be more or fewer nodes in different arrangements.

[0033] Figure 4 This is a block diagram of a system 400 including distributed nodes, based on an example. In this example, multiple nodes can be implemented as server racks in a data center. Each server in the rack can be coupled to a top-of-rack (TOR) switch 410. Other racks, though not shown, can have similar configurations. Each server can include at least one or more nodes. Each node can include a server (e.g., server 404, server 406, or server 408) and each server can be coupled to a TOR switch (e.g., TOR switch 410). Server 404 can include host components, including CPUs such as CPU 414 and CPU 416, which can be coupled to a node 418 including a DNN accelerator via a local link (e.g., PCIe) 420. Each node 418 can also be coupled via a network interface controller 422 (e.g., for communication across the data center's network infrastructure). Figure 4 The system shown allows nodes to process messages sent to (and / or to) TOR switches or other switches. Using this example system, individual nodes can send messages including data packets directly to each other, thus allowing even partitioning of a single neural network across multiple DNN accelerators. For communication, nodes can use lightweight protocols, including, for example, RDMA. Although... Figure 4 A certain number of components of a system arranged in a certain way are shown, but there may be more or fewer components arranged differently.

[0034] Figure 5This is a block diagram of an example DNN accelerator 500. Each DNN accelerator 500 may include an Input Message Processor (IMP) 510 for receiving messages from other nodes and an Output Message Processor (OMP) 540 for processing outgoing messages to other nodes or components. Each DNN accelerator 500 may also include a Control / Scalar Processor (CSP) 520 and a Neural Function Unit (NFU) 530. Although not shown, received messages by the DNN accelerator 500 may be stored in at least two different queues: (1) an IMP-to-CSP auxiliary queue and (2) an IMP-to-NFU data queue. Although not shown, outgoing messages may be stored in at least two different queues: (1) a CSP-to-IMP auxiliary queue and (2) an NFU-to-OMP data queue. In this example, the DNN accelerator 500 may accept off-chip messages containing both auxiliary information (such as control and scalar data) and payload data (e.g., vectors, matrices, or other tensor data structures). Ancillary information may include a request to perform computationally intensive operations on the payload and then return the result as an output message. In this example, the incoming message is processed by a lightweight input message processor (IMP) 510, which sends the ancillary information to a control / scalar processor (CSP) 520 and the payload data (e.g., an input tensor) to a neural functional unit (NFU) 530. As an example, the CSP 520 can then interpret the request and, based on its firmware, can send a series of instructions to the NFU 530. After a certain processing delay, the NFU can produce the requested result, which can be combined with the ancillary data produced by the CSP 520 in a lightweight output message processor (OMP) 540 and then sent off-chip. The CSP firmware can provide instructions to the NFU 530. Input messages can arrive from many sources, including over a network. The IMP 510 can inspect the head of the message queue and can dequeue any instructions that need to be executed and feed them through the system. Although Figure 5 A certain number of components of an example DNN accelerator 500 arranged in a certain way are shown, but there may be more or fewer components arranged differently.

[0035] Continue to refer to Figure 5 As an example, the NFU 530 can be based on Figure 1 or Figure 2The architecture shown is implemented in this example. In this example, the primary goal of the NFU is to perform matrix multiplication with high throughput and low latency using its matrix processing unit (MPU) and to perform vector operations, such as sigmoid operations, using its vector processing unit (VPU). The NFU can receive a matrix of coefficients (e.g., constants) and multiply these coefficients with dynamically generated input vector data. In one example, the coefficients can be loaded from off-chip memory (e.g., DRAM) into the on-chip memory corresponding to the NFU 530 (e.g., the FPGA's block random access memory (BRAM)) at service startup.

[0036] Figure 6 An example implementation of NFU 600 is shown. In this example, NFU 600 may include data paths 640 and 660 coupled via a local switch (e.g., a dual-port Tensor Memory Manager (TMM) 610). The dual-port TMM 610 may also be coupled to memory 620 and network interface 630. Memory 620 may be implemented using any suitable memory, including DRAM. Network interface 630 may allow NFU 600 to receive or send instructions or data via a data center network or another type of network. Each data path may include different functional units, including units such as matrix-vector multipliers and multifunction units. Each such unit may include a command FIFO, which can issue commands sequentially. In this example, the dual-port TMM 610 can ensure that any tensor output from any data path is forwarded in the order of the issued commands. As an example, once the instruction chain has completed its operation, the tensor provided by data path 640 can arrive as input to data path 660. The dual-port TMM 610 can ensure that output tensors are provided to data path 660 sequentially. In one example, the dual-port TMM 610 can be implemented as a dual-port interconnect. The dual-port TMM 610 can be implemented as a switching structure or in any other logical form. As an example, when implemented as part of an FPGA, the dual-port TMM 610 can be implemented by reconfiguring the FPGA's interconnects and configurable logic blocks.

[0037] Data path 640 may include multiple register files for storing instructions and matrix or vector data. As an example, data path 640 may include a matrix-vector multiplication initial vector register file (MVM_IVRF) 642, a matrix register file (MRF) 644, and an addition / subtraction vector register file (ADDSUB_VRF2) 646. Each of these register files can act as local memory for data path 640. Data path 640 may also include a matrix-vector multiplier (MVM) 648. Data path 640 may also include a multi-function unit (MFU: ADD / GELU 652) for performing addition or activation operations (e.g., Gaussian error linear unit (GELU)). Instructions from MVM_IVRF 642 can be provided to matrix-vector multiplier (MVM) 648. MVM 648 can perform matrix-vector multiplication operations as part of inference based on a neural network model (such as the BERT model). Therefore, in this example, a DNN accelerator including NFU 600 can be used to accelerate inference in the BERT model. In this example, MFU: ADD / GELU 652 can perform nonlinear operations on the intermediate results generated by MVM 648. Although not explicitly stated... Figure 6 As shown, however, each hardware block (including MVM 648 and MFU: ADD / GELU 652) used to perform operations on the data may include a local decoder to decode the received instructions and perform operations on the data received via MRF 644 and ADDSUB_VRF2 646, respectively.

[0038] Continue to refer to Figure 6Data path 660 may include multiple register files for storing instructions and vector data. As an example, data path 660 may include a multifunction unit instruction vector register file (MFU_IVRF) 662, an addition / subtraction vector register file (ADDSUB_VRF0) 664, a multiplication vector register file (MULTIPLY_VRF) 666, and another addition / subtraction vector register file (ADDSUB_VRF1) 668. Each of these register files can act as local memory for data path 660. Data path 660 may also include a multifunction unit (MFU: ADD / MULTIPLY) 672 for performing addition / multiplication operations. Data path 660 may also include a multifunction unit (MFU: ADD / EXP) 674 for performing addition / exponential operations. Instructions from MFU_IVRF 662 can be provided to MFU: ADD / MULTIPLY 672. MFU: ADD / MULTIPLY 672 can perform non-linear operations as part of inference based on neural network models (such as BERT models). MFU: ADD / EXP 674 can perform addition and activation operations. (Example...) Figure 6 As shown, the presence of multiple vector register files and associated functional units ensures efficient processing. As an example, addition (e.g., using MFU: ADD / MULTIPLY 672) can be performed after a multiplication operation (e.g., using MFU: ADD / EXP 674). Although not in Figure 6 As shown, additional units can be provided to include more flexibility, such as performing append operations followed by multiplication. Although not shown in... Figure 6 As shown, however, each hardware block (including MFU: ADD / MULTIPLY 672 and MFU: ADD / EXP 674) used to perform operations on the data may include a local decoder for decoding received instructions and performing operations on data received from any vector register files coupled to these blocks.

[0039] Still referencing Figure 6NFU 600 may also include a multiplexer 676. One input of multiplexer 676 may receive the output from MFU_IVRF 662, and a second input of multiplexer 676 may receive the output from data path 640. Thus, data path 660 can receive intermediate results as input from data path 640. Control signals (not shown) can select the input coupled to the output of multiplexer 676. NFU 600 may also include another multiplexer 678. One input of multiplexer 678 may receive the output from MFU:ADD / MULTIPLY 672, and a second input of multiplexer 676 may receive the output from MFU:ADD / EXP 674. Thus, in this example, the output of any MFU can be provided as input to a dual-port TMM 610, which can then forward the results of computations performed by either of the two MFUs. Control signals (not shown) can select the input coupled to the output of multiplexer 678. although Figure 6 A number of components of an NFU 600 arranged in a certain way are shown, but there may be more or fewer components arranged differently. As an example, the NFU 600 may include additional data paths and local switches. Additionally, as an example, the data paths may include additional hardware blocks that can perform operations on values ​​or operands received from multiple sources, including, for example, memory, networks, or another data path. As an example, one block may perform a tangent (e.g., TANH) operation. Another block may perform a sigmoid (SIGM) operation. Furthermore, the outputs of various hardware blocks may be supplied to a multiplexer as needed. Based on one or more control signals from various queue controllers or other control logic, the multiplexer can provide outputs that can be coupled to the next hardware block or output queue. In one example, the control signals may be signals received from an instruction decoder corresponding to the neural functional unit (NFU). As instructions are executed, they may be distributed into separate control queues, which may determine which part of the NFU 600 is activated.

[0040] Figure 7 An example implementation of the NFU 700 is shown. In this example, the NFU 700 may include data path A 740 and data path B 760. Data path A 740 may include as previously described... Figure 6 Similar components are described in data path 640. Similarly, data path B 760 may include components as previously described regarding... Figure 6Similar components are described in data path 660. The NFU 700 may also include two local switches; one of these switches may be implemented as a vector memory manager (VMM) for a matrix-vector multiplier (MVM) (e.g., VMM-MVM 710), and the other may be implemented as a vector memory manager (VMM) for a multifunction unit (MFU) (e.g., VMM-MFU 750). The VMM-MVM 710 may also be coupled to memory 720 and network interface 730. Memory 720 may be implemented using any suitable memory, including DRAM. Network interface 730 may allow the NFU 700 to receive or send instructions or data via a data center network or another type of network.

[0041] Continue to refer to Figure 7 Instructions corresponding to one or more layers associated with the neural network can be queued in instruction queue 770. Instruction scheduler 780 can provide instructions and control signals to each of the data paths, including data path A 740 and data path B 760. Instruction scheduler 780 can also control the operations of VMM-MVM 710 and VMM-MFU 750. In this way, instruction scheduler 780 can synchronize the operations being performed by the various components of NFU 700. As an example, instruction scheduler 780 can provide instruction chains for execution by each data path. As previously described, each data path (e.g., data path A 740 and data path B 760) can execute multiple instruction chains in its local loop, finally exchanging result data through local exchanges (e.g., VMM-MVM 710 and VMM-MFU 750), and repeating these steps as needed. A series of steps can be associated with an instruction chain that may include a SIMD instruction set in a single-threaded program, starting with a read instruction and ending with a write instruction, including all intermediate data operation instructions (e.g., matrix multiplication or activation operations). A matrix / vector register file can act as local memory for each data path used to perform these operations. This arrangement maximizes the use of both data paths whenever back-to-back instruction chain streams using different data paths exist. This is because the latency of one chain hides the latency of the other, thus reducing the overall inference latency.

[0042] Additionally, data paths can provide intermediate results to other data paths via local switches and any buffers needed for buffering results. Therefore, as... Figure 7 As shown, data path A 740 can provide intermediate results to data path B 760 via VMM-MVM 710 and buffer 752. Similarly, data path B 760 can provide intermediate results to data path A 740 via VMM-MFU 750 and buffer 712. Although Figure 7 The diagram shows a certain number of components of an NFU 700 arranged in a certain way, but there may be more or fewer components arranged differently. For example, an NFU 700 may include additional data paths and local switches.

[0043] In an example instruction set architecture (ISA) associated with nodes including NFUs, instructions can always be executed sequentially. Furthermore, all instructions can operate on vectors and / or matrices of original size. ISAs can be used to expose the large-scale pipelined parallelism available in distributed memory systems and NFUs. In one example, this parallelism can be leveraged by executing instruction chains in parallel. Most instructions can take the same common set of parameters: source, destination, and an optional second source. As an example, regarding... Figure 7 In the NFU 700, each local switch in the local switch can process instructions that include both source (SRC) and destination (DEST) as parameters. The source from which a local switch can receive operands or instructions may depend on the type of instruction configured to be processed by the corresponding data path. Similarly, the destination from which a local switch can output instructions or operands may depend on the type of instruction configured to be processed by the corresponding data path. As an example, the source of operands or instructions for the VMM-MVM 710 may include memory 720, network interface 730, and control logic associated with the NFU 700. The destination of operands and instructions for the VMM-MVM 710 may include... Figure 6 MVM_IVRF 642, Figure 6 ADDSUB_VRF2 646, Figure 6 The MRF 644 and other related components, such as the matrix memory manager (not shown). As an example, the source of operands or instructions for the VMM-MFU 750 may include a service processor unit (e.g., Figure 5 The control / scalar processor 520 and the control logic associated with the NFU 700. The destinations of the operands and instructions of the VMM-MFU 750 may include service processor units (e.g., Figure 5 Control / scalar processor 520), network interface 730, Figure 6 MFU_IVRF 662, Figure 6 ADDSUB_VRF0664, Figure 6 MULTIPLY_VRF 666 Figure 6 ADDSUB_VRF1 668, Figure 6 MFU: ADD / MULTIPLY 672, and Figure 6 MFU: ADD / EXP 674.

[0044] In one example, the program can be compiled or structured to attempt to chain as many instructions together as possible to maximize performance. Instructions can be chained together based on implicitly forwarding intermediate results to the next instruction. Alternatively, instruction pairs can be chained together by asserting a forward flag in the destination parameter of the first instruction and asserting a receive flag in the second instruction instead of the source address parameter. The chain continues until there are no more forward assertions in the instructions, or all functional units in the NFU have been used in the chain. Therefore, in this example, it is illegal to assert a forward flag in one instruction and then assert a receive flag in the next instruction (and vice versa). Instructions can be mapped to the data path of the NFU based on their resource requirements. For example, matrix-vector multiplication can only occur in data paths with matrix-vector units (MVUs), while vector-vector addition can only occur in data paths with multifunction units (MFUs).

[0045] Node services can communicate with each node, including the Control / Scalar Processor (CSP) and the NFU, using an Application Programming Interface (API). The API can be used to send instructions to the NFU and accept / formulate requests / responses to the network. A node service can receive a request from the network, initiate a subroutine with parameters from the request, and then send the response back over the network. In this example, the request may include a node header interpreted by the CSP, and a payload forwarded to the NFU. Similarly, outgoing responses may have a node header and (optionally) a payload. Each subroutine can be a series of API calls performing matrix and vector operations. Each of these API calls can correspond to an NFU instruction, and when the CSP encounters one of these API calls, it can send that instruction to the NFU.

[0046] When using Figure 6 NFU 600 or Figure 7When using the NFU 700 for deep neural network acceleration, certain types of transformer-based neural network models can particularly benefit from performance improvements. One such transformer-based model is the Bidirectional Encoder Representation (BERT) model, which includes multiple classes of operations. Such models can encounter pauses and other execution problems when implemented using functional units arranged in a sequential data path. BERT models can be used for word completion in sentences and other natural language processing (NLP) related problems. BERT models can be used to learn contextual relationships between words in text. BERT models can include an input stage, including embedding layers and post-processing layers. BERT models can also include transformers and decoding / loss layers. The embedding layer converts each input token (e.g., a word in a sequence) into a vector representation. The vector size can be a 768-dimensional vector or a 1024-dimensional vector. The post-processing layer can perform additional operations, including position embedding, sentence embedding, and token masking. Position embedding can be used to indicate the position of tokens within a sequence. Sentence embedding can be used to map sentences to vectors. Token masking can include replacing a certain percentage of words in each sequence with masked tokens. The decoding / loss layer can generate predictions based on training.

[0047] A neural network based on the BERT model can include multiple transformer layers. Each transformer layer can use self-attention to transform an input sequence into an output sequence. Transformers can include fully connected network (FCN) layers, including FCN (query), FCN (key), and FCN (value). Self-attention maps the set of queries and key-value pairs to the output. The attention function can be implemented using a softmax MFU. Transformers can also include additional fully connected network (FCN) layers, such as feedforward layers. Fully connected layers are a major factor in model performance in terms of the complexity of the computations associated with transformers.

[0048] To further explain the operations of NFU 600 and NFU 700, Table 1 below shows the instruction chain, including pseudocode corresponding to each layer of the BERT model.

[0049]

[0050]

[0051] Table 1

[0052] Before using the NFU 600 or NFU 700 to process the instructions shown in Table 1, the instructions in the instruction chain can be rearranged for execution. Rearrangement can be performed to utilize independent data paths of the NFU that can be used to perform certain operations in parallel. Table 2 below shows the instruction chains for rearrangement used with the NFU 600 or NFU 700.

[0053]

[0054]

[0055] Table 2

[0056] In the example of the instruction chains described in Tables 1 and 2, some instructions are configured to perform linear operations (indicated by the linear label in the pseudocode), while others are configured to perform non-linear operations (indicated by the absence of a linear label in the pseudocode). These instructions correspond to performing inference using a BERT model that uses multi-head attention as part of the transformer. This example assumes that multi-head attention is related to twelve-head attention, where twelve attention layers running in parallel can perform scaled dot-product attention computations. Each scaled dot-product input can consist of a query and a key, which can be multiplied by a dot product and then scaled. In this example, a softmax function can be applied to the scaled output to obtain the weights of the values. This example assumes that the queries are packed together in matrix Q, the keys are packed together in matrix K, and the values ​​are packed together in matrix V.

[0057] As an example, instruction 1 (q[1],q[2]=linear(x,MVM_IVRF,MVM_IVRF)) in Table 2 is marked as a linear instruction corresponding to performing matrix operations on queries q[1] and q[2], where the numbers 1 and 2 correspond to the indices in the header. Furthermore, for this instruction, both the source and destination of the operands are the vector register file associated with the matrix-vector multiplier (e.g., ...). Figure 6 MVM_IVRF 642). Instruction 2 (k[1],k[2]=linear(x,MVM_IVRF,MVM_IVRF)) in Table 2 is also marked as a linear instruction corresponding to performing matrix operations on keys k[1] and k[2], where the numbers 1 and 2 correspond to the indices in the head. Instruction 3 (v[1],v[2]=linear(x,MVM_IVRF,MVM_IVRF)) in Table 2 is also marked as a linear instruction corresponding to performing matrix operations on values ​​v[1] and v[2], where the numbers 1 and 2 correspond to the indices in the head. Table 2 includes several other instructions marked as linear. Table 2 also includes several non-linear instructions as part of the instruction chain. As an example, instruction 7 (p[1]=softmax(s[1],MFU_IVRF,MVM_IVRF)) in Table 2 corresponds to the non-linear operation: softmax, which is performed using a different data path, for example Figure 6 Data path 660. Although the above instruction chain includes corresponding instruction sets to illustrate various embodiments, the corresponding instruction sets may include more or fewer instructions. Furthermore, these instructions may be executed in different orders.

[0058] Figure 8 This is based on an example timing diagram 800, which illustrates a chain of instructions that are time-overlapping for execution by a neural functional unit. These chained instructions can be executed using any of the DNN accelerators and NFUs described earlier. As an example, Figure 8 This shows the independent data paths (e.g., Figure 7 The time overlap between two different types of operations performed by data paths A 740 and 760. This example involves operations performed as part of processing an attention sublayer associated with the transformer layer of the BERT model. Instruction chain 810 is shown to be processed in such a way that some instructions are chained together to be processed as part of MVM_CHAIN ​​830, while other instructions are chained together to be processed as part of MFU_CHAIN ​​860. As previously mentioned, the instruction scheduler associated with NFU (e.g., Figure 7 The instruction scheduler 780 can acquire instructions as an instruction queue (e.g., Figure 7 Instructions are stored in a portion of the instruction queue 770, and instructions are scheduled to be executed as part of one chain or another chain. As previously mentioned, two instruction chains can be processed in parallel by independent data paths (e.g., data path A 740 and data path B 760).

[0059] Continue to refer to Figure 8 Therefore, in this example, between time instances T1 and T2, data path A 740 can perform first-type operations related to the query (e.g., matrix operations). During this period, data path B 760 can be idle. However, between time instances T6 and T8, first-type and second-type operations can overlap by NFUs (e.g., NFU 700). Thus, in this example, between time instances T6 and T8, data path A 740 can perform linear operations related to the query and key (e.g., matrix operations), and data path B 760 can perform non-linear operations (e.g., softmax1). Similarly, between time instances T8 and T11, data path A 740 can perform linear operations related to the sentence's value and score (e.g., score 3 and score 4) (e.g., matrix operations), and data path B 760 can perform non-linear operations (e.g., softmax2). Furthermore, between time instances T11 and T15, data path A 740 can perform linear operations related to queries and keys (e.g., evaluating contexts such as context 1 and context 2) and other linear operations (e.g., matrix operations), and data path B 760 can perform non-linear operations (e.g., softmax3). Finally, as... Figure 8As shown, between time instances T15 and T16, data path A 740 can perform value-dependent linear operations (e.g., matrix operations), and data path B 760 can perform non-linear operations (e.g., softmax4). Although Figure 8 This demonstrates the use of a specific number of chains to perform a specific number of operations, but different numbers of chains can be used to perform additional or fewer operations.

[0060] Figure 9 This is another timing diagram 900 based on an example, which illustrates a chain of instructions that are time-overlapping for execution by a neural functional unit. These chained instructions can be executed using any of the DNN accelerators and NFUs described earlier. As an example, Figure 9 This shows the independent data paths (e.g., Figure 7 The time overlap between linear and nonlinear operations performed on data paths A 740 and 760. This example involves operations performed as part of the fully connected feedforward network and normalization layer associated with the encoder and decoder layers of the BERT model. Instruction chain 910 is shown to be processed in such a way that some instructions are chained together as part of MVM_CHAIN ​​930, while other instructions are chained together as part of MFU_CHAIN ​​960. As previously mentioned, in one example, the instruction scheduler associated with the NFU (e.g., Figure 7 The instruction scheduler 780 can draw from the instruction queue (e.g., Figure 7 Instructions are obtained from the instruction queue 770, and instructions are scheduled to be executed as part of one chain or another. As previously mentioned, two instruction chains can be processed in parallel by independent data paths (e.g., data path A 740 and data path B 760). Table 3 shows the instructions to be processed by the instruction scheduler (e.g., Figure 7 Example firmware pseudocode for scheduling instructions (780) of the instruction scheduler.

[0061]

[0062] Table 3

[0063] Continue to refer to Figure 9Therefore, in this example, in response to the instructions shown in Table 3, data path B 760 can perform a second type of operation (e.g., layer normalization) between time instances T1 and T2. During this period, data path A 740 can be idle. However, between time instances T2 and T3 and between time instances T3 and T4, the first and second types of operations can be overlapped by NFUs (e.g., NFU 700). Therefore, in this example, between time instances T2 and T3, data path A 740 can perform linear operations (e.g., matrix operations) related to a fully connected feedforward layer (e.g., fullyconnected1), and data path B 760 can perform non-linear operations (e.g., layernorm2). Similarly, between time instances T3 and T4, data path A 740 can perform linear operations (e.g., matrix operations) related to a fully connected feedforward layer (e.g., fullyconnected2), and data path B 760 can perform non-linear operations (e.g., layernorm3). Ultimately, between time instances T4 and T5, data path A 740 can perform linear operations (fully connected 3). During this period, data path B 760 can be idle. Although Figure 9 This demonstrates the use of a specific number of chains to perform a specific number of operations, but different numbers of chains can be used to perform additional or fewer operations.

[0064] Figure 10 This is another timing diagram 1000 based on an example, which illustrates a chain of instructions that are time-overlapping for execution by a neural functional unit. These chained instructions can be executed using any of the DNN accelerators and NFUs described earlier. As an example, Figure 10 This shows the independent data paths (e.g., Figure 7 The time overlap between at least two types of operations performed by data paths A 740 and 760. This example relates to inference operations performed as part of the feedforward layer processing the BERT model. Instruction chain 1010 is shown to be processed in such a way that some instructions are chained together as part of MVM_CHAIN ​​1030, while other instructions are chained together as part of MFU_CHAIN ​​1060. As previously mentioned, in one example, the instruction scheduler associated with the NFU (e.g., Figure 7 The instruction scheduler 780 can be used as an instruction queue (e.g., Figure 7The instruction queue 770 stores a portion of the instructions and schedules instructions to be executed as part of one chain or another. As previously mentioned, two instruction chains can be processed in parallel by independent data paths (e.g., data path A 740 and data path B 760). Table 4 shows the instructions to be processed by the instruction scheduler (e.g., Figure 7 Example firmware pseudocode for scheduling instructions (780) of the instruction scheduler.

[0065]

[0066] Table 4

[0067] Continue to refer to Figure 10 Therefore, in this example, in response to the instructions shown in Table 4, data path A 740 can perform a first-type operation (e.g., linear1) between time instances T1 and T2. During this period, data path B 760 can be idle. However, between time instances T2 and T3 and between time instances T3 and T4, the first and second-type operations can overlap by NFUs (e.g., NFU 700). Therefore, in this example, between time instances T2 and T3, data path A 740 can perform a linear operation (e.g., linear2), and data path B 760 can perform a non-linear operation (e.g., layernorm1). Similarly, between time instances T3 and T4, data path A 740 can perform a linear operation (e.g., linear3), and data path B 760 can perform a non-linear operation (e.g., layernorm2). Finally, between time instances T4 and T5, data path B 760 can perform a non-linear operation (e.g., layernorm3). During this period, data path A 740 can be idle. Although Figure 10 It shows the use of a specific number of chains to perform a specific number of operations, but different numbers of chains can be used to perform additional or fewer operations.

[0068] Figure 11 A flowchart 1100 illustrates a method for processing instructions according to an example. In one example, the method can be executed using any DNN accelerator, including any NFU described above. As an example, the method can be executed on a deep neural network (DNN) accelerator (e.g., Figure 5 Executed in (500), the accelerator includes an instruction scheduler configured to receive one or more instruction chains (e.g., Figure 7The instruction scheduler 780). At least one instruction chain in one or more instruction chains (e.g., the instruction chains shown in Table 2) may include both instructions for performing a first type of operation corresponding to a neural network model (e.g., at least one layer of a BERT model) and instructions for performing a second type of operation corresponding to a neural network model. The NFU of the accelerator corresponding to the DNN (e.g., Figure 6 The NFU 600 may include a first data path (e.g., Figure 6 Data path 640) and a second data path different from the first data path (e.g., Figure 6 Data path 660), the first data path includes at least one matrix register file (e.g., Figure 6 MRF 644) and at least one matrix-vector multiplier (e.g., Figure 6 The second data path includes at least one vector register file (e.g., MVM 648). Figure 6 MFU_IVRF662) and at least one functional unit configured to perform at least one nonlinear operation (e.g., Figure 6 The MFU (Add / Multiple 672) can be configured to execute at least one instruction chain locally before any results are output. The method may include steps (e.g., step 1110) including an instruction scheduler forwarding at least a first set of instructions for performing a first type of operation to the first data path. As an example, this step may include the instruction scheduler forwarding... Figure 8 The instruction set in MVM_CHAIN ​​830 is forwarded to Figure 7 Data path A 740. Before forwarding instructions, the instruction scheduler can chain instructions (e.g., Figure 8 The instruction chain (810) is divided into at least two instruction sets.

[0069] The method may also include a step (e.g., step 1120) in which the instruction scheduler forwards at least a second set of instructions for performing the second type of operation to a second data path. As an example, this step may include the instruction scheduler forwarding... Figure 8 The instruction set in MFU_CHAIN ​​860 is forwarded to Figure 7 Data path B 760.

[0070] The next step (e.g., step 1130) may include using a first data path and a second data path to overlap in time the execution of at least a subset of a first type of operation corresponding to the first instruction set and the execution of at least a subset of a second type of operation corresponding to the second instruction set. As an example, as part of this step, such as Figure 8As shown, between time instances T6 and T8, linear operations corresponding to certain queries and values ​​can overlap with nonlinear operations corresponding to the softmax function. Although Figure 11 The implementation of the method associated with the BERT model is described, but the method can be used with any other neural network model that includes the implementation of multiple classes of operations.

[0071] In summary, this disclosure relates to a deep neural network (DNN) accelerator, including an instruction scheduler configured to receive one or more instruction chains, wherein at least one of the instruction chains includes both instructions for performing a first type of operation corresponding to a neural network model and instructions for performing a second type of operation corresponding to a neural network model. The DNN accelerator may further include a first data path, which includes at least one matrix register file and at least one matrix-vector multiplier. The DNN accelerator may also include a second data path, distinct from the first data path, which includes at least one vector register file and at least one functional unit, wherein each of the first and second data paths is configured to locally execute at least one instruction chain before outputting any result, and wherein the instruction scheduler is configured to forward at least a first set of instructions for performing the first type of operation to the first data path and forward at least a second set of instructions for performing the second type of operation to the second data path, and wherein the first and second data paths are configured to temporally overlap the execution of a subset of at least the first type of operation corresponding to the first instruction set and the execution of a subset of at least the second type of operation corresponding to the second instruction set.

[0072] The first type of operation may include matrix-matrix multiplication or matrix-vector multiplication, and the second type of operation may include activation operation, softmax operation, or layer normalization operation. A first data path may be coupled to a second data path via a first local switch, and the first local switch may be configured to use the first data path to forward at least one intermediate result generated from the execution of any operation corresponding to the first instruction chain to the second data path. A second data path may be coupled to the first data path via a second local switch, and the second local switch may be configured to use the second data path to forward at least one intermediate result generated from the execution of any operation corresponding to the second instruction chain to the first data path.

[0073] The instruction chain may include a single-instruction multiple-data (SIMD) instruction set arranged in a single thread, and the single thread may begin with a read instruction and end with a write instruction. A first data path may be coupled to at least a first local memory and a second data path may be coupled to at least a second local memory. The first data path may be configured to perform operations without accessing at least the second local memory, and the second data path may be configured to perform operations without accessing at least the first local memory. The neural network model may include a transformer-based model.

[0074] In another example, this disclosure relates to a method in a deep neural network (DNN) accelerator, the DNN accelerator including an instruction scheduler configured to receive one or more instruction chains, wherein at least one of the instruction chains includes both instructions for performing a first type of operation corresponding to a neural network model and instructions for performing a second type of operation corresponding to a neural network model, a first data path including at least one matrix register file and at least one matrix-vector multiplier, and a second data path, distinct from the first data path, including at least one vector register file and at least one functional unit, wherein each of the first and second data paths is configured to locally execute at least one instruction chain before outputting any result. The method may include the instruction scheduler forwarding at least a first set of instructions for performing the first type of operation to the first data path. The method may also include the instruction scheduler forwarding at least a second set of instructions for performing the second type of operation to the second data path. The method may further include using the first and second data paths to temporally execute a subset of at least the first type of operation corresponding to the first instruction set and to execute a subset of at least the second type of operation corresponding to the second instruction set.

[0075] The first type of operation may include matrix-matrix multiplication or matrix-vector multiplication, and the second type of operation may include activation operation, softmax operation, or layer normalization operation. A first data path may be coupled to a second data path via a first local switch, and the method may further include the first local switch using the first data path to forward at least one intermediate result generated from the execution of any operation corresponding to the first instruction chain to the second data path. A second data path may be coupled to the first data path via a second local switch, and the method may further include the second local switch using the second data path to forward at least one intermediate result generated from the execution of any operation corresponding to the second instruction chain to the first data path.

[0076] The instruction chain may include a single-instruction multiple-data (SIMD) instruction set arranged in a single thread, wherein the single thread may begin with a read instruction and end with a write instruction. A first data path may be coupled to at least a first local memory and a second data path may be coupled to at least a second local memory, and the first data path may be configured to perform operations without accessing at least the second local memory, and the second data path may be configured to perform operations without accessing at least the first local memory. The neural network model may include a transformer-based model.

[0077] In yet another example, this disclosure relates to a deep neural network (DNN) accelerator including a first data path, the first data path including at least one matrix register file and at least one matrix-vector multiplier. The DNN accelerator may also include a second data path different from the first data path, the second data path including at least one vector register file and at least one functional unit. The DNN accelerator may also include a switch coupled to a first local memory associated with the first data path and a second local memory associated with the second data path. The DNN accelerator may also include an instruction scheduler configured to: (1) access one or more instruction chains from at least one instruction queue, wherein at least one of the one or more instruction chains includes both instructions for performing a first type of operation corresponding to a neural network model and instructions for performing a second type of operation corresponding to a neural network model; and (2) divide the at least one or more instruction chains into a first chain and a second chain, the first chain including instructions for performing only the first type of operation and the second chain including instructions for performing only the second type of operation. The instruction scheduler can also be configured to forward a first chain to a first data path and a second chain to a second data path, wherein the first data path is configured to execute the first chain using a first local memory before outputting any results, and the second data path is configured to execute the second chain using a second local memory before outputting any results, and wherein the first and second data paths are further configured to overlap in time the execution of at least a subset of a first type of operation corresponding to the first chain and the execution of at least a subset of a second type of operation corresponding to the second chain.

[0078] The first type of operation can include matrix-matrix multiplication or matrix-vector multiplication, and the second type of operation can include activation operation, softmax operation, or layer normalization operation. A switch can be configured to forward at least one intermediate result generated from the execution of any operation corresponding to the first instruction chain to a second data path using a first data path. A switch can be configured to forward at least one intermediate result generated from the execution of any operation corresponding to the second instruction chain to the first data path using the second data path.

[0079] The instruction chain may include a single-instruction multiple-data (SIMD) instruction set arranged in a single thread, and the single thread may begin with a read instruction and end with a write instruction. Neural network models may include transformer-based models.

[0080] It should be understood that the methods, modules, and components described herein are merely exemplary. Alternatively or additionally, the functionality described herein may be implemented at least in part by one or more hardware logic components. Illustrative types of hardware logic components that may be used, such as but not limited to, include Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SOCs), Complex Programmable Logic Devices (CPLDs), etc. In an abstract but still explicit sense, any arrangement of components that implements the same functionality is effectively “associated” to achieve the desired functionality. Therefore, any two components combined here to achieve a particular functionality can be considered “associated” with each other to achieve the desired functionality, regardless of the architecture or intermediate components. Similarly, any two components so associated can also be considered “operationally connected” or “coupled” with each other to achieve the desired functionality.

[0081] The functionality associated with some of the examples described in this disclosure may also include instructions stored in non-transient media. As used herein, the term "non-transient media" refers to any medium that stores data and / or instructions that cause a machine to operate in a particular manner. Exemplary non-transient media include non-volatile media and / or volatile media. Non-volatile media include, for example, hard disks, solid-state drives, magnetic disks or magnetic tapes, optical disks or magnetic tapes, flash memory, EPROM, NVRAM, PRAM, or other such media, or networking versions of such media. Volatile media include, for example, dynamic memory, such as DRAM, SRAM, cache, or other such media. Non-transient media are distinct from transmission media but may be used in conjunction with transmission media. Transmission media are used to transfer data and / or instructions to or from a machine. Exemplary transmission media include coaxial cables, fiber optic cables, copper wires, and wireless media such as radio waves.

[0082] Furthermore, those skilled in the art will recognize that the boundaries between the functionalities of the above operations are merely illustrative. The functionality of multiple operations can be combined into a single operation, and / or the functionality of a single operation can be distributed across additional operations. Moreover, alternative embodiments may include multiple instances of a particular operation, and the order of operations can be changed in various other embodiments.

[0083] Although specific examples are provided in this disclosure, various modifications and changes may be made without departing from the scope of this disclosure as set forth in the following claims. Therefore, the specification and drawings are to be considered illustrative rather than restrictive, and all such modifications are intended to be included within the scope of this disclosure. Any benefits, advantages, or solutions to problems described herein with reference to specific examples are not intended to be construed as key, essential, or fundamental features or elements of any or all claims.

[0084] Furthermore, as used herein, the term “a” or “an” is defined as one or more. Moreover, the use of introductory phrases such as “at least one” and “one or more” in claims should not be construed as implying that the introduction of another claim element by the indefinite article “a” or “an” limits any particular claim containing such an introduced claim element to an invention containing only one such element, even if the same claim includes the introductory phrase “one or more” or “at least one” and the indefinite article, such as “a” or “an”. The same applies to the use of definite articles.

[0085] Unless otherwise stated, terms such as “first” and “second” are used to arbitrarily distinguish the elements described by these terms. Therefore, these terms are not necessarily intended to indicate the time or other priority of these elements.

Claims

1. A hardware logic device including a deep neural network (DNN) accelerator, wherein the hardware logic device has a neural functional unit, the neural functional unit including a first data path, a first local memory dedicated to the first data path, a first local switch coupled to the first local memory, a second data path, a second local memory dedicated to the second data path, and a second local switch coupled to the second local memory. The hardware logic device is configured to perform actions, the actions including: Receive instruction chain, wherein the instruction chain includes: The first instruction for performing matrix operations corresponding to the neural network; and A second instruction for performing vector operations corresponding to the neural network; The first instruction is forwarded to the first data path, which includes a matrix register file and a matrix-vector multiplier, wherein the matrix-vector multiplier is configured to perform a matrix-vector multiplication operation based on the first instruction and the matrix register file, and further wherein the first instruction is forwarded to the first data path because it is used to perform the matrix operation; and The second instruction is forwarded to the second data path, which includes a vector register file and a functional unit, wherein the functional unit is configured to perform vector operations based on the second instruction and the vector register file, wherein the second instruction is forwarded to the second data path because it is used to perform the vector operations, and further wherein the matrix-vector multiplier performs the matrix operations in parallel with the functional unit performing the vector operations, and further wherein: The first data path is coupled to the second data path via the first local switch, the first local switch being configured to forward the first result generated by the matrix-vector multiplier to the second local switch, wherein the second local switch places the first result in the second local memory; and The second data path is coupled to the first data path via the second local switch, the second local switch being configured to forward the second result generated by the functional unit to the first local switch, wherein the first local switch places the second result in the first local memory.

2. The hardware logic device according to claim 1, wherein the vector operation is one of the following: activation operation, softmax operation, or layer normalization operation.

3. The hardware logic device of claim 1, wherein the instruction chain includes a single-instruction multiple-data (SIMD) instruction set arranged in a single thread, and wherein the single thread begins with a read instruction and ends with a write instruction.

4. The hardware logic device of claim 1, wherein the first data path is configured to perform operations without accessing the second local memory, and the second data path is configured to perform operations without accessing the first local memory.

5. The hardware logic device according to claim 4, wherein the neural network is a converter-based neural network.

6. The hardware logic device according to claim 1, wherein the hardware logic device is a field-programmable gate array (FPGA).

7. The hardware logic device according to claim 1, wherein the DNN is based on a converter model.

8. The hardware logic device of claim 1, wherein the neural functional unit comprises four data paths, wherein each data path is coupled to a dedicated local memory.

9. The hardware logic device according to claim 1, wherein the hardware logic device is implemented in a cloud computing scenario.

10. A method performed by a hardware logic device including a deep neural network (DNN) accelerator, wherein the hardware logic device has a neural functional unit, the neural functional unit including a first data path, a first local memory dedicated to the first data path, a first local switch coupled to the first local memory, a second data path, a second local memory dedicated to the second data path, and a second local switch coupled to the second local memory, the method comprising: Receive the first instruction for performing matrix-vector multiplication operations corresponding to the neural network; Receive a second instruction for performing vector operations corresponding to the neural network; The first instruction is forwarded to the first data path, wherein the first data path includes a matrix register file and a matrix-vector multiplier, and further wherein the first instruction is forwarded to the first data path because it is used to perform the matrix-vector multiplication operation; The second instruction is forwarded to a second data path that is different from the first data path, wherein the second data path includes a vector register file and a functional unit, and further wherein the second instruction is forwarded to the second data path instead of the first data path because the second instruction is forwarded to perform the vector operation; The matrix-vector multiplier in the first data path performs the matrix-vector multiplication operation based on the first instruction and the matrix register file; as well as The functional unit in the second data path executes the vector operation based on the vector register file and the second instruction, wherein the matrix-vector multiplier performs the matrix-vector multiplication operation in parallel with the functional unit performing the vector operation, and further wherein: The first data path is coupled to the second data path via the first local switch, the first local switch being configured to forward the first result generated by the matrix-vector multiplier to the second local switch, wherein the second local switch places the first result in the second local memory; as well as The second data path is coupled to the first data path via the second local switch, the second local switch being configured to forward the second result generated by the functional unit to the first local switch, wherein the first local switch places the second result in the first local memory.

11. The method of claim 10, wherein the vector operation is one of the following: activation operation, softmax operation, or layer normalization operation.

12. The method of claim 10, wherein the instruction chain comprises a single-instruction multiple-data (SIMD) instruction set arranged in a single thread, and wherein the single thread begins with a read instruction and ends with a write instruction.

13. The method of claim 10, wherein the first data path is coupled to a first local memory and the second data path is coupled to a second local memory, wherein the first data path is configured to perform operations without accessing the second local memory, and the second data path is configured to perform operations without accessing the first local memory.

14. The method of claim 13, wherein the neural network is a transformer-based neural network.

15. A hardware logic device including a deep neural network (DNN) accelerator, wherein the hardware logic device has a neural functional unit, the neural functional unit including a first data path, a first local memory dedicated to the first data path, a first local switch coupled to the first local memory, a second data path, a second local memory dedicated to the second data path, and a second local switch coupled to the second local memory, the DNN accelerator comprising: The first data path includes the matrix register file and the matrix-vector multiplier; The second data path, which is different from the first data path, includes a vector register file and functional units; The local switch is coupled to both a first local memory associated with the first data path and a second local memory associated with the second data path; The instruction scheduler is configured as follows: Accessing an instruction chain from an instruction queue, wherein the instruction chain includes: The first instruction for performing matrix-vector multiplication operations corresponding to neural networks; and A second instruction for performing vector operations corresponding to the neural network; and Divide the instruction chain into a first chain and a second chain: The first chain includes instructions solely for performing the matrix-vector multiplication operation; and The second chain includes instructions solely for performing the vector operations; Forward the first chain to the first data path; The second chain is forwarded to the second data path, wherein the first data path is configured to perform the matrix-vector multiplication operation in the first chain based on the matrix register file and using the first local memory, wherein the second data path is configured to perform the vector operation in the second chain based on the vector register file and using the second local memory, wherein the matrix-vector multiplication operation performed by the first data path and the vector operation performed by the second data path are performed in parallel. The first data path is coupled to the second data path via the first local switch, the first local switch being configured to forward the first result generated by the matrix-vector multiplier to the second local switch, wherein the second local switch places the first result in the second local memory; and The second data path is coupled to the first data path via the second local switch, the second local switch being configured to forward the second result generated by the functional unit to the first local switch, wherein the first local switch places the second result in the first local memory.

16. The hardware logic device according to claim 15, wherein the vector operation includes activation operation, softmax operation, or layer normalization operation.

17. The hardware logic device of claim 15, wherein the instruction chain includes a single-instruction multiple-data (SIMD) instruction set arranged in a single thread, and wherein the single thread begins with a read instruction and ends with a write instruction.

18. The hardware logic device of claim 17, wherein the neural network is a converter-based neural network.

19. The hardware logic device of claim 15, wherein the hardware logic device is a field-programmable gate array (FPGA).

20. The hardware logic device of claim 15, wherein the DNN is a converter-based model.

Images